Banyan: Improved Representation Learning with Explicit Structure

## Abstract

Abstract We present Banyan , an improved model to learn semantic representations by inducing explicit structure over data.
In contrast to prior approaches using structure spanning single sentences, Banyan learns by resolving multiple constituent structures into a shared one explicitly incorporating global context.
Combined with an improved message-passing scheme inspired by Griffin De et al. ( 2024 ) , Banyan learns significantly better representations, avoids spurious false negatives with contrastive learning, and drastically improves memory efficiency in such explicit-structured models.
Using the Self-StrAE Opper et al. ( 2023 ) framework, we show that Banyan (a) outperforms baselines using sentential structure across various settings (b) matches or outperforms unstructured baselines like GloVe ( + augmentations) and a RoBERTa medium ( + SimCSE ) pre-trained on 100M tokens, despite having just a handful of (non-embedding) parameters, and (c) also learns effective representations across several low resource (Asian and African) languages as measured on SemRel Ousidhoum et al. ( 2024b ) tasks.

## 1 Introduction

The principle of composition, states that in order to understand the meaning of a whole, one must understand the meaning of its parts and the structure that dictates how the parts combine. It is the bedrock of many linguistic and cognitive theories of how humans are able to understand efficiently learn to understand language and semantics more generally Chomsky (); Crain and Nakayama (); Pallier et al. (); de Marneffe et al. ().
Generally, most NLP models only consider composition implicitly, i.e., as something internal to the way representations are processed by the model. However, there has been a longstanding line of work that has sought to create models that operate on and learn explicit compositional structure. In general, the results show that explicit structure is an efficient inductive bias. In particular, Hu et al. (, ) and Sartran et al. () demonstrate that learned hierarchies lead to improved perplexity and superior single sentence GLUE Wang et al. () task performance over baseline transformer models Vaswani et al. (). In a similar vein, Opper et al. () show that an inductive bias for hierarchy can allow the learning of effective multi-level semantic representations even when pre-training occurs with limited data and parameter scale. It is this application with which we concern ourselves.

The model that Opper et al. () introduce is known as Self-StrAE (Self-Structuring AutoEncoder). Self-StrAE is a recursive neural network Socher et al. (, ), that learns both representations and structure. Though a very simple model (see 2), Self-StrAE is surprisingly effective. This is due to the power of the inductive bias behind it: letting representations define their own compositional hierarchies. However, we argue that the formulation of Opper et al. () does not fully exploit the inductive bias. To remedy this, we introduce Banyan, a revised formulation of the Self-StrAE with radically improved performance. Our contributions are as follows:

- 1.
We introduce a novel form of structure: entangled trees which leads to improved representations, and is significantly more compute efficient.
- 2.
We take inspiration from the recent resurgence in SSMs to replace the message passing functions of the Self-StrAE further improving performance and reducing compute cost.
- 3.
We test the resulting framework, which we call Banyan, across multiple languages, including in the low resource setting, and demonstrate that together with improved results, we are able to retain the data and parameter efficiency of the Self-StrAE.

## 2 Preliminaries

Given our central theme involves the use of explicit structure, we begin with a brief discussion of the Self-StrAE framework Opper et al. ().
This gives us a modelling basis which, given a sentence, can jointly induce both a hierarchical binary-tree structure as well as concomitant multi-level embeddings over the tree.

Self-StrAE involves three main components that acts over a sentence $\mathbf{w}=\langle w_{n}\rangle_{n=1}^{N}$ represented as a sequence of tokens.
These are:
(a) a procedure to determine which tokens to merge and in what order,
(b) message passing functions—composition and decomposition—that merge and split embeddings respectively, and
(c) a reconstructive objective that leverages both the induced structure and embeddings.
While we refer the reader to Opper et al. (, §2 & Appendix B) for a detailed description of these components, we briefly recap its operation to provide sufficient background for the development of Banyan (§ 3).

At a high-level, Self-StrAE functions by first *embedding* tokens to form an initial frontier using an embedding matrix $\Omega_{\Psi}$.
It then interleaves a greedy pairwise-clustering mechanism using the cosine similarity along with a parametric *composition function* $C_{\Phi}$ to recursively merge tokens upwards until it reduces to a single root embedding.
Viewing the resulting merge sequence as the induced binary tree for the sentence, it then traverses back down this tree structure, recursively splitting embeddings at every node using a *decomposition function* $D_{\Theta}$ to recover embeddings for the leaves.
Now, the model can optionally use a dembedding function $\Lambda_{\Gamma}$ to predict tokens $\widehat{w}_{n}$ from these leaf embeddings.
LABEL:fig:selfstrae illustrates the operation of the model on an exemplar sentence.

Note that the model generates *two* sets of embeddings; one set going up from leaves to root, and another coming back down from root to leaves and denote these $\bar{e}$ and $\underline{e}$ respectively
We also note that an embedding is typically viewed as $e\in\operatorname{{}\mathbb{R}}^{U\times{}K}$ with $K$ independent channels—of particular relevance to the composition and decomposition functions which act independently over the channels.
Tokens are denoted as the vertices $w_{i}\in\Delta^{V}$ in a $V$-simplex for vocabulary size $V$.
All together, the functioning of the model is then characterised by:

$$ $\displaystyle\Omega_{\Psi}(w_{i})$ $\displaystyle=w_{i}\,\Psi,\quad\Psi\in\operatorname{{}\mathbb{R}}^{V\times(U \star{}K)}$ (1) $\displaystyle C_{\Phi}(\bar{e}_{i},\bar{e}_{i+1})$ $\displaystyle=\textsc{hcat}(\bar{e}_{i},\bar{e}_{i+1})\,\Phi+\phi$ (2) $\displaystyle\quad\Phi\in\operatorname{{}\mathbb{R}}^{2U\times U},\phi\in \operatorname{{}\mathbb{R}}^{U}$ $\displaystyle D_{\Theta}(\underline{e}_{i})$ $\displaystyle=\textsc{hsplit}(\underline{e}_{i}\,\Theta+\theta)$ (3) $\displaystyle\quad\Theta\in\operatorname{{}\mathbb{R}}^{U\times 2U},\theta\in \operatorname{{}\mathbb{R}}^{2U}$ $\displaystyle\Lambda_{\Gamma}(\underline{e}_{i})$ $\displaystyle=\underline{e}_{i}\,\Gamma\quad\Gamma\in\operatorname{{}\mathbb{R }}^{(U\star{}K)\times V}$ (4) $$

Given the nature of the model, a straightforward objective would be to simply reconstruct the tokens, formulated for sentence $\mathbf{w}$ and prediction $\widehat{\mathbf{w}}$ as

$$ $\displaystyle\mathcal{L}_{\text{CE}}(\mathbf{w},\widehat{\mathbf{w}})=-\frac{1 }{N}\sum_{n=1}^{N}w_{n}\cdot\log\widehat{w}_{n}.$ (5) $$

An alternate approach developed by Opper et al. () leverages the multi-level structure of the model to define a contrastive objective over a batch of sentences $\{\mathbf{w}_{b}\}_{b=1}^{B}$ with a total of $M$ nodes (internal + leaves).
Noting that the up and down trees share the same underlying structure (modulo reversed edges), this objective draws together corresponding up and down embeddings at a given tree position, whilst pushing away other embeddings across the batch, using the cosine similarity metric.
Denoting the pairwise similarity matrix $A\in\operatorname{{}\mathbb{R}}^{M\times M}$ between upward embeddings $\langle\bar{e}_{i}\rangle_{i=1}^{M}$ and downward embeddings $\langle\underline{e}_{i}\rangle_{i=1}^{M}$, and $A_{i\bullet},A_{\bullet{}j},A_{ij}$ the $i\textsuperscript{th}$ row, $j\textsuperscript{th}$ column, and $(i,j)\textsuperscript{th}$ entry of the matrix respectively, the objective is defined as:

$$ $\displaystyle\mathcal{L}_{\text{CO}}(\bar{e},\underline{e})=\frac{-1}{2M}\sum_ {i=1}^{M}\log\left(\sigma_{\!\tau}(A_{i\bullet})\,\sigma_{\!\tau}(A_{\bullet{} i})\right)$ (6) $$

with tempered softmax $\sigma_{\!\tau}(\cdot)$ (temperature $\tau$) normalising over the unspecified ($\bullet$) dimension.

## 3 Model

A particularly interesting characteristic of learning with explicitly structured models such as Self-StrAE or even earlier models such as the IORNN Le and Zuidema () is the dichotomy between the upward and downward embeddings.
Given how these models are constructed, the upward embeddings are always *non-contextual*: they do not encapsulate any of the surrounding context.
For example, following LABEL:fig:selfstrae, the upward embedding $\bar{e}$ for the span ‹ate doughnuts› is always the same regardless of context, no matter who did the eating.
In contrast however, the downward embeddings are always *contextual*: they must encapsulate the surrounding context by virtual of being decomposed from larger spans or units.
For our example, this implies that there are multiple downward embeddings $\underline{e}^{y}$ for the given span, one for each $y\in\{\text{Lisa, Homer, \ldots}\}$.
To learn effective embeddings then, one must *marginalised* over these different downward embeddings to ensure that their meaning resolves over all these contexts.

### 3.1 From trees to entangled trees

Given this need to resolve meaning across multiple contexts, it is obvious to see for prior models that simply employing a cross-entropy objective over tokens places no constraints that help achieve resolution.
Note however that Self-StrAE with the contrastive objective $(\mathcal{L}_{\text{CO}})$ provides an indirect constraint.
For a given node (i.e. span), by drawing together its upward (non-contextual) embedding and *each* downward (contextual) embedding, it implicitly draws together all the contextual embeddings for that given node.
On the other hand, by being contrastive, this normally also pushes away all other embeddings regardless of whether they correspond to the same span elsewhere in the data, which can typically result in the objective suffering from the presence of false negatives—terms that ought not to be pushed away, but are.

Our innovation here is to address both these issues together by formulating the process in terms of entangled trees—where entangling refers to reduction of a set of disjoint tree structures into a single conjoined graph structure.
An example of this is shown in LABEL:fig:entangling with disjoint trees on the left and resulting the entangled tree on the right.
Here, as shown, all instances of ‹night› and ‹some are born to› are captured by a single node representing that constituent.
We call our model Banyan on account of this entangling.

#### Entangling

Constructing an entangled tree given a set of disjoint trees is a relatively straightforward process and is formally specified in Algorithm 1.
In contrast to the agglomerative clustering employed in Self-StrAE, here we employ a global frontier spanning all leaf nodes across the given data.
The key differences to the prior methods are mainly to do with constructing a graph jointly with progressing the frontier and ensuring that new nodes are never duplicated, for which we employ a node identity $s_{n}$ in addition to the node embedding $e_{n}$.

Figure: Algorithm 1 Banyan: Entangled Compose

#### Incorporating context

Following the entangling of trees described, the model proceeds in a similar vein to Self-StrAE, by composing upwards from leaves to roots (multiple roots corresponding to multiple trees), and then decomposing downwards back to the leaves.
With entangled trees, while traversing upwards each node is always composed from the same two children, but on the way back down, things are different as each separate context for a given node provides a different downward embedding.
This is shown in LABEL:fig:banyan focussing on a subgraph of the entangled tree from LABEL:fig:entangling(right).
Note that the node in question (in blue) corresponds to the span ‹some are born to›, and has downward embeddings that incorporate context both from ‹endless night› and ‹sweet delight›.
This is exactly as desired, as Banyan allows explicit aggregation to derive the downward embedding that resolves over the contexts.
For any upward embedding $\bar{e}$ whose span occurs in different contexts $y\in\mathcal{Y}$, the corresponding downward embedding is derived by simply averaging over the different contextual down embeddings; i.e., $\underline{e}=\frac{1}{\lvert\mathcal{Y}\rvert}\sum_{y}\underline{e}^{y}$.

#### Effectiveness and efficiency

Beyond the ability to explicitly incorporate context across data, entangled trees also help the contrastive objective by avoiding false negatives since they do not admit duplicate nodes by construction.
Furthermore, the lack of duplicate nodes also drastically impacts the memory footprint of the model as one deals with the set of all nodes rather than counting each instance as its own node.
These effects becomes more pronounced when entangling a larger set of instances as the likelihood of false negative and duplicates goes up together.

#### Practical estimation

Given the advantages conferred by entangled trees, one would ideally want to construct it over *all* the available data.
This however is not practically feasible as the size of data typically grows exponentially with time.
To address this, we construct our model to estimate the given objective by taking steps over *batches* of data that are of a more manageable size, noting that this estimator is unbiased.
To see this is the case, note that entangled trees only affects the downward embeddings directly, and that batching simply means that the resolved embedding is an average over *samples* instead of over all the data (*population*)—the sample mean is always an unbiased estimator of the population mean.

### 3.2 Simplified Message Passing

Complementary to the development of entangled trees to incorporate context, we also explore avenues to improve the message passing with the composition $C$ and decomposition $D$ functions.
The original formulations of these from Eqs. 2 and 3 employ concatenation and splitting along with simple single-layer linear neural networks.
A key question for such functions is how well they propagate information (and gradients) through the recursive hierarchical structure.

Following the recent resurgence in state-space models (SSMs) and in particular, diagonally parametrised recurrent neural networks (RNNs) Peng et al. (); Orvieto et al. (); De et al. () that better handle information flow through sequence recurrence, we adopt the same for our functions.
Under this new parameterisation, we now define

$$ $\displaystyle C(\bar{e}_{i},\bar{e}_{i+1})$ $\displaystyle\!=\!(\bar{e}_{i}\!\cdot\!\sigma(\Phi_{l})+\bar{e}_{i+1}\!\cdot\! \sigma(\Phi_{r}))+\phi$ (7) $\displaystyle\quad\Phi_{l},\Phi_{r},\phi\in\operatorname{{}\mathbb{R}}^{U}$ $\displaystyle D(\underline{e}_{i})=\bigl{(}$ $\displaystyle\underline{e}_{i}\!\cdot\!\sigma(\Theta_{l})+\theta_{l},\;\; \underline{e}_{i}\!\cdot\!\sigma(\Theta_{r})+\theta_{r}\bigr{)}$ (8) $\displaystyle\quad\Theta_{l},\Theta_{r},\theta_{l},\theta_{r}\in\operatorname{ {}\mathbb{R}}^{U}$ $$

Here, $\sigma$ represents the sigmoid non-linearity applied to the parameters for numerical stability following De et al. ().

These relatively simple changes have a pretty drastic effect, both in terms of performance (see experiments) as well as memory footprint, with parameters now reduced by a factor of $U$ compared to the functions from Eqs. 2 and 3

## 4 Experimental Setting

**Table 1: Banyan compared with Self-StrAE pre-trained on 10 million tokens. Higher is better. Results represent the average across four random initialisations. Each of the plus rows represents an additional change to the model. Banyan and Self-StrAE are differentiated in terms of the structure they learn, +diag functions means additionally changing the message passing functions, and ++CE loss means additionally switching the pre-training objective.**
| Model | STS-12 | STS-13 | STS-14 | STS-15 | STS-16 | STS-B | SICK-R | SemRel | Score |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Self-StrAE | 31.98 $\pm$ 0.58 | 53.88 $\pm$ 0.68 | 37.73 $\pm$ 0.70 | 55.23 $\pm$ 0.58 | 55.55 $\pm$ 0.47 | 39.53 $\pm$ 1.61 | 51.78 $\pm$ 0.29 | 50.05 $\pm$ 0.92 | 46.59 $\pm$ 0.43 |
| + diag functions | 35.13 $\pm$ 0.33 | 56.05 $\pm$ 0.24 | 40.58 $\pm$ 0.05 | 58.83 $\pm$ 0.10 | 56.78 $\pm$ 0.21 | 44.10 $\pm$ 0.14 | 53.35 $\pm$ 0.17 | 52.65 $\pm$ 0.17 | 49.68 $\pm$ 0.06 |
| ++ CE loss | 47.10 $\pm$ 1.04 | 61.85 $\pm$ 1.44 | 58.60 $\pm$ 1.34 | 70.45 $\pm$ 0.57 | 62.45 $\pm$ 0.70 | 59.50 $\pm$ 0.53 | 59.00 $\pm$ 0.26 | 60.33 $\pm$ 0.26 | 59.91 $\pm$ 0.54 |
| Banyan | 38.98 $\pm$ 0.39 | 61.75 $\pm$ 0.14 | 43.65 $\pm$ 0.46 | 58.21 $\pm$ 0.41 | 55.29 $\pm$ 0.23 | 46.15 $\pm$ 0.71 | 53.93 $\pm$ 0.16 | 52.53 $\pm$ 0.09 | 51.31 $\pm$ 0.13 |
| + diag functions | 44.15 $\pm$ 0.002 | 62.80 $\pm$ 0.002 | 48.30 $\pm$ 0.001 | 64.60 $\pm$ 0.002 | 60.30 $\pm$ 0.001 | 49.80 $\pm$ 0.002 | 55.14 $\pm$ 0.001 | 57.70 $\pm$ 0.001 | 55.23 $\pm$ 0.001 |
| ++ CE loss | 51.20 $\pm$ 0.007 | 69.10 $\pm$ 0.002 | 63.20 $\pm$ 0.004 | 73.20 $\pm$ 0.002 | 66.60 $\pm$ 0.002 | 61.50 $\pm$ 0.002 | 55.50 $\pm$ 0.003 | 61.60 $\pm$ 0.002 | 62.70 $\pm$ 0.001 |

**Table 2: Banyan compared with Self-StrAE on lexical evaluation. Higher is better. Results are the average across four random intialisations. Additional rows represent the same changes as described in Table 1.**
| Model | Simlex | Wordsim-S | Wordsim-R | Score |
| --- | --- | --- | --- | --- |
| Self-StrAE | 13.80 $\pm$ 0.41 | 54.38 $\pm$ 0.78 | 52.85 $\pm$ 1.27 | 40.34 $\pm$ 0.66 |
| + diag functions | 16.90 $\pm$ 0.29 | 57.15 $\pm$ 0.29 | 58.53 $\pm$ 0.62 | 44.18 $\pm$ 0.33 |
| ++ CE loss | 22.85 $\pm$ 1.01 | 60.15 $\pm$ 2.08 | 63.43 $\pm$ 0.86 | 48.81 $\pm$ 0.48 |
| Banyan | 16.34 $\pm$ 0.87 | 57.10 $\pm$ 0.35 | 56.80 $\pm$ 0.47 | 43.41 $\pm$ 0.50 |
| + diag functions | 18.04 $\pm$ 0.002 | 55.50 $\pm$ 0.001 | 60.00 $\pm$ 0.002 | 44.50 $\pm$ 0.001 |
| ++ CE loss | 16.57 $\pm$ 0.02 | 63.25 $\pm$ 0.03 | 69.00 $\pm$ 0.01 | 49.61 $\pm$ 0.02 |

Figure: (a) Changing Structure
Refer to caption: extracted/5753996/tangvsent.png

We want to test whether our modifications to the inductive bias lead to more effective acquisition of multi-level semantics. Consequently, we evaluate on a series of tasks that measure correspondence between representation similarity and human judgements on the word and sentence levels. As we want to focus on the effects of our unsupervised pre-training regime, we utilise zeroshot evaluation. On the word level, we use Simlex-999 Hill et al. () and WordSim-S/R Agirre et al. (). All tasks measure semantic similarity, but do so on differing axes. To understand this, we must first qualify the difference between semantic similarity and relatedness. Semantic similarity measures the extent to which entities act the same way, e.g. ‘running’ and ‘singing’ are similar as they share the role verb. Semantic relatedness measures conceptual association, e.g. ‘singing’ and ‘fame’ may be highly related. Simlex measures similarity at the exclusion of relatedness, Wordsim S measures similarity without penalising relatedness, and Wordsim R measures relatedness.
On the sentence level we utilise STS-12 through 16 Agirre et al. (, , , , ), the STS-B Cer et al. (), SICK-R Marelli et al. () and SemRel Ousidhoum et al. (). Each measures slightly different aspects of sentential semantics. We use an embedding dimension of 256 for all experiments; see A for additional hyperparameters.

## 5 Comparison to Self-StrAE

**Table 3: Banyan compared with unstructured baselines on sentence level semantics. Higher is better. Results represent the average across four random initialisations. Only columns where there is no standard deviation overlap between models are bolded.**
| Model | STS-12 | STS-13 | STS-14 | STS-15 | STS-16 | STS-B | SICK-R | SemRel | Score |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GloVe | 31.61 $\pm$ 0.31 | 21.69 $\pm$ 0.12 | 27.37 $\pm$ 0.10 | 40.42 $\pm$ 0.09 | 29.27 $\pm$ 0.12 | 28.25 $\pm$ 0.08 | 50.20 $\pm$ 0.25 | 41.20 $\pm$ 0.43 | 33.75 $\pm$ 0.04 |
| + stopword rm | 39.00 $\pm$ 0.57 | 41.61 $\pm$ 0.19 | 39.31 $\pm$ 0.18 | 51.06 $\pm$ 0.35 | 45.14 $\pm$ 0.14 | 48.40 $\pm$ 0.07 | 52.80 $\pm$ 0.04 | 42.37 $\pm$ 0.13 | 44.96 $\pm$ 0.10 |
| RoBERTa | 42.77 $\pm$ 1.27 | 51.70 $\pm$ 1.30 | 45.67 $\pm$ 1.42 | 63.97 $\pm$ 0.81 | 59.60 $\pm$ 0.61 | 39.97 $\pm$ 0.95 | 52.93 $\pm$ 0.23 | 52.73 $\pm$ 0.58 | 51.08 $\pm$ 0.61 |
| + SimCSE | 50.63 $\pm$ 1.45 | 62.23 $\pm$ 2.51 | 54.17 $\pm$ 2.10 | 68.77 $\pm$ 3.00 | 66.67 $\pm$ 1.40 | 53.53 $\pm$ 1.18 | 56.87 $\pm$ 1.16 | 59.27 $\pm$ 0.93 | 59.02 $\pm$ 1.45 |
| Banyan | 51.20 $\pm$ 0.007 | 69.10 $\pm$ 0.002 | 63.20 $\pm$ 0.004 | 73.20 $\pm$ 0.002 | 66.60 $\pm$ 0.002 | 61.50 $\pm$ 0.002 | 55.50 $\pm$ 0.003 | 61.60 $\pm$ 0.002 | 62.70 $\pm$ 0.001 |

**Table 4: Banyan compared with unstructured baselines on lexical semantics. Higher is better. Results represents the average across four random initialisations. Only columns where there is no standard deviation overlap between models are bolded.**
| Model | Simlex | Wordsim-S | Wordsim-R | Score |
| --- | --- | --- | --- | --- |
| GloVe | 27.47 $\pm$ 0.25 | 62.53 $\pm$ 0.42 | 51.00 $\pm$ 0.56 | 47.00 $\pm$ 0.38 |
| RoBERTa | 29.23 $\pm$ 0.64 | 61.97 $\pm$ 2.38 | 46.00 $\pm$ 2.13 | 45.73 $\pm$ 1.71 |
| Banyan | 16.57 $\pm$ 0.02 | 63.25 $\pm$ 0.03 | 69.00 $\pm$ 0.01 | 49.61 $\pm$ 0.02 |

For our first set of experiments, we compare Banyan against Self-StrAE Opper et al. (). As the strength of the original model was on data efficiency, we adopt the same pre-training regime of $\approx$10M tokens subsampled from the human-written section of English Wikipedia.

Our first test involves simply changing the structure type from sentential trees to entangled trees. Results are in Tables 1 and 2 and Fig. 1(a). Across both the word and the sentence level, Banyan, which utilises entangled trees, consistently outperforms the original Self-StrAE. However, it should be noted that a fundamental difference between entangled trees and the standard tree encoding setup is that the latter introduces false negatives in the contrastive loss, as each repeated instances of a particular node in the batch is treated as a negative to the other instances. Meanwhile, when using entangled trees, because we are dealing with the set of all nodes such false negatives cannot arise.

To assess whether it is entangled trees themselves rather than the absence of false negatives causing the improvement, a simple test employs a pre-training objective where false negatives have no effect. The cross entropy objective $(\mathcal{L}_{\text{CE}})$ where the model simply reconstructs its input tokens, suits this purpose. Initial attempts simply using $\mathcal{L}_{\text{CE}}$ with Self-StrAE led to very unstable performance between seeds (see Appendix B). To remedy this, we switched to the new message passing paradigm described in § 3, utilising diagonal composition and decomposition functions. We can now compare entangled trees with sentential trees and avoid potential issues with false negatives as a confounding factor. As before, we find entangled trees yield near universal benefits across the board. The only difference between the two models is that Banyan uses entangled trees, and while both structural variants show improvements from diagonal message passing functions, Banyan still retains an edge, indicative of the utility of entangled trees. As a final ablation, we also compare Banyan and the original Self-StrAE using the new functions with the contrastive loss (see Fig. 1(b)). While this underperforms compared to the cross entropy objective, we again see that Banyan shows improved performance, cementing the benefits of entangled trees.

Beyond the benefits afforded by entangled trees, the change to diagonal functions brings two key benefits. Firstly, it helps avoid the contrastive loss which, while conceptually elegant, in practice comes with a host of problems (see Robinson et al. ()). Secondly, we find that with the new functions, performance is either maintained or improved by increasing channel number $K$ and correspondingly decreasing channel size $U$ (see Appendix C for ablations). As the number of parameters is directly proportional to $U$, this fact means that we can reduce the number of non-embedding parameters to 14 (our chosen setting has $K=128$ and $U=2$); a further order of magnitude reduction compared with the already minimal Self-StrAE. We note that this is the same pattern observed by Opper and Siddharth (), however, unlike their findings we are able to achieve consistent good performance between seeds.

Summary: We make three changes on top of Self-StrAE:
(i) learning over entangled trees,
(ii) message passing with diagonal functions, and
(iii) switching from the contrastive to the cross-entropy objective.
Put together, these changes result in a significantly more effective model.

## 6 Comparison to Unstructured Baselines

Next, we compare to baselines that do not have an inductive bias towards explicit hierarchy. For these we chose GloVe Pennington et al. (), a classic yet robust baseline which encodes the distributional hypothesis, and RoBERTa Liu et al. () a transformer encoder which may encode hierarchy implicitly but does not do so explicitly. For RoBERTa we chose the ‘medium’ configuration as defined by Turc et al. (). Further hyperparameters can be found in Appendix A. We also include augmentations to the baselines to make them more challenging. For GloVe on the sentence level, we report results both for simple averaging, and averaging with stopwords removed following Reimers and Gurevych ()—intended to produce better sentence embeddings. For RoBERTa we report results using both the standard model, and after an extra round of SimCSE training Gao et al. (), as a further STS baseline. In both cases, we generate sentence embeddings through mean pooling. To produce static embeddings from RoBERTato use in lexical evaluation, we follow Bommasani et al. () and average the contextualised representations of all occurrences of the word in the training set. Finally, since these baselines were not designed with the data efficient focus of Self-StrAE we train them on the 100M token Wiki-103 benchmark dataset Merity et al. ().

Results are shown in Tables 3 and 4. On the sentence level Banyan is only rivalled by RoBERTa medium with SimCSE augmentation, and even here it retains an edge despite being a *far* smaller model (see Table 6) and having less pre-training data. On the word level, the picture is more mixed. All models learn to capture semantic similarity well, however, both RoBERTa and GloVe do so at the expense of semantic relatedness—indicated by their higher Simlex score, which penalises models that capture relatedness. Banyan  on the other hand, captures both similarity and relatedness well, with its performance on relatedness surpassing performance on similarity. We believe that Banyan’s strong ability to capture relatedness is a direct consequence of its inductive bias. Related concepts might often be siblings in the tree structure, and so if Banyan is capturing informative structure, it follows that it would learn lexical relatedness well.

Summary: We compare Banyan against unstructured baselines to assess the utility of modelling explicit interconnected tree structure. We find that despite its minimal parameterisation Banyan manages to be performant, demonstrating the utility of the inductive biases employed.

## 7 Performance Across Languages

**Table 5: Banyan performance on sentence level semantic relatedness across different languages. Higher is better. Results for Banyan are the average across four random initialisations. Results for competition and XLM-R are taken from Ousidhoum et al. ().**
| Model | Banyan | Self-StrAE | RoBERTa (M) | XLM-R |
| --- | --- | --- | --- | --- |
| Params | 14 | 1072 | $\approx$10M | $\approx$85M |

## 8 Efficiency

**Table 6: Number of parameters for our model and the baselines. Following convention we exclude the embedding matrix and LM head from the count.**
| Model | Banyan | Self-StrAE | RoBERTa (M) | XLM-R |
| --- | --- | --- | --- | --- |
| Params | 14 | 1072 | $\approx$10M | $\approx$85M |

Banyan is a significantly more efficient model than Self-StrAE for two primary reasons. Firstly, the diagonal message passing functions are both easier to compute and have fewer parameters than standard weight matrices ($2U$ rather than $2U\times U$). This means a further order of magnitude reduction compared with the already minimal Self-StrAE (see Table 6). Secondly, and crucially, Banyan utilises entangled trees where the number of nodes grows at a significantly reduced rate with batch size compared with sentential trees (see Fig. 2). This is because the number of reused constituents also grows as batch size increases, and entangled trees capture the set of all constituents, which consequently does not grow as drastically. In practical terms because entangled trees requires fewer nodes, and each node requires two distinct embeddings ($\bar{e}$ and $\underline{e}$) to be held for it, reducing their number leads to radical improvements in memory efficiency.

Figure: Figure 2: Total number of nodes in entangled trees vs sentential trees as batch size grows
Refer to caption: extracted/5753996/nnodes.png

## 9 Related Work

Recursive Neural Networks (RvNNs): Similar to RNNs but traversing hierarchically over a tree rather than a sequence. First popularised by Socher et al. (, ) and have inspired numerous successor frameworks, to which Self-StrAE and therefore by extension Banyan, also belong.

Learning Structure: Prior work on learning structure tends to fall into two categories. The first centres on language modelling, where an unsupervised pre-training objective, usually some form of masked language modelling, is used to score a parsing module Drozdov et al. (, ); Hu et al. (, ). Common to these approaches is using a chart parser as the unsupervised module, which has proved successful though highly memory intensive. The second regards compositional generalisation tasks, commonly ListOps Nangia and Bowman (), where learning the correct parse for given input constitutes success, and where non-explicitly structured models commonly fail. Recent models either use beam search Ray Chowdhury and Caragea () or reinforcement learning Havrylov et al. (), to learn parses, while the inner traversal is handled by an RvNN module. Here, we use entangled agglomerative composition based on representation similarity as it is more scalable/less resource intensive than other methods and, for representation learning with structure, remarkably effective.

Semantic Representations: The distributional hypothesis Harris () posits that the meaning of a word is derived from the company it keeps (i.e. context dictates meaning). This principle has been leveraged by Word2Vec Mikolov et al. () and later GloVe Pennington et al. () in order to learn static word embeddings, where a pre-defined window around the word is utilised to define the context for prediction. One issue with these approaches is that they don’t capture change in meaning from context, which led to the development of transformer embeddings. Though on the face of it these models produce fairly mediocre embeddings Reimers and Gurevych (), they can be enhanced by utilising contrastive learning Gao et al. (). Considering what structure enables, one can see its benefits for learning both distributional and contextual semantics. On the distributional side, structure provides a more nuanced way to determine the context window, as the nearest structural neighbours can have the greatest influence on the context window. On the contextual side, ambiguity can be resolved by taking the representation of the node which covers the contextualised meaning.

Representation Learning with Structure: An underexplored area, prior work focuses on using dependency parses to define the context window for learning lexical embeddings Levy and Goldberg (); Vashishth et al. (). More recently, Opper et al. () use constituency parses to learn representations for multiple levels of hierarchy (from word to sentence). This is extended so representations define their own structure, removing the need for an external parser. This model, the Self-StrAE  is the basis for our investigations here.

## 10 Discussion and Future Work

We present, Banyan  which use explicit hierarchical structure in order to learn representations. We identify two core features which make the model radically more effective than the original Self-StrAE. Firstly, that modelling entangled trees rather than individual sentential trees is more beneficial for representation learning, as well as considerably more memory efficient. Secondly, we find that lessons from the recent RNN resurgence can also be applied in the recursive case, with diagonal message passing functions leading to better representations. Banyan is significantly smaller than the majority of baselines we used in this paper (see Table 6), but despite this is able to compete above its weight. We believe this demonstrates the utility of hierarchy when equipped with the appropriate inductive biases. For future work, it would be interesting to see the effect of more flexible architectures. The recent RNN/SSM literature already provides a pathway towards scaling, and it would be interesting to see what an increased parameter budget could achieve. We refrain from this question here in order to keep the focus on the inductive bias. Secondly, our merge algorithm (agglomerative clustering) despite its efficacy, remains very simple. It would be interesting to see if it would possible to integrate self-attention, as it already models a fully connected graph between inputs and all that is required is to add a discretisation step.

## 11 Limitations

The same limitations which apply to the original Self-StrAE also apply to Banyan. It has extremely few parameters, and makes locally greedy decisions regarding merge. It is highly likely that a more sophisticated induction algorithm and a more powerful model would lead to downstream performance. The reason we adopt this setting is because we want to isolate the effects of the inductive biases we use, and we know that performance for StrAE style models has proven to be directly dependent on structure Opper et al. (). We believe that we have demonstrated the utility of the biases advocated for in this paper: explicit hierarchy, specifically, entangled trees. However, our aim is to conduct basic research, not to directly compete with SoTA.

## 12 Ethics Statement

In this paper, we investigate what inductive biases enable more resource efficient representation learning. We believe this can provide considerable benefits in making research more accessible, both in terms of compute and language coverage.

## Appendix A Hyperparameters

We trained Self-StrAE and Banyan for 15 epochs (circa 15k steps- sufficient for convergence) using the Adam optimizer Kingma and Ba () with a learning rate of 1e-3 for cross entropy and 1e-4 for contrastive loss, using a batch size of 512. We applied dropout of 0.2 on the embeddings and 0.1 on the composition and decomposition functions. The temperature hyper-parameter for the contrastive objective was set to 0.2 and the r value 0.1. These settings were obtained by a grid search over the values r= 1.0, 0.1, 0.01, batch size = 128, 256, 512, 768, $\tau$ = 0.2, 0.4, 0.6, 0.8 and learning rate 1e-3, 5e-4, and 1e-4. To process the graphs we used DGL Wang et al. (). The GloVe baseline was trained for 15 epochs with a learning rate of 1e-3, and a window size of 10. RoBERTa medium was trained for 200,000 steps, with a 10% of which were used for warmup, a learning rate of 5e-5 and a linear schedule. We used relative key-query positional embeddings. The configuration for RoBERTa medium is 8 layers, 8 attention heads and 2048 dimensional feedforward layers. We used the Transformers library to train the model Wolf et al. (). For SimCSE training, we used the default parameters and implementation for unsupervised RoBERTa training from Gao et al. (). Training was performed using a single A40 (40gb GPU).

## Appendix B Cross Entropy Instability

Without our changed functions, the cross entropy objective demonstrated significant instability between seeds. We show these results in Tables 8 and 7.

**Table 7: Cross entropy instability with old functions. Results are the average across four random seeds.**
| Model | STS-12 | STS-13 | STS-14 | STS-15 | STS-16 | STS-B | SICK-R | SemRel | Score |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Banyan old functions | 29.4 $\pm$ 19.4 | 42.2 $\pm$ 26.6 | 35.8 $\pm$ 26.08 | 45.8 $\pm$ 28.6 | 43.2 $\pm$ 23.3 | 36.5 $\pm$ 25.4 | 41.9 $\pm$ 19.05 | 39.8 $\pm$ 24.5 | 39.33 $\pm$ 24.0 |

**Table 8: Cross entropy instability with old functions. Results are the average across four random seeds.**
| Model | Simlex | Wordsim-S | Wordsim-R | Score |
| --- | --- | --- | --- | --- |
| Banyan old functions | 11.6 $\pm$ 10.1 | 32.4 $\pm$ 38.8 | 34.5 $\pm$ 30.0 | 26.2 $\pm$ 26.2 |

## Appendix C The k and u balance

The change to diagonal composition functions allows us to reduce the number of total parameters while maintaining performance. This is because the number of parameters is directly proportional to channel size u. We show ablations for this finding in Table 9.

**Table 9: Performance Depending on k and u values using new functions. Scores are the average of four random seeds.**
| k | u | Lex Score | STS Score |
| --- | --- | --- | --- |
| 4 | 64 | 42.9 $\pm$ 0.01 | 43.5 $\pm$ 0.04 |
| 8 | 32 | 43.2 $\pm$ 0.02 | 48.6 $\pm$ 0.01 |
| 16 | 16 | 47.02 $\pm$ 0.03 | 62.2 $\pm$ 0.01 |
| 32 | 8 | 49.2 $\pm$ 0.01 | 62.9 $\pm$ 0.01 |
| 64 | 4 | 48.7 $\pm$ 0.01 | 62.9 $\pm$ 0.01 |
| 128 | 2 | 49.61 $\pm$ 0.02 | 62.7 $\pm$ 0.001 |
| 256 | 1 | 48.7 $\pm$ 0.01 | 62.9 $\pm$ 0.001 |

## Appendix D Data Sources

For Afrikaans, Spanish and Amharic we obtained corpora from Leipzig Corpora Collection(^1^11For both Spanish, Afrikaans and Hindi we selected the mixed corpus and took a uniform subsample to reduce size to the requisite scale if necessary.). Meanwhile for Amharic we utilised a MiT licenced pre-training set of 1 million sequences available on the Huggingface hub at this [link.](https://huggingface.co/datasets/surafelkindu/Amharic_corpus/tree/main) We utilise a pre-trained BPE tokenizer for each language from the BPEMB Python package Heinzerling and Strube (). Though the package also provides pre-trained embeddings, we solely use the tokenizer and learn embeddings from scratch.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please refer to the comments in the weaknesses section.
soundness: 2
strengths: - The proposed new strategies for constructing tree structures are intuitive
- The proposed framework is parameter efficient and easy to learn.
summary: This paper presents a strategy of representation learning by utilizing structural information discovered during learning. The proposed work is built upon a prior work with two specific pieces of improvement on building the structures and propagating the information during representation learning. Empirical evaluation was performed on multiple NLP tasks.
weaknesses: - The writing of this paper can be significantly improved. There are some inconsistent statements and inconsistent terminology, which should be easily fixed after proofreading. For example, the paper uses both "under resourced languages" and "under represented languages", which I think it should be "low-resource languages". 
- There are some unsupported claims, for example, in the first section, the claim "It is thought that this principle lets humans generation ..." should be supported with references from prior work. 
- There are also some technical details that are unclear in the paper. For example, in line 204 - 207, what are criteria of entangling different trees together, and what are the benefits of entangling subtrees together? 
- There is a minor issue with the citation format; it should be, for example, (Tai et al., 2015).
- There are some existing works along the line of upward and downward along a tree structure to learn representation, which are not discussed in this paper. For example, Ji and Eisenstein, One Vector is Not Enough: Entity-Augmented Distributed Semantics for Discourse Relations, 2015. 
- The experiments are not sufficient; for example, some recent large language models are not included in the comparison.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: There are composition and decomposition (corresponding up and down embeddings), what are the embedding used for sentence-level and word-level?
soundness: 3
strengths: 1. The proposed method is a novel approach for semantic representation. Tree structure is inject into the representation.

2. According to the experiments results (Table 1 and Table 2),  Banyan achieves competitive performance with transformer based baselines on sentence level and work level. The results are also much better than previous baseline (Self-StrAE).

3. There are a clear ablations study on the effect of different modeling changes. It is helpful for others to understand the method.
summary: In this work, the authors present a method called "Banyan" that can learn the semantic representations with hierarchical structure. It has two main improvements (entangled tree structure and diagonalized message passing functions) comparing to SELF-STRAE. According to their experimental results, Banyan achieves competitive performance with transformer based baselines. It also shows the low cost and efficiency.
weaknesses: 1. For this proposed semantic representations, both the structure and representation are learned. The first  embedding tokens are used to determine which tokens to merge and in what order. The initial token embeddings have a large impact on the proposed method. It is a little unclear about this part (are they randomly initialized? Are they updated during model training? etc) 

2. Only 14 non-embedding parameters are used in the proposed method. It could also limit the ability the proposed model. It would be great if the proposed method can be used in the transformer-based embedding in the future.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Regarding low resource language evaluation, the authors only used four languages, while there are many other languages in the released semeval dataset, is there any cherry-picking on the languages evaluated ? How will the model perform on other low resource languages ?
2. The authors trained Banyan on datasets from Leipzig Corpora Collection, is there any overlap between the training corpora and the testing dataset ? and if the baseline methods such as XLM-R are also pre-trained on these corpora, how will the models perform ?
soundness: 3
strengths: This proposed method is technically solid, and makes substantial improvement compared with existing methods.
The proposed method is highly efficient and lightweight, especially when compared with Large language models.
summary: The paper presents Banyan, an efficient and lightweight framework for representation learning on low resource languages. The method has been evaluated in common English tests as well as a series of low-resource languages. This method demonstrates great performance as well as remarkable efficiency. While the technical innovations are impressive, there are still several remaining concerns.
weaknesses: Some evaluation parts still need further validation. The details are described in questions

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 2
questions: Previous approaches used 1 tree per sequence. Banyan uses one tree for all sequences sharing the same tokens in a batch.
Some analysis of scale would be nice to complement Figure 4 in the paper. For instance, how many trees do you usually have in an entire batch? How many (sub-)sequences are represented in 1 tree?
soundness: 3
strengths: This paper introduces a novel recursive model that learns textual representations and its learning mechanism.
The proposed architecture is novel and seems promising as it yields good results when compared to other more classical methods.
In addition, the proposed method is very efficient: it requires very little training and has only 14 non-embedding parameters.
summary: This paper introduces a new recursive graph neural network for learning text representations in low-resource languages: Banyan (like the tree). This model extends previous work by building nested trees over sequences that share the same tokens. In Banyan the same tokens will have the same tree node, even if it comes from different sequences. For scalability reasons, the trees are constructed from a batch of samples rather than from an entire dataset. Embeddings are learned from a simplified message passing algorithm that traverse the trees both in bottom-up and top-down directions.
Having nested trees provides multiple advantages, notably the reduction of duplicated nodes and multiple context representations within the same node (in downward embeddings).
These advantages translate to strong semantic representations in both English (when compared to RoBERTa) and lower-resourced languages (when compared to XLM-R, Llama 3.1, Mistral).
weaknesses: The task being tackled is not clear from the abstract or the introduction. The motivation is well described (learning powerful text representations for under-resourced languages), but the task used to evaluate the Banyan model (Semantic Textual Similarity - STS task) is not described. The paper could gain clarity by mentioning the task earlier.

Evaluation is based on cosine similarity between sentences compared to human judgment. The paper would attract greater significance if the representations from the proposed model were tested in targeted applications such as sentiment classification or machine translation.

Meta-review: This paper proposes a recursive autoencoder for learning text representations. The method works by recursively merging adjacent embeddings from bottom up to build a tree, and then top down splitting to reconstruct leaf embeddings. Experiments on semantic text similarity demonstrate the effectiveness of the proposed approach.

Strengths:
1. The method itself is simple and provides an alternative to existing representation learning methods by focusing on structures.

Weaknesses:
1. The evaluation is mainly conducted using semantic text similarity but not directly downstream applications such as machine translation or sentiment analysis, or better yet, the GLUE benchmark which is often used for evaluating representation learning methods.

Overall, while this is a very interesting method, the evaluation itself is weak. I think this paper can be significantly improved by evaluating it on downstream applications. I'm recommending rejection for the current version, but I wouldn't mind if the paper gets accepted.

Decision: Reject