Banyan: Improved Representation Learning with Explicit Structure
Authors: Mattia Opper
Siddharth N
Venue: ICML 2025 poster (2025)
Keywords: Representation Learning
Structure
Semantics
Syntax
Induction
Composition
Primary area: general_machine_learning->representation_learning
TL;DR: We investigate inductive biases for learning representations with explicit structure, and are able to significantly improve performance while making our model even simpler.

Abstract: We present Banyan, a model that efficiently learns semantic representations by leveraging explicit hierarchical structure. While transformers excel at scale, they struggle in low-resource settings. Conversely recent structured models have shown promise as efficient learners, but lack performance. Banyan bridges this gap with two key innovations: an entangled hierarchical tree structure and diagonalized message passing, enabling it to outperform larger transformer models with just 14 non-embedding parameters. It excels in low-resource settings, offering a viable alternative for under-represented languages and highlighting its potential for efficient, interpretable NLP in resource-constrained environments.

Banyan: Improved Representation Learning with Explicit Structure
Mattia Opper'
N. Siddharth
Abstract
ing the need store individual meanings. It allows humans to learn efficiently from relatively little data and enables effective and efficient learners (Lake et al., 2016; Ito et al., 2022; Wiedemer et al., 2023).
We present Banyan, model that efficiently learns semantic representations by leveraging explicit hierarchical structure. While transformers ex- cel at scale, they struggle in low-resource set- tings. Conversely recent structured models have shown promise efficient learners, but lack per- formance. Banyan bridges this gap with two key innovations: entangled hierarchical tree struc- ture and diagonalised message passing, enabling it to outperform larger transformer models with just 14 non-embed parameters. It excels in low-resource settings, offering a viable alternative for under-represented languages and highlighting its potential for efficient, interpretable NLP in resource-constraine environments.
To incorporate such inductive biases into models, the tradi- tional information flow within neural networks needs alter- ing. Instead of relying on implicit processing alone, models must learn representations for atomic components and an explicit computation graph that dictates how these compo- nents combine. Additionally, models must learn functions to govern information flow through this graph. Such ap- proaches have demonstrated improved language modelling perplexity at cognitively plausible scales (Hu et al., 2021; 2022), better systematic generalisation (Sartran et al., 2022: Murty et al., 2023), and, especially relevant here, enhanced efficiency in acquiring semantics (Opper et al., 2023b).
The SELF-STRAA AE model (Opper et al., 2023b) learns rep- resentations that explicitly model compositional seman- tics, and achieves promising performance while requiring minimal resources, both in terms of data and model size. It opened the door to exploring more compute-efficient solutions-parricularty valuable for low-resource languages where scaling is often infeasible. However, while innovative, it still falls short compared to large-scale pre-trained trans- formers, even in languages outside standard pre-training corpora. Here, we introduce BANYAN, a model which signif- icantly outperforms SELF-STRAE while achieving greater resource efficiency. Our approach involves modifying the structural optimisation process induce an entangled graph that models global relations between nodes and employs message passing mechanism using diagonal functions, reducing parameters while enhancing expressiveness.
1. Introduction
Semantic representations are foundational for various NLP applications, such as retrieval-augmented generation (RAG) (Lewis al., 2020), question answering, and summarisa- tion (Abdalla et al., 2023; Wang et al., 2022). They are also crucial for clustering and organising textual data when labelled training data is unavailable. Typically, such repre- sentations are generated by large-scale transformer models (Vaswani et al., 2017); highly effective but needing substan- tial amounts of data and computational resources train.
An alternative approach draws inspiration from lin- guistics and cognitive science, incorporating structured compositions-4 principle that posits that understanding the semantics of a whole requires knowing the meanings of its parts and the structural rules that determine how they assemble (Chomsky, 1956; Crain & Nakayama, 1987; Pal- lier al., 1; de Marneffe et al., 2006). This principle is highly efficient because novel utterances can be decomposed into familiar components using systematic rules, minimis-
BANYAN, achieves performance comparable to transformer- based baselines and represents a low-cost, viable alterna- tive to transformers for producing representations in low- resource languages, measured by semantic textual similar- ity (STS) tasks. By leveraging cognitively inspired inductive biases, our work enables semantic representation learning that rivals or surpasses large-scale pre-trained LLMs-using only 14 non-embedding parameters. Our model, BANYAN, offers a new direction for efficient and effective semantic understanding in resource-constrained environments.
School of Informatics, University of Edinburgh, UK. Corre- spondence to: Mattia Opper <m.opper add.uckkdd.... <m..ppor@ed..c....... N. Siddharth <n.siddharth @ ad.ac.uk>.
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'Code available at:

---

Banyan: Improved Representation Learning with Explicit Structure
2. Background and Related Work
Pagliardini al. (2017) realised that if the average of the embeddings was going to be used to create the sentence representation, it makes sense to optimise it directly. Conse- quently, they modified the pre-training objective in order to have the average predict a missing word from a sentence. At scale this proved tremendously effective. However, despite offering substantial improvements, all these methods require scale and more importantly do not tackle the central limita- tion of word embeddings the inability to handle changes in meaning dependent on context.
Banyan is a graph neural network, specifically a recursive neural network (RvNN) that learns both structure and repre- sentations. We unpack these components below.
Like regular re- current neural networks (RNNs), RvNNs process data by repeatedly applying a function to update their state in se- quence. However, instead of relying on temporal ordering (like the sequence of words in a sentence), RvNNs use hierar- chical structures, often provided as input-moss commonly as binary tree-and can be applied either bottom-up (from leaves to root) top-down (root to leaves). They were popularised in the deep learning era by Socher et al. (2011; 2013), inspiring many successor models that vary in how they define the recursive function, including Tree-LSTMs (Tai et al., 2015) and IORNN (Le& & Zuidema, 2014).
Recursive Neural Networks (RvNNs):
On the other hand, transformers, through self-attention, are able to represent contextualised meanings. However, early encoder-only transformer models produced poor represen- tations (Reimers & Gurevych, 2019), especially compared to the more sophisticated approaches based on word em- beddings. This was largely due to the anisotropy issue (Godey et al., 2024), which required the development of techniques using contrastive fine-tuning (Gao et al., 2021) to finally remedy. These approaches eventually surpassed word embeddings, and have become the method of choice for producing semantic representations. However, they still rely heavily on scale for success, as contrastive refinement is final fine-tuning step applied to pre-trained models rather than directly incorporated within pre-training.
Learning Structure: RvNNs often require structural input, which limits their flexibility since this structure may not always be available or easily obtainable. To address this, re- searchers have developed methods to induce structure within the model during recursive computation; using differentiable chart parsing (Drozdov et al., 2019; 2020; Hu et al., 2021; 2022), beam search (Ray Chowdhury & Caragea, 2023), continuous relaxation (Chowdhury & Caragea, 2021; Sou- los et al., 2024), and reinforcement learning (Havrylov et al., 2019). However, these methods can struggle with memory issues and require careful tuning of hyperparameters. Here, we adopt method from Opper et al. (2023b) that uses rep- resentation similarity to determine how nodes should be merged during computation, which is both computationally efficient and surprisingly effective.
Semantic Representation Learning through Structure: Transformer embeddings have become more successful than static word embeddings due to their ability to handle vary- ing contextual influences. Unlike attention mechanisms in transformers, which route information based on token rela- tionships, some approaches use explicit graphs or structures, such as dependency (Levy & Goldberg, 2014; Vashishth et al., 2019) or constituency parses (Pham et al., 2015), to determine the focus of context windows. These models have the potential to bridge the gap between the efficiency of word embeddings and the contextualisation offered by transformers. This is because the discrete structure provides input specific routing order which dictates interactions between atoms and consequently determines their influence on higher level representations allowing for more flexibil- ity than simple averaging. Most related to our work, Opper et al. (2023b) introduce two models. StrAE, which use con- stituency parsers to learn sentence-level embeddings along- side word embeddings, and SELF-STRAE, which learns its own structure using representations. This latter model, SERFES serves the foundation for BANYAN and is described next.
Semantic Representations of Text: Systems like Werd2Vec (Mikolov al., 2013) and Glo (Pennington et al., 2014) use the distributional hypothesis (Harris, 1954) to model word semantics, which posits that words are de- fined by the context in which they appear. To learn represen- tations, these models use a fixed context window and predict a missing word in a sequence. While initially effective, this approach is limited because representations for higher level objects (i.e. phrases, sentences etc.) are computed by sim- ply averaging word embeddings. However, some notable follow on works attempted to improve upon this, Arora et al. (2017); Ethayarajh (2018) introduce a more sophisticated form of taking an average over word embeddings by using SVD, while Rücklé et al. (2018) use the power mean. These approaches yielded improvements, but relied on represen- tations pre-trained at scale which limited their applicability to specialised domains or low resource languages. Wieting et al. (2021) attempt to refine the sentence representation through the use of paraphrase corpora, looking to increase alignment between language pairs, again improving perfor- mance, but requiring large scale parallel corpora. Finally,
3. Preliminary: Self-StrAE
SELF-STRAB involves three main components that act over sequence of tokens w (w,>>_i (a) an algorithm for merging tokens based on their similarity, (b) functions for
composition and decomposition of embeddings, and (c) an
2

---

Banyan: Improved Representation Learning with Explicit Structure
contrastive objective is formulated over a batch of sentences. As up and down trees are structurally identical (modulo edge reversal), it draws together an embedding and its dual on the other tree, while pushing away all other embeddings across the batch, using cosine similarity. Denoting pair- wise similarity matrix A E IR M between up and down embeddings over M nodes in the batch, the objective is: (co(e,e) 2M Di=1 log (o, (A,)) oy (A.4)) with ay (-) the tempered softmax over the unspecified (-) dimension.
root smbedding sbared
ng
4. Banyan
dougSacte
Homer
dkoughonts
Given their construction, the upward embeddings are always locally-cottextual: only encapsulating the context of the span they cover. For example, in Figure 1, the upward embedding for the span "ate doughnuts" is always the same regardless of context, matter who did the eating. In contrast, downward embeddings are always globally- contextual: necessarily encapsulating surrounding context, being decomposed from embeddings of larger spans. In our example, this implies multiple downward embeddings e", one for each y E {"Lisa", "Homer". ....e.. }. embeddings requires amortisation over these differences to ensure meaning resolves over all these contexts.
Figure 1: Self-StrAE operation. Red lines indicate cosine similarity. Shared colours imply shared parameters.
objective that leverages both the induced structure and em- beddings. While full details are available in Opper et al. (2023b), we provide a brief overview to establish context for our model development ($ 4).
At high level, SE learns representations that define their own structure while being shaped by it. Starting with an initial embedding matrix Qg. tokens are merged into single embeddings using a composition function C2 based on best cosine similarity (e.g., see Figure 1). This process reduces the sequence to a single root embedding while cap- turing semantic relationships. The resulting merge history forms a binary tree structure, over which the model then operates in reverse by decomposing embeddings at each node using a decomposition function De. to reconstruct the leaf embeddings. Optionally, it can further predict tokens (w,)) using a dembedding function Ar. Figure 1 illustrates the autoencoding process. During training. tokens that are frequently merged together develop correlated representa- tions, leading the model learn meaningful compositional semanties. This results in embeddings that reflect both their own structure and the semantic patterns they encode.
Learning effective
4.1. From trees to entangled trees
Algorithm 1 BANYAN: Entangled Compose
Input: Global frontier ((8m: en)) compose (o), concat (o), similarity CSIM(e, e')
1: A ((sn, en))
initalise frontier
2: (V.E) t (00 R)
> Initialise graph
3: while 3i : 8; O Sit1 E. Vo
4: i + arg max; CSIM (e;, ei+1)
D. locate closest pair
5: Ep = o(ei,,
L compose
V t fl o 01)+
E E EU(P (ufp~it it p (i* + 1)}
More formally, one denotes tokens as the vertices E V in a V-simplex for vocabulary size V, and note that the models generates two sets of embeddings-on going up (e: leaves + root) and one coming down (e: root - leaves). The embeddings are viewed e E R allowing the composition and decomposition functions to act indepen- dently over K channels, and be defined as
8:
I + (i (8])
(Sxr, 81++
locate all occurrences this pair
9:
A + A\ fE
A As+1k
D delete occurrences from those locations
10:
A+
+ A AUy ((8
o 11,+1, ep)}
D insert composition into those locations
return: Graph (V,g)
Co (e,, = HCAT(E,, E 1 + d.
4 2U E UUU
(1)
De(e)) = HSPLIT(C O + 0),
(2)
We wish to have composition embeddings amortise over all possible contexts, and simultaneously, all decompositions embeddings to resolve to the same thing. The representation of an entity "Lisa" should encapsulate everything she could possibly eat. Simultancously, the average of everything she could eat we should get back to "Lisa". Self-StrAE does not explicitly model this behaviour in its structure. Decomposition embeddings of the same entity only interact
To learn this model from data, Opper et al. (2023b) derive two objectives. The first is straightforward cross-entropy over reconstructed tokens, for sentence w and prediction w as (ce(w, w) = N Σx w, log Wr. This however, places little constraint on the intermediate nodes in the hi- erarchical model. To address this an alternate structural
3

---

Banyan: Improved Representation Learning with Explicit Structure
born
some
are
born
to
sweet
delight
born
born
born
endless
night
(a) upward composition
(b) downward decomposition
Figure 3: Upward and downward traversals for section of the entangled tree from Figure 2.
agglomerative clustering employed in SELF-STRAE here we employ global frontier spanning all leaf nodes across the given data. The key differences to the prior methods are mainly to do with constructing a graph jointly with progressing the frontier and ensuring that new nodes are never duplicated, for which employ node identity 8n in addition to the node embedding en.
every and
every
night
Entangling
sweet
delight
Following the entangling of trees described, the model proceeds a similar vein to SELF- STRAE, by composing upwards from leaves to roots (multi- ple roots corresponding to multiple trees), and then decom- posing downwards back the leaves. With entangled trees, while traversing upwards each node is always composed from the same two children, but on the way back down, things are different as each separate context for given node provides a different downward embedding. This is shown in Figure focussing on a subgraph of the entangled tree from Figure 2(right). Note that node in question (in blue) corresponds the span "some born to", and has downward embeddings that incorporate context both from "endless night" and "sweet delight". This is exactly de- sired, as BANYAN allows explicit aggregation to derive the downward embedding that resolves over the contexts. For any upward embedding é whose span occurs in different contexts E V, the corresponding downward embedding is derived by simply averaging over the different contextual down embeddings; i.e., 1/D1
Incorporating context:
some
are
born to endless night
every morn
and every
Figure Entangled trees: Example of disjoint trees ing transformed into an entangled tree. Internal functions (C4, De....) are elided avoid clutter.
when we calculate the loss. On top of this, because the loss taken over the batch, they are actually treated as false negatives to each other. Even though they are terms that ought not be pushed away, the objective ask them to be.
Our innovation is to address both these issues by formulating the process in terms of entangled rees-wheree entangling describes the transformation of disjoint tree structures into a conjoined graph structure. An example is shown in Fig- ure Here, all instances of "night" and "some are born to" are captured by a single node representing that constituent. We call our model BANYAN account of this entangling, because, like the tree, it can have many roots consisting of nodes frequently reused across contexts.
Effectiveness and efficiency: Beyond the ability to explic- itly incorporate context across data, entangled trees also help the contrastive objective by avoiding false negatives since they do not admit duplicate nodes by construction. Furthermore, the lack duplicate nodes also drastically impacts the memory footprint of the model one deals with the set of all nodes rather than counting each instance as its own node. These effects becomes more pronounced when entangling a larger set of instances as the likelihood of false negative and duplicates goes together.
Constructing an entangled tree given set of disjoint trees a relatively straightforward process and formally specified Algorithm In contrast the
Entangling:
Practical estimation: Given the advantages entangled trees, one would ideally want to construct it over all the

---

Banyan: Improved Representation Learning with Explicit Structure
available data-not practically feasible with the exponential growth in dataset sizes. To address this, we construct our model to estimate the given objective by taking steps over batches of data that are of a more manageable size, noting that this estimator is unbiased. To see this is the case, note that entangled trees only affects the downward embeddings directly, and that batching simply means that the vesolved embedding is an average over samples instead of over all the data (population) the sample mean is always an unbiased estimator of the population mean.
5. Experiments: English Evaluation
Goal: We wish to test whether BANYAN can efficiently learn semantics. We start by evaluating on English, which is well resourced and has a wide array of test sets available with which we measure the efficacy of our embeddings. This is crucial to establish, because when we turn to low resource languages later on, the amount of reliable evalu- ation sets will become limited. We want to make use of broad spectrum of tests available for English to reliably demonstrate embedding quality before moving forward.
4.2. Simplified Message Passing
5.1. Experimental Setup and Evaluation:
Complementary to the development of entangled trees to incorporate context, we also explore avenues to improve the message passing with the composition (C) and decom- position (D) functions. The original formulations (1, 2) concatenate or split using simple single-layer linear neural networks. These were found to lead to better representations than e.g., Tree-LSTM cells, because they forced the model to conform to the compression order of the structure.
We want to evaluate how well BANYAN learns effective semantic representations. Ideally we want to probe this at different levels of hierarchy, because allows us test whether structured models can do what they are supposed to i.e., seamlessly transfer semantic knowledge across different levels of hierarchy via composition. Our evaluation is unsu- pervised, both to directly probe the effect of the inductive bias, and for greater parity with what may be expected in a low-resource domain. It consists of three parts:
But if all that was required for success is to respect the compression order, then one could possibly do better with a simpler solution that exploits diagonalised functions (Ba et al., 2016) a crucial component in the resurgence of recurrent neural networks (Peng et al., 2023; Orvieto et al., 2023; De et al., 2024) introducing decayed memory across time. Thus, rather than using linear layers, we now define:
Correlation with human judgements: We compare the cosine similarity of embedding pairs produced by the model with human judgements of their semantic correspondence. On the word level, we use Simlex-999 (Hill et al., 2015) and WordSim-S/R (Agirre et al., 2009). All tasks measure semantics, but do so on differing axes. Simlex measures sim- ilarity the exclusion of relatedness. Wordsim S measures similarity without penalising relatedness. And Wordsim R measures relatedness. On the sentence level, we use STS-12 through 16 (Agirre et al., 2012; 2013; 2014; 2015; 2016), the STS-B B (Cer et al., 2017), JICKRR (Marelli et al., 2014) and SemRel (Ousidhoum et al., 2024) - which combined
+ ( -
(3)
D(e)) = (c + 01 e (0,) + 0,) t1, Fr o O ,r, 01, 0, E R
(4)
with sigmoid non-linearity (a) applied to parameters both for numerical stability and to enforce a decayed memory over structure depth. Repeated application of the diagonal composition function will decay the influence of nodes fur- ther down in the tree, thereby respecting its compression order. During composition representations can increase in magnitude they are the sum of the two children. During decomposition representations will, by necessity, reduce back down in magnitude towards the leaves. Further mim- icking the information flow specified by the entangled trees. Finally, they restrict encoder (comp) and decoder (decomp) embeddings to remain the same space. Which makes amortisation required for successful reconstruction, letting us switch objective to cross entropy over the vocabulary. We provide analysis to support this claim later in $ 77
cover a wide array of semantic correspondence.
Retrieval:
This is a cornerstone of Cetrieval-Augmentée Generation (RAG)-based systems and perhaps the most important use case for embedding models. We use two retrieval datasets from the BEIR suite (Thakur et al., 2021). Quora: evaluates success of matching questions to answers and capturing the response relation. Arguana: evaluates matching arguments to counter arguments, testing if our semantic space captures the notion of dialectical opposition.
Classification: We also include two test sets from the GLUE benchmark (Wang et al., 2019). Sentiment clas- sification (SST-2) tests whether the representation space captures semantic polarity. Paraphrase detection (MRPC): tests whether our representation space capture semantic equivalence. While our other evaluation is applied to the embeddings zero-shot, for the classification tasks we train GeLU MLP with a 512D hidden size, though we leave models frozen direct test of representation quality.
These relatively simple changes have pretty drastic effect, both in terms of performance (see experiments), as well as efficiency, with parameters now reduced by a factor of U compared to the functions from (1, 2).
5

---

Banyan: Improved Representation Learning with Explicit Structure
Table I: Sentence level results for models pretrained on English. Higher is better. Results represent mean and standard deviation across four random initialisations. Spearman's 's p is * 100 following convention.
Model
STS-12
STS-13
STS-14
STS-15
STS-16
STS-B
SICK-R
SemRel
Score
SELF-STRAE
31.98 1: 0.58
53.88
0.68
37.73 4: 0.70
55.23
0.58
55.55 +: 0.47
39.53
1.61
51.78 + 0.29
50.05 0.92
46.59 0.43
GLOVE
31.61 + 0.31
21.69
0.12
27.37 + 0.10
40.42
0.09
29.27 +
28.25
0.08
50.20
41.20 0.43
33.75 0.04
+ stopword r
39.00 0.57
0.19 41.61
39.31 +: 0.18
0.35 $1.06
45.14 = 0.14
48.40 0.07
52.80 +: 0.04
42.37 +: 0.13
44,96 +: 0.10
Sent2Vec
38.14 + 0.29
51.37
0.48
48.64 + 0.09
67.28
0.02
56.26 0.06
53.39 0.11
59.67
51,47 0.03
53.28 0.11
ROBERTA
42.77 4: 1.27
51.70
1.30
45.67 4: 1.42
63.97
0.81
59.60 + 39,97
0,95
52.93 4: 0.23
52.73
0.58
$1.08 0.61
SimCSE
50.63 + 1.45
62.23
2.51
54.17 = 2.10
68.77 3.00
66.67 = 1.40
53.53 + 1.18
56.87 + 1.16
59.27 0.93
59.02 +: 1.45
BANYAN
51.38 +: 0.15
69.60 0.37
63.20 :: 0.28
73.08 0.26
67.18 = 0.56
61.90 +: 0.63
55.23 +: 0.13
61.88 0.22
62.97 0.03
Table 2: Word level results analogous to Table 1.
model seemed to perform. We refer the reader to Appendix A for ablations. We also note that because we can perform this reduction in channel size, the number of non-embedding parameters for BANYAN drops to just 14, as these are di- rectly proportional to U. The configuration for RoBERTa medium is 8 layers, 8 attention heads, 2048 dimensional feed forward layers, and relative positional embeddings. For SELF-STRAL AEE BANYAN and Sent2Vec we pre-train on a uniform subsample of English Wikipedia consisting of circa 10 million tokens. This represents the lower end of how many tokens might be available in a low resource setting, and allows to test whether these methods efficient. Meanwhile for GLOVE and ROBERTA Wiki-103 (Merity et al., 2016), to ensure that they are not penalised by insufficient scale. Wiki-103 comprises circa 100 million tokens and therefore represents the very upper end of what might be available in a low resource setting. Further training details are in Appendix B.
Model
Simlex
Wordsim-S
Wordsim-R
Score
SELF-STRAE
13.80 + 0.41
54,38 + 0.78
52.85 + 1.27
40,34 + 0.66
GLOVE
27.47 +: 0.25
62,53 + 0.42
51.00 + 0.56
47.00 0.38
Sen2Vee
28.88 1: 0.42
68.32 4 1.28
54.49 1.51
50.56 0.79
ROBERTA
29.23 + 0.64
61.97 + 2.38
46.00 + 2.13
45,73 + 1.71
BANYAN
14.65 + 2,90
63.23 + 2:21
67.73 + 0.3 48.53 1.33
Baselines: We compare against the SELF-STRAE, GLOVE
(Pennington et al., 2014), Vent2Vee (Pagliardini et al., 2017) and a ROBERTA (Liu et al., 2019) in the medium con- figuration from (Ture et al., 2019; Opper et al., 2023a). SELF-STRAF, the closest point of comparison to BANYAN, indicates where the current performance level of structured representation learning lies. GLOVE lets us compare to traditional static embeddings, and tests whether our model is learning anything more than just simple bag of word fea- tures. To obtain sentence embeddings, we report results using both the simple average of the word embeddings and the average with stopwords removed following (Reimers & Gurevych, 2019). The latter is generally stronger. An even more powerful variant is Sent2Vec, which directly opti- mises the averaged representation by using to represent the context. This comparison measures the utility BANYAN'S flexible and parametrised composition process, compared to an optimised average. Finally, for ROBERTA, we report results using both the standard model, and again after en- hancing ROBERTA through an extra round of contrastive SIMCSE training (Gao al., 2021), a further non-lexical baseline. Our pooling strategy is mean. To produce static embeddings from ROBERTA to use in lexical evaluation, we follow Bommasani et al. (2020) and average the contex- tualised representations of all occurrences of the word in the training set. The ROBERTA is intended as a stronger base- line. It has significantly more parameters than BANYAN and can model meaning in context unlike GLOVE and Sent2Vec.
pre-train on
5.2. Results:
Results are shown in Tables to 3. On both the word level and sentence level BANYAN does much better than SELF- STRAE. We ablate the reasons for this in more detail later in the manuscript. Both models suffer on SimLex because they need to model both similarity and relatedness as the latter dictates merge (related concepts often compose together). However, the important thing to note is that the structured models effectively transfer the same performance from the word level to the sentence level. They can take advantage of composition, and transfer the meaning of the parts to under- standing the meaning of the whole. The GLOVE baseline is good on the word level, but does not generalise to the sen- tence level as well as the transformer, even when we give it stopword removal. Similarly, Sent2 is extremely strong on the word level, and while more effective than GLOVE, neither approach seamlessly transfers semantic knowledge to different levels of complexity. BANYAN can, and is able to achieve comparable or better performance than the Sim- CSE ROBERTA despite being much smaller and exposed to 10x less pre-training data. This means we have a structured model that remains efficient and cheap, and also effective at representation learning.
Hy Elyperparameters and Pre- Pra-irain Details: For all mod- els we set the embedding size to 256. For SEL.-STRA AE we use the configuration of (Opper et al., 2023b) and set embeddings as square matrices (i.e., K=16 and U=16). For BANYAN we set these values to K=128 and U=2, because the more independent channels we allowed the better the
6

---

Banyan: Improved Representation Learning with Explicit Structure
Table 3: Sentence level results on retrieval and classification tasks for models pretrained on English.
Quora
Arguana
SST-2
MRPC
Model
NDCG@1 1
NDCG@10
RØ
R
NDCG@1
NDXG@ 10
R@1
R 10
Acc
FI
Self-StrAR
32.88 0.28
40.02
4.94
29.59 + 0.23
44.77
0.28
09.96 + 0.11
15.48 + 0.13 09.96 0.11
21.48 0.22
74.67 + 0.52
80.34 + 0.42
GloVe
29.99 0.14
35.71
0.15
26.08
0.15
43.17
06.19
12.77
06.18
0.19
24.68
7.40
75.83
0.62
--sopwoord mm
44.41 + 0,13
52.54
= 0.17
38.78 + 0.15
62.15
= 0.25
09.89 + 0.19
20.27 + 0.09
0.19 09.89
33.00
0.26
76.50 + 1.08
81 +
Seni2Vec
36.12 0.21
43.26
0.15
31.33 + 0.21
52.38
0.05
09:60 + 0.31
23:24 + 0.15
09.60
0.31
39.73
0.89
76.53 + 0.98
81 +-0
RoBERTa
43.26 0.76
49,97
0.72
37.67 4: 0.68
58.78
17.60
08.
0.43
28.85
0.94
75.68
0.96
SimCSE
51.79 2 2.12
39.30
2.10
45,09 + 1.60
68.74
2.01
10.06 1.27
21.84 2.23
10.06
1.27
37.36
2.16
75.97 4: 1,08
80.83 4: 0.24
Banyan
57.74 :: 0.10
65.71 = 0.14
50.14 = 0.10
75.71 = 0.14
12.42 = 0.40 28.28 =: 0.17
12.42
0.40
48.19 +: 0.18
75.96 #: 0.57
79.48 +: 0.42
Table 4: Multilingual Results. BANYAN performance over four random seeds. Baselines marked finetuned on supervised
semantic similarity datasets. FT-unsupervised finetuning using masked language modelling on same corpora as BANYAN.
Model
Indonesian
Arabic
Telugu
Marathi
Hausa
Afrikaans
Spanish
Amharic
Hindi
Score
XLM-R
46.7
31.6
46.3
55.7
4.1
56.2
68.9
57.3
52.7
46.61
Llama-3.1 (8B)
53,4
31.1
65.6
63.4
6.1
65.4
66.7
64.1
61.7
53.06
Mistral Nemo
50.7
20.1
57
52.3
1.8
58.3
66.2
53.2
55.8
16.16
MiniLM-L121
39
16.1
34.8
39.5
32.7
74.1
58.8
9.6
43.8
38.71
Paraphrase XLM-RF
46.1
61
58.1
79.6
22.5
76.8
71.7
64.6
52
59.16 16
XLM-R (FT)
47.9
33.6
68.8
75.1
14.6
72.6
72.8
59.6
57.6
55.84
BANYAN
41.90
42,28
71.58
66.38
49.68
79.35
60.88
66.40
61.63
60.01
+ 0.56
+ 1.57
+ 1.24
+ 0.84
+ 0.75
+ 0.65
+ 0.86
+ 0.90
+ 0.43
+ 0.35
6. Experiments: Multilingual Evaluation
that have been finetumed on supervised datasets designed to produce high quality semantic representations. The base- lines we select here are emblematic of the kind of models one might reach for in order to embed corpus. For all base- lines use mean pooling following (Reimers & Gurevych, 2019). Finally, for parity we include an XLM-R baseline which is finetuned on the same corpora as BANYAN.
We've established that BANYAN is an efficient learner. This implies potential use for languages that are not well covered by current NLP approaches. Here we test that.
Goal:
6.1. Experimental Setup and Evaluation:
Learning semantic representations for low-resource languages remains an ongoing challenge. A core problem is not just the lack of training data, but also the lack of eval- uation datasets. Recent work by Ousidhoum al. (2024) has sought to address this issue, providing semantic relat- edness test sets for several low resource Asian and African languages, evaluated by comparing embedding similarity to human judgements. This means we can BANYAN'S ability on Indonesian, Arabic, Telugu, Marathi, Hausa, Afrikaans, Spanish, Amharic and sindi-covering a broad spectrum of resource extent. For example, Spanish is generally well represented, while Hausa is considerably less
Tasks:
Banyan Pre-training and Hyperparameters: For Afrikaans, Spanish and Amharic we obtained corpora from Leipzig Corpora Collection (Goldhahn al., 2012). For Amharic utilised MIT licenced pre-training set of mil- lion sequences available at this link. Hausa data was sourced from Opus (Nygaard & Tiedemann, 2003). Each dataset consists of roughly 10 million tokens. We utilise a pre- trained BPE tokeniser for each language from the BPEMB Python package (Heinzerling & Strube, 2018). BANYAN'S hyperparameters remain the same as before. For XLM-R we finetune for 100k steps with early stopping, using a lin- early scheduled learning rate of 5e-5 with 10% of steps as warmup. KLM-R runs at batch size 128 across 4xA40 cards.
Baselines: We select XLM-R (Conneau et al., 2019): transformer encoder trained on 2TB of multilingual data. Llama 3.1 8B (Dubey et al., 2024): a decoder only L.MM trained on 15 trillion tokens. Mistral Nemo 12B: decoder only LLM designed with multi-lingual capacities in mind. In addition we also compare against two specialised embed- ding models from the sentence transformers range (Reimers & Gurevych, 2019): Mini-LM-LI22-2 and Paraphrase- XLM-R R- M-Muuilinguull-VE These are pre-trained encoders
6.2. Results
See Table 4. In Spanish, well-resourced language with high coverage, the transformer baselines almost all outper- form BANYAN. However, as languages become lower re- sourced the picture changes, and BANYAN outperforms or is comparable to the baselines. This even includes the multilin- gual XLM-R -R that has undergone supervised training. While
7

---

Banyan: Improved Representation Learning with Explicit Structure
Table 5: Number of non-embedding parameters.
Model
BANYAN
SELF-STRAE ROBERTA (M)
All-MiniLM-LI2-V XLM-R
Llama 3.1
Mistral Nemo
Params
14
1072
100M
221M
85M
RBB
12
Table 6: Ablations of modelling changes made for Banyan. Higher is better. Results averaged across four random
initialisations. Bolded results indicate no standard deviation overlap. Spearman's 's p * 100 following convention.
Model
STS-12
STS-13
STS-14
STS-15
STS-16
STS-B
SICK-R
SemRel
Score
Standard Troes
31.98 + 0.58
53.88 +: 0.68
31.73 + 0.70
55.23 = 0.58
55.55 + 0.47
39.53 = 1.61
51.78 = 0.29
50.05 = 0.92
46.59 = 0.43
diag functions
35.13 + 0.33
56.05 0.24
40.58
58.83
0.10
44.10
53.35 0.17
52.65
49.68 0.06
++ CE loss
47.10 1. 1.04
61,85 1.44
58.60 1.34
70.45
0.57
62.45 0.70
59.50 1 0.53
0.26 59,00
60.33 L 0.26
59.91
0.54
Entangled Trees
38.98 1: 0.39
61.75 1: 0.14
43.65 +: 0.46
58.21
0.41
55.29 0.23
46.15 1: 0.71
$3.93 0.16
52.53 = 0.09
51.31 :: 0.13
+ diag functions
44.15 1. 0.002
62,80
0.002
48.30 1. 0.001
64.60
0.002
60.30 1. 0.001
49.80 1. 0.002
55.14 1 0.001
57.70 1 0.001
55,23 1. 0.01
++ CB
51.38 0.15
0.37
63.20
73.08
0.26
0.56
61.90
55.23 0.13
61.8
62.97 0.03
finetuning XLM-R improves performance the amount of benefit it provides is not uniform and is insufficient to prove viable in the very low resource cases. BANYAN is able to learn competitive representations consistently across lan- guages, unsupervised and with very little data, providing a viable alternative for cheaply and efficient embeddings for low resource languages.
TRtal Number Nodien Batch
200003
250003
300093
ves1
7. Improvements and Ablations
7.1. Efficiency
Bach Sbc
Other than its embedding matrix, BANYAN only has com- position and decomposition functions. Diagonalising these makes them easier to compute and more lightweight than standard weight matrices, (2U rather than 2U U), achiev- ing further order of magnitude reduction in parameters compared with the already minimal SELF-STRAL Table Table 5 shows the difference, including a comparison to the various baselines used throughout the paper. Despite its size BANYAN remains competitive.
Figure 4: Total number of nodes in entangled vs. sentential trees as batch size grows.
similar results in about two hours with smaller batch size. Inference can also be performed on CPU on typical laptops, because the model is so small. Combined with its data efficiency, we believe this provides a promising alternative for low resource languages and communities.
Secondly, by exploiting entangled tree structure the number of nodes grows at a significantly reduced rate with batch size compared with standard sentential trees (see Figure 4). This is because the number of reused constituent nodes also grows as batch size increases, and entangled trees capture the set of all constituents, which consequently does not grow as drastically. In practical terms, because entangled trees requires fewer nodes, and each node requires two distinct embeddings (ē and e) to be held for it, reducing the number of nodes required leads to radical improvements in memory efficiency. Put together, these changes mean that we can train BANYAN very quickly as we can use large batches and its small number of parameters ensure quick convergence. On a single Nvidia A40 GPU with a batch size of 1024, Banyan trains from scratch in under 50 minutes, meaning that the total cost of pretraining a BANYAN model sits at around 30 cents?. Free-tier Google Colab users achieve
7.2. Ablations
Why is BANYAN so much more effective than its SELF- STRAE predecessor? To probe the impact of each change, we perform ablations using the English STS tasks (Table 6).
Beyond improving efficiency, changing entangled trees yields some benefits in terms of performance. The effect is significantly more pronounced when using the contrastive objective, as it removes the issue of false negatives as dis- cussed in Section 4. However, it also yields some slight benefit with the CE leaf reconstruction objective. Entan- gling explicitly allows the model to take advantage of shared constituency structure between complex sequences, as combines the information from all incoming parent mes- sages. The slight edge this provides indicates that explicitly allowing the model to take advantage of such systematic- ity may be useful. However, in terms of performance, we find that the choice of functions and objective plays a much bigger role than entangling.
"Cloud computing costs from: https//wwww.unpoo.oo/piii.
8

---

Banyan: Improved Representation Learning with Explicit Structure
CoMp
DECOme
DECOM
-32
-32
-3.0
-2.0
-25
(a) Uniformity (1) and Alignment loss (L) analysis BANYAN's representations using:
(b) UMAP analysis of COMP and DECOMP
(c) UMAP analysis of COMP and DECOMP embeddings for 30k node entangled tree learned by CE with diagonal functions.
embeddings for a 30k node entangled tree learned by CE w/o diagonal functions.
CONTRASTIVE, CE and CE+DIAG.
Figure 5: Representation Analysis UMAP Parameters in Appendix C.
Diagonal Functions: Perhaps the clearest benefits come from the introduction of the diagonalised composition and decomposition functions. These are bounded scalar values (sigmoid) multiplied with embeddings to mimic the time mixing blocks of SSMs (Gu & Dao, 2024). Hierarchically decaying in the influence of embeddings further down the structure through repeated application. This means that the representations are restricted conform to the com- pression order it dictates, and we know from (Opper al., 2023b) that the more we enforce this constraint the better our representations. Secondly, such simple message passing functions bias the representation space towards informative separability. There has to be some signal from which to perform reconstruction, and all the pressure now on the representations. This is beneficial with the contrastive loss, but really shines when we combine them with cross entropy.
loss, without its propensity for shortcut solutions (Robinson et al., 2021). As result, we can switch objective without compromising representation quality.
8. Conclusion, Limitations and Future Work
We introduce BANYAN, a Self-Structuring AutoEncoder. BANYAN's focus on global, entangled structure and sim- plified message passing exploits the benefits of structured compositions inherent in language data. It is more effective and efficient than prior work from which we draw three central conclusions.
Firstly, explicitly modelling structured compositions an effective inductive bias. Table 5 shows the parameters for the structured models versus the baselines. The structured models are far smaller, with tens or thousands of parameters instead of millions or billions. And nonetheless, BANYAN is still competitive across several metrics, indicating we have found an efficient learning procedure.
Changing Objective: Our more instructive finding is that cross entropy now outperforms the contrastive loss used by Opper et al. (2023b), contrary to the earlier result that the contrastive objective was critical. Figure provides an analysis with Figure 5a showing uniformity vs. alignment metrics (4) from Wang & Isola (2020). These measure (a) how evenly spread embeddings are (uniformity) and (b) the proximity of locally contextual composition embeddings to their globally contextualised counterparts (alignment). Success requires having low scores on both. We can see that the contrastive loss does well, while eross entropy achieves high uniformity, but not alignment without diagonal func- tions. This is further confirmed when we look the UMAP (MeInnes et al., 2018) analysis of BANYAN's embeddings with (5c) and without (5b) diagonal functions. Without diagonal functions, composition and decomposition embed- dings largely occupy separate subspaces, failing to amortise over context. Introducing diagonal functions results in in- credibly high overlap between the two, indicating effective amortisation. We believe that this is due their simplicity and scaling constraints, restricting complex transformations during message passing. This implicitly forces representa- tions to optimise the same useful qualities as the contrastive
Secondly, we have not yet fully exploited the potential of the inductive bias. BANYAN still relies on greedy agglom- erative clustering to induce structure. This is effective, but sub-optimal. Future work could learn the structure induction procedure. The type of structure models are exposed to im- pacts the quality of learnt semantic representations (Opper et al., 2023b). So if how we induce structure improves, the model should learn better representations.
Finally, good and cheap embedding models are useful for many applications. For example, the digital humanities need to organise corpora of ancient languages, making it easier for researchers to access texts they need. But these corpora are small, and these languages are unlikely to be present in pretraining corpora of larger models. BANYAN provides an efficient solution for producing representations for both these use cases and low resource languages and under repre- sented communities more generally. To conclude, Banyan addresses the problem of efficient learning in low-resource settings.

---

Banyan: Improved Representation Learning with Explicit Structure
Acknowledgements
2013. Association for Computational Linguisties. URL https:/ http::s///cllatthhllloooo ogyy . wrgg//3-10/00
MO was funded by a PhD studentship through Huawei- Edinburgh Research Lab Project 10410153. We also wish to thank Henry Conklin, Seraphina Goldfarb-Tarrant Vivek Iyer, Ivan Vegner, Sahil Verma, Su Kara, Ivan Titov and Edoardo Ponti for their valuable comments and feedback, throughout the research and development of this work. We'd d also like to thank the ICML reviewers for their suggestions to help improve the paper. Finally, a special thank you goes to James Morrison, for helping to think through the ideas and providing invaluable feedback and edits to the manuscript.
Agirre, E., Banea, C., Cardie, C., Cer, D., Diab, M., Gonzalez-Agirre, A., Guo, W., Mihalcea, R., Rigau, G., and Wiebe, J. SemEval-2014 task 10: Multilinguul se- mantic textual similarity. In Nakov, P. and Zesch, T. (eds.), Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pp. 81-91, Dublin, Ireland, August 2014. Association for Compu- tational Linguisties. doi: 10.31155/11$14422000 URL https: pttps://aalartthl..ooo . org/ 20////1422001
Agirre, E., Banea, C., Cardie, C., Cer, D., Diab, M., Gonzalez-Agirre, A., Guo, W., Gazezo I., Maritx- alar, M., Mihalcea. R., Rigau, G., Uria, L., and Wiebe, J. SemFval-2015 task 2: Semantic textual similar- ity, English, Spanish and pilot on interpretability. In Nakov, P., Zesch, T., Cer, D., and Jurgens, D. (eds.), Proceedings of the 9th International Workshop on Se- mantic Evaluation (SemEval 2015), pp. 252-263, Den- ver, Colorado, June 2015. Association for Computa- URL
Impact Statement
This paper represents an effort to enable resource efficient embedding techniques. We hope that our model makes a positive step towards making Al research and technologies more fair, equitable and accessible.
References
tional Linguistics. doi: 10.18653/ https: /aa/aclaaatttt wowlaanhhooogyy...oo...... wew.llyooooooroogeeeeeeee st////15522055
Abdalla, M., Vishnubhotla, K., and Mohammad, S. What makes sentences semantically related? textual relat- edness dataset and empirical study. In Vlachos, A. and Augenstein, I. (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics, pp. 782-796, Dubrovnik, Croa- tia, May 2023. Association for Computational Linguis- tics. doi: 10.18653//1/2023..... eacl- 55,. URL https: 55.
Agirre, E., Banea, C., Cer, D., Diab, M., Gonzalez- Agirre, A., Mihalcea, R., Rigau, G., and Wiebe, J. SemEval-2016 task I: Semantic textual similarity, mono- lingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Æval- uation (SemEval-2016), pp. 19 San Diego Cali- fornia, June 2016. Association for Computational Lin- guistics. doi: 10.18653/ / //aclanthology.oo hww.:/ntwooooy....gs//---11-11
//aclanthology org. 2023
URL https:
Agirre, E., Alfonseca, E., Hall, K., Kravalova, J., Paşca, M., and Soroa, A. A study on similarity and relatedness using distributional and WordNet- based approaches. In North American Chapter of the Association for Computational Linguistics (NAACL.), pp. 19-27, 2009.
Arora, S., Liang, Y., and Ma, T. A simple but tough- to-beat baseline for sentence embeddings. In In- ternational Conference on Learning Representations, 2017. URL i id=Syk00v5xx.
hetp:/iwwwwweeeeeeeeeeeeeeeeeeee net//orumm
Agirre, E., Diab, M., Cer, D., and Gonzalez-Agirre, A. Semeval-2012 task 6: A pilot on semantic textual similar- ity. In Proceedings of the First Joint Conference on Lexi- cal and Computational Semantics- Volume 1: Proceed- ings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Work- shop on Semantic Evaluation, SemEval '12, pp. 385-393, USA, 2012. Association for Computational Linguistics.
Ba, J., Hinton, G., Mnih, V., Leibo, J. Z.,, and Ionescu, Using fast weights to attend to the recent past, 2016. URL https: httpaaaivvvvv.o....a.aaaa org/ abs/
Bommasani, R., Davis, K., and Cardie, C. Interpreting Pre- trained Contextualized Representations via Reductions to Static Embeddings. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 4758 4781, Online, July 2020. Association for Com- putational Linguistics. doi: 100118665/112002.ada-mmmmi 431. URL https: http:::s//aclaathhllogooooo wrg.///0..oo//.00//////// ael-main... 431.
Agirre, E., Cer, D., Diab, M., Gonralez-Agirre, A., and Guo, W. *SEM 2013 shared task: Semantic textual similarity. In Diab, M., Baldwin, T., and Baroni, M. (eds.), Second Joint Conference on Lexical and Compu- tational Semantics (*SEM), Volume I: Proceedings of the Main Conference and the Shared Task: Semantic Tex- tual Similarity, pp. 32 43, Atlanta, Georgia, USA, June
Cer, D., Diab, M., Agirre, E.,, Lopez-Gazpio, I., and Spe- cia, L. SemEval-2017 task 1: Semantic textual similar-
10

---

Banyan: Improved Representation Learning with Explicit Structure
ity multilingual and crosslingual focused evaluation. In Workshop on Semantic Evaluation (SemEval), pp. 1-14, 2017.
Ethayarajh, K. Unsupervised random walk sentence em- beddings: A strong but simple baseline. In Augenstein, I., Cao, K., He, H., Hill, F., Gella, S., Kiros, J., Mei, H., and Misra, D. (eds.), Proceedings of the Third Work- shop on Representation Learning for NLP,, pp. 91-100, Melbourne, Australia, July 2018. Association for Compu- tational Linguisties. doi: 10. 10.18663/1//118-30010 URL betps://aclanthoon
Chomsky, N. Three models for the description of language. IRE Transactions on Information Theory, 233):1 1956. doi: 10.1 111.0000...999.11166000000
Chowdhury, J. R. and Caragea, C. Modeling hierarchi- cal structures with continuous recursive neural networks. CoRR, abs/2106.06633, 2021. URL htttsss://aaxxx org/abas aas/ 2106..66000 06038.
Gao, T., Yao, X., and Chen, D. SimCSE: Simple con- trastive learning of sentence embeddings. In Moens, M.-., Huang, X., Specia, L., and Yih, S. W.t. (eds.), Proceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pp. 6894-6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 1/t.665////222...o//
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. Unsupervised cross-lingual repre- sentation learning at scale. CoRR, attpaa////111111/11 2019. URL http: //ariiv.aaiiv////a www.rxii....rraeoooo http:/bbs11911111 02116.
552. URL https:// aclantho ogy orgg 2021 emn 1 p-ma i....... 552..
Crain, S. and Nakayama, M. Structure dependence in gram- mar formation. Language, 63(3):522-543, 1987.
Godey, N., Clergerie, É., and Sagot, B. Anisotropy inher- ent to self-attention in transformers. In Graham, Y. and Purver, M. (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computa- tional Linguistics (Volume I: Long Pupers), pp. 35-48, St. Julian's s, Malta, March 2024. Association for Compu- tational Linguistics. URL https: ht////aaaaaatoooo........ org 2024 0ts//2224.accl1oome
De, S., Smith, S. L., Fernando, A., Botev, A., Cristian- Muraru, G., Gu, A., Haroun, R., Berrada, L., Chen, Y., Srinivasan, S., Desjardins, G., Doucet, A., Budden, D.,
Teh, Y. W. Pascanu, R., Freitas, N. D., and Gulcehre, C. Griffin: Mixing gated linear recurrences with local attention for efficient language models, 2024.
3/.
Goldhahn, D., Eckart, T. and Quasthoff, U. Building large monolingual dictionaries the Leipzig corpora collection: From 100 to 200 languages. In Calzolari, N., Choukri, K., Declerck, T., Dogan, M. U., Maegaard, B., Mariani, J., Moreno, A., Odijk, J., and Piperidis, S. (eds.), Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pp. 759-765, Istanbul, Turkey, May 2012. European Language Resources Association (ELRA). URL http:/ /twww/wwww.// www.lrrecooo conf. prgg/proceedings// 1rec2012, /ddp Saiff/322_PPP 32tp332pePPPPPaae_pee
de Marneffe, M.-.., MacCartney, B., and Manning, C. D. Generating typed dependency parses from phrase structure parses. In Proceedings of the Fifth International Conference on Language Re- sources and Evaluation (LREE'006). Genoa, Italy, May 2006. European Language Resources Associa- tion (ELRA). URLhttp: 1rec proceedings/eeee006///
rec- conf. org/ pdf. pdf.
Drozdov, A., Verga, P., Yadav, M., lyyer, M., and Mc- Callum, A. Unsupervised latent tree induction with deep inside-outside recursive autoencoders. CoRR, abs/1904.02142,, 2019. URL http://arxiv.orrt abs/19044 02142.
Gu, A. and Dao, T. Mamba: Lincar-time sequence mod- eling with selective state spaces, 2024. URL https:
/ //arxiv.
Drozdov, A., Rongali, S., Chen, Y.P., O'Gorman, T., Iyyer, M. and McCallum, A. Unsupervised parsing with S- DIORA: Single tree encoding for deep inside-outside recursive autoencoders. In Proceedings the ference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pp. 4832-4845, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.118653/11/200.eemmlpp m9....3.. URL https:// aclanthology wttg//2200000 wttp://22.e.emmmmm.a.... emnlp-main. 392.
Harris, Z. S. Distributional structure. Word, 10:146-162, 1954.
Havrylov, S., Kruszewski, G., and Joulin, A. Cooper- ative learning of disjoint syntax and semantics. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume / (Long and Short Papers), pp. 1118-1128, Minneapolis, Min- nesota, June 2019. Association for Computational Lin- guistics. doi: 10.18653/v1/N19-11 //aclanthology. www.aagy.orr/gg/ org/N1 wwgg/19-11/111111111111111111111
the 2020 Con-
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., et al. The lama 3 herd of models, 2024.
URL https:
URL https::///// /ttp///aaiii arxiv. wttpaawwvr.g///.aa./////../ooote abs/2407 21783.

---

Banyan: Improved Representation Learning with Explicit Structure
Heinzerling, B. and Strube, M. BPEmb: Tokenization- free Pre-trained Subword Embeddings in 275 Languages. In chair), N. C. C., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Hasida, K., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., Piperidis, S., and Tokunaga, T. (eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 7-12, 2018 2018. European Language Resources Association (ELRA). ISBN 979-10-95546-000-
Levy, O. and Goldberg, Y. Dependency-basee word em- beddings. In Association for Computational Linguisties (ACL) 2: Short Papers), pp. 302-308, 2014.
Lewis, P. S. H., Perez, E., Piktus, A., Petroni, E, Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. Retrieval- augmented generation for knowledge-intensiv NLP tasks. CoRR, 000020055111401, 2020. URL https: / www.iv.org/ab.... 2005. 11401.
Liu, Y., Ott, M., Goyal, N., Du, Joshi, M., Chen, D., Levy, O., Lewis, M.. Zettlemoyer. L., and Stoyanov, V. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692 2019.
Hill, F., Reichart, R., and Korhonen, A. SimLex-999; Eval- uating semantic models with (genuine) similarity estima- tion. Computational Linguistics, 41(4):665-695, 2015.
Hu, X., Mi, H., Wen, Z., Wang. Y., Su, Y., Zheng, J., and de Melo, G. R2D2: Recursive transformer based differ- entiable tree for interpretable hierarchical language mod- cling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1tth International Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pp. 4897-4908, Online, August 2021. Association for Computational Linguis- URL https: acl- acl--ong 379.
Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Bernardi, R., and Zamparelli, R. A SICK cure for the evaluation of compositional distributional semantic mod- els. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14), pp. 216-223, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA). URL http:/ / //www wwwrlrrccco conf wttpprooceeddggs/// 1rec201 4 http//ff/36/ 1ae3336p__eeppp____
tics. doi: 10. 10.186631w1/20021allllooooo //tclanthologyy.oo...o. wow.::lllolooooo/o/oe
McInnes, L., Healy, J., Saul, N., and Grossberger, L. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018.
Hu, X., Mi, H., Li, L., and de Melo, G. Fast-R2D2: A pre- trained recursive neural network based on pruned CKY for grammar induction and text representation. In Gold- berg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2809-2821,, Abu Dhabi, United Arab Emirates, December 2022. Association for Compu- tational Linguistics. doi: 10.18655/w1/2022..mmipp 181. URL https: https/////aaaaanoolloooooo 2wtp://hooooooo.ooo/2/2/22222 2002. monlp-main...
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. CoRR, abs/1609.07843, 2016. URL Utt.htp::/ / 6009 07843.
Mikolov, T., Chen, K., Corrado, G., and Dean, J. Efficient estimation of word representations in vector space. In International Conference on Learning Representations (ICLR), 2013.
Ito, T., Klinger, T., Schultz, D. H., Murray, J. D., Cole, M. W., and Rigotti, M. Compositional generalization through abstract representations human and artificial neural networks, 2022. URL hwtpsst//////tttteeee/ee//ee //aaxvvv.. org/
Murty, S., Sharma, P., Andreas, J., and Manning, C. Push- down layers: Encoding recursive structure in transformer language models, 2023. URL https: /ttsss///xxvvooooe 19089.
abs/2209. 07431.
Nygaard, L. and Tiedemann. J. Opus open source paral- lel corpus. In Proceedings of the 13th Nordic Conference on Computational Linguistics, 2003.
Kingma, D. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.
Opper, M. and Siddharth, N. Self-strae at semeval-2022 task 1: Making self-structuring autoencoders learn more with less. arXiv preprint 24Xiv:2404.01880 2024.
Lake, B. M., Ullman, T. D., Tenenbaum, B., and Ger- shman, S. J. Building machines that learn and think like people. CoRR, abs//660400022 2016. URL http: //arxiv.orgg org/ abs / 604.664400 00289.
Opper, M., Morrison, J., and Siddharth, N. On the effect of curriculum learning with developmental data for grammar acquisition. arXiv preprint arXiv:2311.00123 2023a.
Le, P. and Zuidema, W. Inside-outside semantics: A frame- work for neural models of semantic composition. In NIPS 2014 Workshop on Deep Learning and Representation Learning, 2014.
Opper, M., Prokhorov, V., and Siddharth, N. Strae: Au- toencoding for pre-trained embeddings using explicit
12

---

Banyan: Improved Representation Learning with Explicit Structure
structure. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7544-7560, 2023b.
Learning Research, pp. 28768-28791. PMLR, 23-29 Jul 2023. URt.tttps::/prroeeeeee www.gnnw...............eeeeeeeee r.rress/ v202/ a000/ray--o www.aayyhhwwhhuyy...eeee.. .tmml.
ml
Orvieto, A., Smith, S. L.., Gu, A., Fernando, A., Gulcehre, C., Pascanu, R., and De, Resurrecting recurrent neural networks for long sequences, 2023.
Reimers, N. and Gurevych, I. Sentence-BERI: Sentence em- beddings using Siamese BERT-networks. In Empirical Methods in Natural Language Processing and Interna- tional Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982-3992, 2019.
Ousidhoum, N., Muhammad, S. H., Abdalla, M., Abdulmu- min, I,, Ahmad, I. S., Ahuja, S., Aji, A. F., Araujo, V., Beloucif, M., De Kock, C., Hourrane, O., Shrivastava, M., Solorio, T., Surange. N., Vishnubhotla, K., Yimam, S. M., and Mohammad, S. M. SemEval-2024 task 1: Seman- tic textual relatedness for african and asian languages. In Proceedings of the 18th International Workshop on Semantic Evaluation 2022 Association for Computational Linguistics, 2024.
Robinson, J., Sun, L., Yu, K., Batmanghelich, K., Jegelka, S., and Sra, S. Can contrastive learning avoid shorteut solutions? CoRR, abs/2106. 11230, 2021. URL https: //arxiv. org/ abs 2210 11230.
Rticklé, A., Eger, S., Peyrard, M., and Gurevych, I. Concate- nated p-mean word embeddings as universal cross-lingual sentence representations.. CoRR, wwwpaa/3.0114400000 2018. URL
Pagliardini, M., Gupta, P., and Jaggi, M. Unsupervised learning of sentence embeddings using compositional n-gram features. CoRR, abs/ 1703.02507, 2017. URL http:://axxvvorrt owg// abs /1003. .02507.
wtt.://.gggabbbbbboooooo.oe 01400.
Sartran, L., Barrett, S., Kuncoro, A., Stanojević, M., Blun- som, P., and Dyer, C. Transformer grammars: Aug- menting transformer language models with syntactic inductive biases at scale. Transactions of the Asso- ciation for Computational Linguistics, 10:1423-1439, 2022. doi: 10.1162/tacl URL https:// aclanthology org 2022. tacl-1.81.
Pallier, C., Devauchelle, A.-D., and Dehaene, S. Cortical representation of the constituent structure of sentences. Proceedings of the National Academy of Sciences, 108 (6):2522-252) 2527, 2011. doi: 10.1073/pnas.1011 101871 1108.
Peng, B., Alcaide, E., Anthony, Q., Albalak. A., Arcadinho, S., Cao, H., Cheng, X., Chung, M., Grella, M., GV, K. K., He, X., Hou, H., Kazienko, P., Kocon, J.. Kong, J., Kop- tyra, B., Lau, H., Mantri, K. S. I., Mom, F., Saito, A., Tang, X., Wang, B., Wind, S., Wozniak, S., Zhang, R., Zhang, Z., Zhao, Q., Zhou, P., Zhu, J., and Zhu, R.-J. Rwkv: Reinventing rnns for the transformer era, 2023.
Socher, R., Pennington, J., Huang, E. H., Ng. A. Y., and Manning, C. D. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Empirical Meth- ods in Natural Language Processing (EMNLP), pp. 151- 161, 2011.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1631-1642, 2013.
Pennington, J., Socher, R., and Manning, C. GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543.
2014.
Soulos, P., Conklin, H., Opper, M., Smolensky, P., Gao, J., and Fernandez, R. Compositional generalization across distributional shifts with sparse tree operations. arXiv preprint arXiv:2412.14077 2024.
Pham, N. T., Kruszewski, G., Lazaridou, A., and Baroni, M. Jointly optimizing word representations for lexical and sentential tasks with the C-PHRASE model. In Zong, C. and Strube, M. (eds.), Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics Natural and the 7th International Joint Conference Language Processing (Volume I: Long Papers). pp. 971 981, Beijing, China, July 2015. Association for Compu- tational Linguistics. doi: 10.31155/w1/115-0044 URL https: / acl antha httrg//15-11990
Tai, K. S., Socher, R., and Manning, C. D. Improved seman- tic representations from tree-structured long short-term memory networks. In Association for Computational Linguistics (ACL), pp. 1566-1566, 2015.
Thakur, N., Reimers, N., R"uckl'e, A., Srivastava, A., and Gurevych, I. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https: httpps://ppenzeeiee //openrevieee net/ forum? i dd=CCuGT 5xF
Ray Chowdhury, J. and Caragea, C. Beam tree recur- sive cells. In Krause, A., Brunskill, E., Cho, K., En- gelhardt, B., Sabato, S., and Scarlett, J. (eds.), Pro- ceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine
13

---

Banyan: Improved Representation Learning with Explicit Structure
Ture, I., Chang, M., Lee, K., and Toutanova, K. Well-read students learn better: The impact of student initialization on knowledge distillation. CoRR, w9w///99////.. 009622 2019. URL
M., Lhoest, Q., and Rush, A. M. Transformers: State- of-the-art natural language processing- In Proceedings of the 2020 Conference Empirical Methods in Natu- ral Language Processing: System Demonstrations, Pp. 38-45, Online, Octoher 2020. Association for Compu- tational Linguistics. URL https: : /tt://wwww /wwww aclweb. org/anthology/2200 2020. emnlp demos.
org/ abs/aaas........... 08962.
Vashishth, S., Bhandari, M., Yadav, P.. Rai, P., Bhat- tacharyya, C., and Talukdar, Incorporating syntac- tic and semantic information in word embeddings using graph convolutional networks. In Association for Compu- fational Linguistics (ACI.), pp. 3308-3318, 2019.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L.,, and Polosukhin, I. Atten- tion all you need. In Advances Neural Information Processing Systems (NeurIPS), pp. 5998 6008, 2017.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations (ICLR), 2019.
Wang, B., Kuo, C.-C. J., and Li, H. Just rank: Rethink- ing evaluation with word and sentence similarities. Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 6060-6077, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/0022 acl-long.419, URL https::///// http:s//aaaaaathhlooooooo 2022. acl-long 4.9.
Wang, M., Zheng, D., Ye, Z., Gan, Q. Li, M., Song, X., Zhou, J., Ma, C., Yu, L.,, Gai, Y., Xiao, T., He, T., Karypis, G., Li, J., and Zhang, 7. Deep graph library: A graph- centric, Sebhl--performnttt package for graph neural net- works, 2020.
Wang, T. and Isola, P. Understanding contrastive repre- sentation learning through alignment and uniformity the hypersphere. CoRR, abs/2005.10242. 2020. URL https: wtwp/v.xvvvv././/.e//..a... org/ abs// 2005. 10242.
Wiedemer, T., Mayilvahanan, P., Bethge, M., and Bren- del, W. Compositional generalization ples, 2023. geniriaiiioonn from first princi- 05596.
aaaabb/2220
Wieting, J., Gimpel, K., Neubig, G., and Kirg-Kirkpatrickk T. Paraphrastic representations at scale. CoRR, abs/2104.15114, 2021. URL https://arxiv..or 0/./2104..11111
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M.. Davison, J., Shleifer, S., von Platen, P., Ma, C., Jer- nite, Y., Plu, J., Xu, C., Scao, T. I.,, Gugger, S., Drame,
14

---

Banyan: Improved Representation Learning with Explicit Structure
A. The k and balance
The change to diagonal composition functions allows us to reduce the number total parameters while maintaining performance. This because the number of parameters directly proportional channel size u. We show ablations for this finding in Table Our findings are similar to those of (Opper & Siddharth, 2024) the smaller the channel size the better the model performs, although in case keep things stable between seeds whereas before simplification caused issues with extreme instability during training. This is thanks to the new message passing functions.
Table 7: Performance Depending on and a values using new functions. Scores are the average of four random seeds.
u
Lex Score
STS Score
4
64
47.83 + 0.2
55.50 + 0.22
8
32
47.41 = 1.0
62.58 + 0.11
16
16
48.01 +
62.73 + 0.1
32
8
47.65
1.1
62.79 0.07
64
4
48.48
0.7
62.63 0.16
128
2
48.53
1.33
62.97 0.23
256
0.6 49.15
62.61 0.23
B. Hyperparameters:
We trained SELF-STRAE and BANYAN for 15 epochs (circa 15k steps and sufficient for convergence) using the Adam optimiser (Kingma & Ba, 2015), with a learning rate of 1e-3 for BANYAN and le-4 for SELF-STRAE using a batch size of 512. To process the graphs we used DGL (Wang et al., 2020). The GLOVE baseline was trained for 15 epochs with learning rate of le-3, and a window size of 10. We used the official C++ implementation. ROBERTA medium was trained for 200,000 steps, (10% of which were used for warmup). We used a learning rate of 5e-5, and linear schedule. Positional embeddings are relative key-query. We used the Transformers library implement and train the model (Wolf et al., 2020). For SimCSE training, used the default parameters and the official implementation for unsupervised ROBERTA training from Gao et al. (2021). For Vent2Vec used their official implementation and recommend hyperparameters.
C. UMAP Parameters:
For the UMAP visualisations, we set number of neighbours to 100, minimum distance to 0.3, the metric to cosine and local connectivity to 3. However, the same patterns can we observed through wide array of hyperparameters, and when changing the metric euclidean distance. The behavioural changes induced by the diagonal functions remain clear. We selected the above purely based on aesthetic preference for the resulting plots.
15


Reviews:
[{"review_id": "ePqFGYhFqh", "reviewer": "Reviewer_PYAv", "summary": "The paper proposes BANYAN, a graph-based autoencoder that learns sentence representations by explicitly encoding hierarchical structures. It extends a prior structured model (SELF-STRAE) in two major ways: 1) Entangled Trees: Instead of building a separate tree per sentence, the model merges identical token spans across all sentences (in a batch) into shared nodes. Each node can thus gather training signals from multiple contexts, rather than duplicating the same span across different sentences. This “entangling” reduces memory usage (fewer total nodes) and helps prevent conflicting training signals. 2) Diagonalized Message Passing: The composition and decomposition functions that build and break down embeddings along the tree are replaced with tiny diagonal gating operations. Rather than a full matrix multiply, each dimension is scaled by a learned scalar in [0, 1], allowing the model to control how much information from each child flows upward or downward. This sharply cuts parameter count (just 14 scalars beyond the embeddings) and enforces a rigid “compression order” over the tree. Training uses a cross-entropy reconstruction of the original sentence, ensuring that the root embedding and each intermediate node carry enough information to decode their children. The structure itself is induced via a greedy merging algorithm that repeatedly combines the most similar pairs of embeddings, forming a tree bottom-up. By reusing nodes across sentences, BANYAN effectively averages how each repeated phrase is used in different contexts.\n\nBANYAN is tested mainly on semantic textual similarity (STS) tasks at word and sentence level, plus a few classification and retrieval tasks. Key findings are that, despite having very few non-embedding parameters, Banyan matches or beats larger transformer models on many STS benchmarks. It excels especially in low-resource languages, where big multilingual transformers often struggle without abundant data.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, and positive appraisal of our work! We will make sure to fix the typo - thanks for pointing that out!\n\nRegarding the trend of bigger models and more data, this is definitely the way things are moving, but it also leaves a lot of languages behind and limits who can participate in research. Efficient solutions can help bridge that gap.", "claims_and_evidence": "The paper’s core claim is that explicit structure and minimal gating can yield strong, data-efficient embeddings. Results on various English and multilingual STS sets support this, showing BANYAN is both much smaller than conventional models and still highly competitive. Another claim is that merging identical spans across sentences removes duplication and leverages repeated phrases more effectively. This is demonstrated by the model’s consistent gains over the previous approach that used one tree per sentence, as demonstrated in ablations.", "methods_and_evaluation_criteria": "Banyan is an unsupervised method. The primary measure of success is whether the learned embeddings rank sentence pairs in alignment with human-rated similarity.\nIt also checks retrieval metrics (e.g., NDCG, Recall) and downstream classification performance with a frozen encoder to assess practical utility. Overall, the evaluation makes sense.", "theoretical_claims": "The paper doesn’t present formal proofs but makes some conceptual claims, for example, that diagonal gating enforces a strict “compression order” over the tree and that averaging entangled nodes (batch-wise) is an unbiased estimator of their global context. These claims appear logically consistent with standard ideas in gating (e.g. decaying influence across multiple composition steps) and stochastic training. No glaring theoretical issues were found.", "experimental_designs_or_analyses": "The experiments compare BANYAN to relevant baselines (both structured and transformer-based) on unsupervised semantic similarity tasks, plus a few downstream evaluations. The training setup (e.g., matching embedding sizes across models, consistent optimizer settings) is well-justified. The ablation study (entangled vs. standard trees, diagonal vs. full matrices, contrastive vs. cross-entropy) clarifies each modeling choice’s impact. No obvious flaws in methodology were detected.", "relation_to_broader_scientific_literature": "Banyan follows a line of structured representation learning models but innovates by entangling repeated spans across sentences and using diagonal gating in the composition functions. This ties into prior work on compositionality, unsupervised parsing, and efficient RNN gating, yet it is unique in merging identical spans globally for more efficient, context-rich representations. The discussion references all key works I can think of.", "essential_references_not_discussed": "None I can think of.", "other_strengths_and_weaknesses": "Strengths:\n\n* Original: Entangled composition across multiple sentences is novel and addresses false negatives or duplicate merges.\n* Efficiency: The model relies on only ~14 non-embedding parameters yet can rival large transformers, especially in low-resource contexts.\n* Clarity: The paper is logically organized and methodologically transparent (ablation results, multilingual experiments). It is really well written.\n\nWeaknesses:\n\nThere are no clear weaknesses. The paper supports all its claims quite well. One could point out that the model goes against the obvious trend of training larger models on more data, and will likely not play a big role in the future. But I wouldn't consider that a real weakness.", "other_comments_or_suggestions": "line 163: ask -> asks", "questions_for_authors": "I don't have questions that would change my evaluation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "akF7z5Whky", "reviewer": "Reviewer_GCxG", "summary": "This paper studies the problem of learning semantic representations for language in low-resource settings. While word embeddings can be learned with little data, they are non-contextual; on the other hand, transformers can produce contextual embeddings but are data-hungry. In this work, the authors build on an existing architecture, that of Self-StrAE, and propose Banyan. The two main changes to Self-StrAE are that (1) in the downward pass, embeddings for spans with the same sequence of tokens are averaged following an entangled graph structure, and (2) neural models for embedding combination and decombination are replaced with simple diagonalized functions. The resulting model is very efficient in that training is very quick and inference can be done on CPU. The authors evaluate the model on various word-level and sentence-level tasks, first on English, and then on various low-resource languages, comparing it to word embeddings, Self-StrAE, and various transformer baselines trained on much more data. Banyan outperforms all baselines in most settings. Finally, they perform ablations and show that each proposed architectural change is beneficial.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your extensive review and extremely helpful feedback!\n\nOther sentence embedding baselines:\n\nWe appreciate you providing the missing references and will be sure to include them in the paper. \n\nOur key focus is to assess whether we can create an efficient method for learning representations for use with low resource languages. [1] Baseline Sentence Embeddings and [2] Unsupervised Random Walk both initialise their embeddings from pre-trained GloVe vectors trained on 840 billion tokens of text. This does naturally lead to better performance, but is only possible for super high resource languages like English, and not the low resource settings that we are primarily concerned with.  Power mean combination [4] also use pre-trained embeddings. Further the difference in performance [4] reports on sst is negligible. When we attempted to replicate their method we also found no meaningful change. \n\nSent2Vec [3] reports several pretraining settings, the smallest of which is 900 million tokens with 700D vectors, which again is beyond low resource scale. However, the method is strong and certainly seems worth comparing to. We used the official implementation and followed their hyperparameter recommendations to train our own version.  We used our English Wikipedia subsample and at 256D for parity. Results are as follows:\n\n| Model    | STS-12 | STS-13 | STS-14 | STS-15 | STS-16 | STS-B | SICK  | SemRel | Score |\n|----------|--------|--------|--------|--------|--------|-------|-------|--------|-------|\n| Banyan   | 51.2 +- 0.007   | 69.1 +- 0.002  | 63.3 +- 0.004  | 73.2 +- 0.002  | 66.6 +- 0.002  | 61.5 +- 0.002  | 55.5 +- 0.003  | 61.6   +- 0.002 | 62.7 +- 0.001  |\n| Sent2Vec | 38.14 +- 0.29  | 51.37 +- 0.48  | 48.64  +- 0.09 | 67.28  +- 0.023 | 56.26 +- 0.06 | 53.39 +- 0.11 | 59.67 +- 0.02 | 51.47 +- 0.03 | 53.28 +- 0.11 |\n\n| Model    | Q N@1 | Q N@10 | Q R@1 | Q R@10 | A N@1 | A N@10 | A R@1 | A R@10 | SST   | MRPC |\n|----------|-------|--------|-------|--------|-------|--------|-------|--------|-------|------|\n| Banyan   | 57.83 +- 0.04 | 65.78 +- 0.05 | 50.19 +- 0.08 | 75.80 +- 0.18 | 13.21 +- 0.25 | 29.28 +- 0.11  | 27.41 +- 0.68 | 49.60 +- 0.52  | 79.51 +- 0.16 | 77.2 +- 0.27 |\n| Sent2Vec | 36.12 +- 0.21 | 43.26 +- 0.15  | 31.33 +- 0.21 | 52.38 +- 0.05  | 9.6 +- 0.31  | 23.24 +- 0.15  | 9.6 +- 0.31  | 39.73 +- 0.89  | 76.53 +- 0.98 | 81 +- 0.0  |\n\n| Model    | Simlex | WordSim S | WordSim R | Score  |\n|----------|--------|-----------|-----------|--------|\n| Banyan   | 16.57 +- 0.02  | 63.25 +- 0.03     | 69   +- 0.01    | 49.61 +- 0.02 |\n| Sent2Vec | 28.88 +- 0.42 | 68.32 +- 1.26    | 54.49  +- 1.51   | 50.56 +- 0.79 |\n\nSent2Vec is stronger than our original WE baselines, though Banyan generally retains an edge. \n\nMultilingual Word Embedding Baseline:\nWe only compare to large pretrained models here because we hope we already prove Banyan’s utility compared to other methods you could easily train from scratch in our initial experiments. Nevertheless we have trained sent2vec for a few of the languages and find similar trends to the results on English:\n\nAfrikaans:\nBanyan: 78.68 +- 0.30\nSent2Vec: 73.36 +- 0.55\n\nTelugu:\nBanyan: 71.13 +- 0.91 \nSent2Vec: 68.58 +- 0.58\n\nSpanish:\nBanyan: 60.95 +- 0.76 \nSent2Vec: 55.15 +- 0.54\n\nSent2Vec requires a sentence tokenised corpus as input and finding and then running tokenisers for all the languages is proving to be quite time consuming and in some cases challenging. We are however happy to work on filling out a sent2vec baseline for all languages if you strongly feel this would improve the paper. \n\nSuggestions: Thank you for pointing these out, all are good improvements and we will edit accordingly! \n\nC > CE: It depends what you mean by brittle?  Using the contrastive loss does lead to consistent performance. However, it can lead to certain unfavourable behaviour whereby tokens like ‘the’ are excessively pushed away from all other embeddings which introduces oddities in the structure. However, C is still superior to CE because it adds a whitening effect, which is vital for tasks like STS. To retain whitening while using CE we had to switch to the diagonal functions. These are simple and therefore need easily separable embeddings to still do well on reconstruction. \n\nThe Trees:  To an extent they do seem reasonable, although our merge algorithm is context free which is a limitation on the types of structure. It is maybe best to think of them as akin to running BPE all the way till the whole sequence is compressed. Qualitatively we do find that this leads to some reasonable patterns, and consistent behaviours such as segmenting phrases can be observed. We would be happy to include some examples in the appendix in the final version!\n\nThank you for your constructive feedback and time taken with the paper, we look forward to engaging with you during the rebuttal period!  If you feel like our rebuttal has addressed your concerns please consider raising your score.", "claims_and_evidence": "- The main claim is that Banyan outperforms various baselines (word embeddings, Self-StrAE, and transformers) in representation learning for low-resource settings. This claim is well-supported by evidence, and the authors perform thorough evaluation on a very wide range of tasks across many languages.\n\n- The other main claim in the paper is that the proposed modifications to Self-StrAE are beneficial (entangled graphs, diagonalized functions, replacing contrastive loss with cross entropy). This claim is also well-supported by thorough ablations.\n\n- The main limitation of the experiments is that the non-neural baselines are relatively weak. In particular, there are many well-known sentence embedding approaches that are not cited or compared to (please see \"Essential references not discussed\"). The lack of such baselines undermines the claim that Banyan represents a breakthrough for producing representations in low-resource settings.", "methods_and_evaluation_criteria": "- The method makes sense and the modifications to Self-StrAE are well-motivated. The efficiency of the method also makes it well-suited for low-resource settings.\n\n- The benchmarks chosen are also reasonable in that they study (1) how well the representations match human judgments of semantic similarity, (2) how well the representations perform in retrieval, and (3) how well the representations perform for classification. Banyan performs well for (1) and (2) and is slightly worse than baselines for (3).", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design (English experiments, multilingual evaluation, ablations) all seem sound. The main limitation of the results is the weak non-neural baseline (in particular, the lack of sentence embedding methods), as discussed below. The paper also lacks any non-neural baselines in the multilingual evaluations (Table 4), omitting the word embedding baseline present in the English experiments.", "relation_to_broader_scientific_literature": "The paper proposes a new architecture for structured representation learning which is novel, and it does a good job of positioning and motivating the proposed changes with respect to past work (Self-StrAE). However, it is somewhat unclear how this method relates to simpler existing approaches for sentence embeddings (please see the next section).", "essential_references_not_discussed": "The main set of missing references (and baselines) is the large body of past work on sentence embeddings. While the method proposed in this paper is novel with respect to these past works in that it induces structure, it still competes with these sentence embedding methods as a way to produce sentence representations in low-resource settings.\n\nThe following are a few methods that are most related to their setting:\n- [Arora, Liang, Ma (2017)] (https://oar.princeton.edu/bitstream/88435/pr1rk2k/1/BaselineSentenceEmbedding.pdf), and its extension [Ethayarajh (2018)] (https://aclanthology.org/W18-3012.pdf): at a high level, these methods take a weighted average of the word embeddings and apply an SVD\n- [Pagliardini, Gupta, Jaggi (2017)] (https://arxiv.org/pdf/1703.02507): this method (sent2vec) extends the word2vec objective to sentences\n- [Ruckle et al. (2018)] (https://arxiv.org/pdf/1803.01400): they show that averaging embeddings with a power mean (e.g. max pooling) and doing concatenation performs better than just taking the average.\n\nIn particular, the STS numbers reported in these papers are much higher than those of the GloVe baseline presented in this work.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "- Overall, the paper is very well-written, but section 5.4 is slightly harder to follow than the rest of the paper (but it is still clear enough).\n- In my first pass through the paper, I didn't realize that Banyan used CE loss instead of contrastive loss. Maybe that is worth mentioning at the end of Section 4.\n- In Section 4, I think it's also worth briefly describing how K and U relate to word embedding dimension, and what it means to have independent channels.\n\nTypos:\n- 163: ask -> asks\n- 383: lightweigh -> lightweight\n- 426: elegant. -> elegant,", "questions_for_authors": "- I am a bit confused that switching from C to CE loss on standard trees is better (Table 5), given that the Strae paper found the opposite conclusion. My guess is that C > CE for supervised StrAE, but CE > C for Self-StrAE, which makes intuitive sense because supervising intermediate nodes directly can be brittle if the trees are self-induced and slightly wrong. Is this the correct intuition?\n\n- I am curious about the induced trees, whether they seem qualitatively reasonable, and whether they match human-designed trees (e.g. on the Penn Treebank). In particular, if the method does in fact recover reasonable trees, that would set it apart from previous sentence embedding methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4GWLavGN7F", "reviewer": "Reviewer_Kous", "summary": "This paper introduces Banyan: a new recursive graph neural network for learning text representations in low-resource languages. This model extends previous work by building nested trees over sequences that share the same tokens. In Banyan the same tokens will have the same tree node, even if they come from different sequences. For scalability reasons, the trees are constructed from a batch of samples rather than from an entire dataset. Embeddings are learned from a simplified message-passing algorithm that traverses the trees both in bottom-up and top-down directions. Having nested trees provides multiple advantages, notably the reduction of duplicated nodes and multiple context representations within the same node. These advantages translate to strong semantic representations in both English (when compared to RoBERTa & GLOVE) and lower-resourced languages (when compared to XLM-R (fine-tuned), Llama 3.1 8B, Mistral Nemo 12B, MiniL12, and Paraphrase XLM-R).\n\n## update after rebuttal", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback regarding the paper!\n\nThe overall logic for our experiments is as follows:\n\nWe care about whether we can create a resource efficient model by applying an inductive bias. This can be very useful for low resource languages. However, such languages do not have many high quality labelled datasets. The SemEval STS benchmark is the only wide ranging benchmark that we know off. What we want to do is take a high resource language (i.e. English) with a lot of labelled datasets and use that to establish whether we can learn high quality embeddings. STS provides one view of that. However, we should also make certain that Banyan succeeds at other tasks you might care about. The retrieval and classification tasks are there to demonstrate that the embeddings learned by Banyan capture multifaceted aspects of semantics. More concretely:\n\nQuora: This is about matching questions to answers. Crucial for applications like RAG and lets us see whether our embeddings capture the response relation.\n\nArguana: This requires matching arguments to counter arguments. It lets us see whether our semantic space captures the notion of dialectical opposition. \n\nSST: Sentiment classification - does our representation space capture semantic polarity. \n\nMRPC: Paraphrase detection: does our representation space capture semantic equivalence. \n\nMoreover, these tasks utilise measures other than correlation, which provides a broader basis for measuring the success of embeddings. By demonstrating broad spectrum capabilities in English, and then showing similar trends but under a more limited evaluation in the low resource languages, we aim to show that Banyan a) learns effective representations b) can therefore provide a solution in case where other methods that require scale are inadequate.\n\n\nYou are right that this could be better clarified and we will make sure to use the extra page if accepted to do so! Let us know if you have any further questions, and we look forward to engaging with you during the discussion period.", "claims_and_evidence": "yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation benchmarks make sense for the application at hand.", "theoretical_claims": "I did not check the correctness of any proofs or theoretical claims as I did not find any.", "experimental_designs_or_analyses": "Experimental designs are sound. I checked their sentence and word level evaluation as well as their ablation study, and retrieval and classification tasks on English. I also checked their multi-lingual results.", "relation_to_broader_scientific_literature": "The contribution of this paper is related to the STRAE model (Opper et al., 2023) published at EMNLP in 2023. This previous work is a similar, tree-based, sentence-level embedding method, that models compositional semantics with minimal data and model size requirements. Banyan differentiates itself by introducing entangled trees, which provides better performance and resource efficiency.", "essential_references_not_discussed": "No essential reference missing to the best of my knowledge", "other_strengths_and_weaknesses": "### Strengths\n\nThis paper introduces a novel recursive model that learns textual representations and its learning mechanism. The proposed architecture is novel and seems promising as it yields good results when compared to other more classical methods. In addition, the proposed method is very efficient: it requires very little training and has only 14 non-embedding parameters.\n\n### Weaknesses\n\nThe task used to evaluate models is not immediately clear from the abstract or the introduction. While Semantic Textual Similarity - STS is mentioned in the introduction, it is not well explained what the objective is. The paper could gain clarity by explaining briefly what the goal is: having cosine similarities that map sentence pairs in a similar way to human judgements. \n\nTable 3 shows results for retrieval and classification tasks from the beir and glue benchmarks, but very little information is given regarding the experimental setup and the result analysis. The paper would be stronger with detailed experimental design choices and result analysis on these benchmarks.", "other_comments_or_suggestions": "In table 3, the last two columns should also bold the best numbers (Banyan @ 77.2 accuracy for sst-2 and Glove & RoBERTa @ 81 for MRPC)", "questions_for_authors": "No additional questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Banyan, a graph-based autoencoder for learning sentence representations, particularly targeting low-resource languages. Banyan builds on prior work (Self-StrAE) by incorporating "entangled trees" (sharing identical spans across sentences in a batch) and using minimal gating parameters, resulting in a model with few non-embedding parameters. The efficiency and strong performance demonstrated on Semantic Textual Similarity (STS) and retrieval tasks, especially in low-resource settings compared to large transformer models, were recognized and appreciated by all reviewers.

The review process highlighted one major concern regarding the lack of comparison to relevant simple sentence embedding baselines (like Sent2Vec) suitable for low-resource scenarios and related work in general as it largely goes from word embeddings to pre-trained models. The authors provided a rebuttal, presenting new experimental results comparing Banyan against Sent2Vec (trained under comparable conditions). These results showed Banyan maintaining an edge. One other line of work worth mentioning (See "Paraphrastic Representations at Scale" for instance and original paper from 2015) first showed the effectiveness of averaging embeddings for textual similarity in transfer learning which has been shown to be competitive and outperform large pre-trained neural networks even today - just like Banyan. Especially with the rise of LLMs, assuming only a (non-parallel) corpus to learn embeddings can be weakened since it is easy enough to get some form of weak supervision with parallel data. This is because there is lots of translation data and LLMs can also translate themselves without being explicitly trained.

For the final version, the authors must incorporate the new baseline results (Sent2Vec, etc.) and analysis discussed in the rebuttal. They must also follow through on their commitment to enhance clarity. Addressing these points will strengthen the final paper.
Decision: Accept (poster)