StrAE: Autoencoding for Pre-Trained Embeddings using Explicit Structure
Authors: Mattia Opper
Victor Prokhorov
Siddharth N
Venue: EMNLP 2023 Main (2023)
Keywords: NLP
Representation Learning
Structure
Semantics
Syntax
Induction
Composition
TL;DR: We investigate how explicit structure affects embedding quality by developing an autoencoder, we then introduce an inductive bias and show that even letting the model define its own compositions is beneficial.

Abstract: This work presents StrAE: a Structured Autoencoder framework that through strict adherence to explicit structure, and use of a novel contrastive objective over tree-structured representations, enables effective learning of multi-level representations. Through comparison over different forms of structure, we verify that our results are directly attributable to the informativeness of the structure provided as input, and show that this is not the case for existing tree models. We then further extend StrAE to allow the model to define its own compositions using a simple localised-merge algorithm. This variant, called Self-StrAE, outperforms baselines that don't involve explicit hierarchical compositions, and is comparable to models given informative structure (e.g. constituency parses). Our experiments are conducted in a data-constrained (circa 10M tokens) setting to help tease apart the contribution of the inductive bias to effective learning. However, we find that this framework can be robust to scale, and when extended to a much larger dataset (circa 100M tokens), our 430 parameter model performs comparably to a 6-layer RoBERTa many orders of magnitude larger in size. Our findings support the utility of incorporating explicit composition as an inductive bias for effective representation learning.

StrAE: Autoencoding for Pre-Traine Embeddings using Explicit Structure
Mattia Opper* and Victor Prokhorov and N. Siddharth u.b a University of Edinburgh, UK; The Alan Turing Institute, UK {m. opper, vprokhor, n. siddharth)@ed.. uc.uk
Abstract
composition. However, the models that have come to dominate NLP in recent years generally do not explicitly take this property into account.
This work presents StrAE: a Structured Autoen- coder framework that through strict adherence to explicit structure, and use of a novel con- trastive objective over tree-structured represen- tations, enables effective learning of multi-level representations. Through comparison over dif- ferent forms of structure, we verify that our results are directly attributable to the informa- tiveness of the structure provided as input, and show that this is not the case for existing tree models. We then further extend StrAE to allow the model to define its own compositions us- ing simple localised-merge algorithm. This variant, called Self-StrAE outperforms base- lines that don't involve explicit hierarchical compositions, and is comparable to models given informative structure (e.g. constituency parses). Our experiments are conducted in data-constrained (<> 10M tokens) setting to help tease apart the contribution of the inductive bias effective learning, However, we find that this framework can be robust to scale, and when extended to a much larger dataset (+= 100M to- kens), our 430 parameter model performs com- parably to a 6-layer RoBERTa many orders of magnitude larger in size. Our findings support the utility of incorporating explicit composition as an inductive bias for effective representation learning.
Transformers (Vaswani al., 2017) have the ca- pacity to model hierarchical compositions through the self-attention mechanism, whereby tokens can come to represent varying degrees of the surround- ing context. However, should such behaviour oc- cur, it is incidental to the training process, as there is no strict requirement for tokens represent higher-order objects, and tokens never explicitly merged and compressed with one another. Whether Transformers are able to acquire knowledge of syn- tax, as understood from the linguistics perspective, remains unclear (Kim et al., 2020). However, there is evidence that these models do become increas- ingly tree-like with sufficient scale and training steps (Jawahar et al., 2019; Murty et al., 2022). This raises an interesting question. To what extent is this drive towards tree-likeness responsible for their representation-leariig capability? Further- more, evidence from Lake et al. (2017) suggests that incorporating appropriate inductive biases to- wards composition can help bridge the stark dis- parity between the quantity of data required for learning between ML models and humans, and al- low computational models generalise better. If is indeed an inductive bias towards hierarchical composition that enables humans to acquire multi- level semantics efficiently, and NLP architectures generally aren't explicitly tasked with modelling such structure, what happens when they are?
1 Introduction
Human understanding of natural language is gener- ally attributed to the understanding of composition. The theory that smaller constituents (words, to- kens) recursively combine into larger constituents (phrases, sentences) in a hierarchically structured manner, and that it is our knowledge of said struc- ture that drives understanding (Chomsky, 1956; Crain and Nakayama, 1987: de Marneffe et al., 2006; Pallier et al., 2011). Compositionality can help drive efficient and effective learning of seman- tics as it presumes knowledge only of the immedi- ate constituents a phrase and the process of their
To investigate this, we develop StrAE, a Structured AutoEncoder framework. It takes as input a tree structure that represents a hierarchical composition process, and uses it to directly specify the flow of information from leaves to root (en- coder) and back from root leaves (decoder). To disentangle the effect of structure from other con- founding factors, we constrain StrAE to only have access to the information immediately provided to

---

it by the input tree. This means that for given node the model can only use information from the nodes immediately connected to (constituents): a property we refer to as faithfulness. Following this, we investigate which training objectives are best suited to enable structured multi-level representa- tion learning, and present novel application of the contrastive loss to hierarchical tree-like structure.
coo eubedding shared
Eo
C
D
Ea
To investigate the utility of compositional struc- ture for representation learning, we employ a data- constrained setting (=10 million tokens), and eval- uate on a series of tasks that measure semantics at both the sentence and word level. We compare the representations learned by StrAE to series of baselines consisting of other tree- models which do not enforce faithfulness as rigidly as StrAE and series of baselines that do use explicit structure at all. We also verify that our performance is at- tributable to the form of composition expressed by the tree structures by comparing results across a range of different input structures.
sins (Humer)
F3
64
Fly
fg
NOT
AgT
wa doughnuts
(ale)
(dogghnudd)
Figure 1: StrAE operation to encode and decode a sen-
tence. Shared colours indicate shared parameters.
recover the sentence itself-eefeccivvely structured autoencoder. While this generates embeddings for multiple levels of the sentence dictated by the structure, it also generates embeddings for node in two ciirections-compeee posing downwards. The composition embeddings represent the local context for a node, while the de- composition embeddings represent its full context given the sequence.
upwards and decom-
Finally, to investigate how useful simple bias towards hierarchical composition is, we extend StrAE to allow it to define its own compositional hi- erarchy. This variant, called StrAA,, utilises the learned representations and the encoder to define its own "merge" sequence, which is then employed the tree structure for the decoder.
Denoting embeddings e; E R N we distin- guish embeddings formed traversing upwards and downwards e; and e; respectively. Encodings for the tokens are denoted the vertices wo AV, in a V-simplex for vocabulary size V. Note that while the token encodings ut are effectively one- hot, reconstructions W; can be any point on the simplex-interprrteed as a distribution over vertices, and thus the vocabulary, with an argmax retrieving the appropriate token. We define the four core com- ponents of StrAE as
Our results indicate that knowledge of struc- ture is indeed beneficial, and that surprisingly even a simple bias for hierarchical composition leads to promising results. In light of these find- ings, we then extended our experiments to the 100 million token range and analyse how well Self-StrAE performs in significantly larger set- ting. Even more surprisingly, despite solely having 430 non-embedding parameters, Self-StrAB is able achieve comparable performance to a layer RoBERTa (Liu al., 2019) model with 3.95 mil- lion parameters.
Sy AV H RNN
(embedding)
C R NxN x NXN + NNN
(composition)
Do R.N. N NNN
2 Model
(decomposition)
Avr NNN AV
(indexing)
We develop a framework (StrAE) that processes a given sentence to generate multi-level embeddings by faithfully conforming to the given structure. In- tuitively, it involves embedding the tokens of a sen- tence onto the leaves of the structure, composing these embeddings while traversing up the structure to generate the full-sentence embedding, and then traversing back down the structure while decompos- ing embeddings from parent to children, to finally
where the subscripts on the functions denote learn- able parameters. Table shows the model compo- nents and their associated parameters. We assume square embeddings during composition and decom- position, to allow for independent channels to cap- ture different aspects of meaning. Figure I shows how the model operates over a given sentence.

---

Table I: Definitions model components following $ 2. Functions square and flatten transform between column vectors and square matrices. Functions hcat and hsspit perform horizontal concatenation and (middle) splitting respectively, a--) denotes the softmax function. e and denote additive biases.
to each other and dissimilar all negative exam- ples. To adapt its use for structured embeddings, we apply the objective so as to task the model with maximising the similarity between corresponding upwards and downwards embeddings (7e, Ci) while simultancously minimising the similarity between all other embeddings. This has the additional at- tractive characteristic that it forces amortisation of the upwards embeddings é innooooprratinng con- text from the sentences the word or -enmme might have occurred in, as the corresponding down- ward embedding e has full-sentence information it. Figure illustrates an example of this objective.
(ux) = square( un M Colee Foz) heat(fe,, P2y +
RUXN
RANNN RN
De(cp) lsslit(e,,)) +++0
D E DAN V VERVAN RUN
Agr (e.) (flatteen()))
90
50
Ca
D
For given batch of sentences 8j, we denote the total number of nodes (internal + leaves) in the associated structure as M. We construct a pairwise similarity matrix A E RMMx between normalised upward embeddings (ei) and nor- malised downward embeddings (ex) using the cosine similarity metric (with appropriate flatten- ing of embeddings). Denoting A Aoj, Aig the 8th row, jth column, and (t,j)th entry of matrix respectively, we define
E2
22
A-pT
101
Homer
Ely
Agg
Agu
112 452
MIS donghituts
(loughutte)
Figure 2: Contrastive objective over structure: corre- sponding node (green) is pulled closer, and other nodes (red) are pushed away.
M
M log or (Aaj)
-1 2M
Coont
log or (Ai)) +
> j=1
2.1 Objectives
(2)
Given this autoencoding framework, a natural ob- jective to employ is the cross entropy (CE), which simply measures at each leaf node how likely the target token wy is given the reconstructed distribu- tion wi over the vocabulary. Given sentence sj (mx) 10
where (-)- is the tempered softmax (tempera- ture T), normalising over the unspecified (.) dimen- sion. Note that Lcont extends to the batch setting. Finally, initialise our embedding matrix from uniform distribution with the hyper-paramettr denoting range- range- corresponding to the finding that contrastive loss seeks to promote uniformity and alignment as established by Wang and Isola (2020).
Tj this objective is formulated as
T W. log wi
1 LCE
T₃
Experimental Setup
However, the CE objective places fairly min- imal constraints on the embeddings themselves; it only requires the leaf embeddings to be 'close enough' for retrieval to be successful. The upward and downward intermediate embeddings are wholly unconstrained, and these can end up being quite different. Given we are learning embeddings for the whole tree, we would like to define an objective over all levels, not just the leaves.
We divide our experiments into two separate sec- tions, but both share the same overall setup for pre- training data and evaluation. For pre-training data we take a 500k sentence (== 10M tokens) subset of English Wikipedia and 40k sentence develop- ment set. We restrict our pre-training data to this scale in order to measure our efficiency hypothe- sis presented in the introduction. For evaluation, assess on three categories of tasks: word level semantics, sentence level semantics, and sentence pair classification, with each aiming to capture sep- arate areas of semantic understanding. On the word level we use Simlex (Hill et al., 2015), Wordsim
For this purpose, we turn contrastive loss (Chen et al., 2020; Radford et al., 2021; Shi al., 2020). Contrastive loss involves optimising the embeddings of target pairs so that they are similar

---

Table 2: Overview of Evaluation Tasks
the Tree-LSTM (Tai al., 2015). Both architec- tures take structure as input and are able traverse through the tree to learn representations. However, they differ in the constraints they impose on infor- mation flow. We conduct our experiments along two different axes: how well do the representa- tions perform, and to what extent do the models discriminate between input structures?
Name
Level
Task
Requires Classilier?
SimLex
Word Level
Semantic Similarity
No
WordSim S
Word Level
Semantic Similarity
No
WordSim
Word Lovel
Semantic Relisedness
No
STS 12
Sentence Level
Semantic Similarity
No
STS
Sentence
Semantic Similarity
No
STS B
Sentence Level
Semantic Similarity
No
SICK R
Sentence Level
Semantic Relatednces
No
MRPC
Sentence Level
Pamphraas Detection
Yes
RTE
Sentence Level
Texhual entailment
Yes
To achieve these evaluations, we parse our pre- training set into three kinds of structure. The first are constituency trees extracted from CoreNLP (Manning et al., 2014) and binarised using NLTK (Bird et al., 2009). The two other kinds are purely right-branching and balanced binary trees, which we extract using standard algorithms. The result- ing structures are then converted DGL graphs (Wang et al., 2020). paaeer-prraeeter descriptions for each model can be found in Appendix A.
S and Wordsim R (Agirre et al., 2009). On the sentence level, we use three tasks from the STS suite (Agirre et al., 2016, 2012; Cer et al., 2017), the SICK relatedness dataset (Marelli et al., 2014), and for sentence pair classification tasks we use RTE and MRPC taken from the GLUE Benchmark (Wang al., 2019a). Table 2 provides an overview. Note that a subset of the tasks distinguish between similarity and relatedness. Briefly, the former mea- sures semantic similarity, as between "running" and "dancing". as both words act as verbs. Relatedness measures semantic relationships such as between "running" and "Nike" where the words often co- occur together, but belong to different grammatical categories. Finally, note that Simlex only mea- sures similarity the exclusion of relatedness.
Baselines
The Inside-Outside Recursive Neural Network (IORNN) processes data hierarchically, working from the "inside" to the "outside". At each node in the tree, an IORNN maintains two vectors: The inside vector é, which represents the local meaning of given node, obtained by composing up the tree, The outside vector e, represents the context for the given node obtained by decomposing down the tree. While superficially similar to StrAE, the models differ in an important aspect. For given parent p and children cl, c2 the outside representation:
All tasks apart from the final two classification tasks are measured using the Spearman correlation of the cosine similarity of model embeddings for each pair and human judgements; classification tasks are measured using accuracy. To emphasise the impact of the pre-training with structure, do not fine-tune any models for the evaluation tasks, instead keeping them frozen. Where a classifier is required, we fine-tune only a task-specific classi- fication head consisting of a FFN with a 512 di- mensional hidden layer and an intermediate Tanh activation function. We choose this setup to match the GLUE Benchmark. For all experiments, we pre- train the models across 5 random seeds and present the averaged performance. Downstream classifiers are themselves also trained across 5 random seeds and the average reported. Additionally, for all ex- periments we set the embedding dimension to 100, Finally, for all models trained on the word level we filter the vocabulary to exclude words occurring fewer than two times in the data.
StrAE:
Cet, 622 = hsplit(cp@)), with θ E IR Nx2N
IORNN:
Ea = tanh 22) with θ E R
Ex2 tanh (a)
where [ denotes concatenation. In StrAE the outside representation solely depends on the parent and therefore enforces a compression bottleneck at the root of the structure. No other information may be shared from the composition process, and all e embeddings are created recursively based on the root. The outside vector for IORNN is derived from both the parent and the inside vector of a given child's sibling node. As the root has no sib- lings or parent nodes, the outside vector consists of global bias parameter, intended to represent the context of the whole pre-training corpus. Conse- quently, IORNN does not enforce a compression bottleneck and information flows between both the local compositional (bottom up) and the global de- compositional (top down) contexts.
4 Comparing Tree Models
Here, we compare StrAE with two existing tree- architectures. These are the IORNN (Drozdov et al., 2020, 2019; Le and Zuidema, 2014) and

---

Table Comparison of StrÅF, IORNN and Tree-I.STM embedding performance on our evaluation suite. Higher is better. All tasks that Spearman's 's p have the result * 100 and are marked with a t. Score represents the average across all tasks. All models were trained over five random seeds using constituency parses as input. The pre- pre-training objective is indicated by C or CE, representing contrastive loss or cross entropy respectively. Only results where there is no standard deviation overlap between model performance are bolded.
Model
Simllex 1
Wordsim S
Wordslm R
STS 12
STS 16
STS B
SICK R
MRPC
RTR
Score
StAAB
15.54 1: 0.25
56.1 1: 0.47
43.77 =: 1.01
34.59 =: 2.58
50.4 1: 0.97
41.83 :: 2.23
50.3 1: 0.88
67.24 1: 0.39
56.16 : 1.39
46.21
SMAB
17.57 1 1.08
50.72 a 297
36.73
6.19
5.02 2 1.1
25,86 = 1.03
7.76 0.95
39.06 1.51
67.48 0.
53.6 1 0:46
33.76
1ORNNC
1.2++0.25
27.29 + 0.57
11.29
0.5
4.2 + 0,97
17.47 = 1.14
1.87 +0.55
30.48 + 0.55
67.08 + 0.16
+.44
23.13
JORNN CR
24.36 +
57.53 1.59
36.55
1-3++100 1.78
.0.17 0.9
22.68 0.8
5.04 0.55
38.11 0.63
67.24
0.63
54.28 0.55
33,99
Tee--LTM C
6.4
37.39 12
1.49
-1.34 4: 1.33
23.95 =: 0:45
4.25 4: 1.11
$2.9 0.35
68.16 0.32
52.76 4: 1.09
27.24
Tee-LSTM
1: 2.01
3.22
33.24
1 1.29
19.38
121
L04
34.55
52.5 1:
Our second baseline, the Tree-LSTM, is a recur- sive variant of the LSTM (Hochreiter and Schmid- huber, 1997). The main difference is that inputs are processed recursively rather than recurrently. At each node, the inputs for the cell are the children's s hidden and cell states. Here the flow of information differs to StrAE because while StrAE has to com- press embeddings according to the order dictated by the input structure, the Tree-LSTM is able to se- lectively retain information from lower down in the tree according to the cell state. Tree-LSTMs can be applied both bottom up (Choi et al., 2017; Maillard et al., 2017: Tai et al., 2015) encoders, and top down as decoders (Dong and Lapata, 2016; Xiao et al., 2022). Consequently, they can be pre-trained in the same way as StrAE. Parameter counts for all models can be found in Table 5. Performance on Constituency Parses
it. The objective asks the model to reconstruct each node embedding such that it is distinctive from all other embeddings in the batch. If the LSTM is retaining information in its cell state, it makes this distinction difficult to enforce. IORNN faces diffi- culties in two regards: the outside representation of the root a global parameter which makes distinc- tion significantly more difficult, and the sharing of information between sibling nodes. We tested the effects of removing the sharing of information and still found little improvement, possibly because the degree of information sharing between inside and outside representations is so high that it renders the requirement for meaningful compression void.
Evidence of IORNN's reliance on the informa- tion sharing from inside sibling nodes can be seen in its Simlex performance with the cross entropy objective. Simlex actively penalises capturing se- mantic relatedness, which in the case of IORNN is information that would be provided by the outside vector. Its poor performance indicates that IORNN is not using the parent outside representation sig- objectives for the sake of parity with StrAE. Our re- sults demonstrate that while all models are able to capture word level semantics to some degree, it is only StrAE coupled with the contrastive objective that is able to extend this sentence level seman- tics, though StrAE with the cross entropy objective still performs better than the other architcctures in this regard. These results indicate that enforcing faithfulness is beneficial in learning multilevel rep- resentations. It also follows that the contrastive objective is beneficial for StrAE as it is directly ap- plied to all nodes, and therefore directly optimises the root's relations other sentences. In the classi- fication tasks, there appears be little difference between architectures.
We first evaluate model performance purely on the constituency parse structure, as in theory this should be the most informative. Results can be found in Table 3. We train all models using both
nificantly when reconstructing embeddings, which may explain the difficulties on the sentence level tasks as the model can simply try to predict a word given its immediate left or right neighbour.
Performance Across Input Structures
We also evaluate how dependent the performance of each model on input structure. We com- pare the performance of each model between con- stituency parses, fightbbranchin and balanced bi- nary trees as input. Plots of these results are in Fig. 3, and the full tables can be seen in Ap- pendix D. StrAE is dependent on input structure for performance, particularly when using the con- trastive objective, though this still holds true across all task areas even for cross entropy. On the other hand, the baseline models are not, though this varies in degree. IORNN trained with cross en- tropy actually performs best with branchnrnn
clear that the contrastive objective is only useful when the model imposes constraints on infor- mation flow, as both baselines do not benefit from

---

EMCC KPR LPMC UmN.E
WHARO maal
GALC M8A
UPMC
unw
SPARC
107009
Lexical Semantics Figure 3: Average performance for models on different task areas by structure type, higher is better.
Overall
(c) Sentence Level Semantics
(d) Classification
structure, though this is skewed by word level per- formance. Conversely, the LSTM appears dis- criminate, but this is solely due to word level results and does not extend to other areas evaluation. Summary
#2
StrAE is tasked with taking sequence and com- pressing it into single representation through or- dered merges dictated by the input tree. Each non-leaf node acts as compression bottleneck. As a result, the contribution of each input token the final root representation is directly dependent on the merge order. This makes StrAE structure sen- sitive. When coupled with contrastive loss, StrAE must learn embeddings such that similarity is de- pendent how sequences compose. This is be- cause the objective is non-leaf node embeddings defined by merge order. Sccondly, requires that the merge order have some degree consistency to enable recon- struction, something which purely right-branching and balanced binary trees do not provide. is the combination of strict compositional bottlenecking and the contrastive objective that enables StrAE to learn effective multi-level representations and serve probe for the utility of structure, unlike the structured baselines.
0..
Ca
0.4
0.6
$2.
12
doughnuts
Figure 4: Self-StrAB composition. Red dotted arows indicate cosine similarity between nodes.
all nodes, and all
this is dictated by the embeddings. Self-StrAE or- ders compositions according to cosine similarity between adjacent node embeddings in the frontier, merging the argmax each step. Figure Fig, 4 shows this process. In the figure, the similarity between 'ate' and 'doughnuts' is greater than that of 'ate' to 'Homer', so the model first merges 'ate doughnuts' into single embedding using the com- position function. At the next step, the model merges 'Homer' and "ate doughnuts" into single embedding and arrives at the root. The algorithm is provided in Appendix B.
5 Comparison to Unstructured Baselines
We save the merge history in an adjacency ma- trix that by the time the encoder reaches the root node, the adjacency matrix represents the whole tree over the input sequence. We then convert the adjacency matrix into a DGL graph, and pass that as input to the decoder. The decoder operates ex- actly the same as in vanilla StrAE, only that this case the input graph is defined by the encoder's merge history opposed to, c.g., syntactic parser. Baselines
Here, compare StrAE to a series of baselines that not provided any form of parse tree input. The aim evaluate the usefulness of explicit structure against other inductive biases. We also introduce& a variant StrAE called Self-StrAE that is not provided structure as input but must learn its own mode of composition. It serves as a measure of the utility of an explicit compositional bottleneck. Self-StrAE
Self-StrAB SelffStructuring AutoEncoder) mod- ifies the encoder so uses greedy agglomerative clustering order decide which tokens to com- pose. Unlike StrAE where the order of composi- tions is defined by the input structure, in Self-StrAE
We selected Fasttext (Bojanowski al., 2016), Bi-LSTM (Hochreiter and Schmidhuber, 1997) and RoBERTa (Liu et al., 2019) as baselines. Fasttext leverages distributional semantics and subword in- formation to learn word embeddings, but only op-

---

Table 4: Comparison to unstructured baselines. Higher is better. All tasks that use Spearman's 's p have the result 100 and are marked with f. Score represents the average across all tasks. All models were trained over five random seeds. Only results where there is no standard deviation overlap between model performance are bolded.
Model
Simlex
Wordsim S
Wordsim R
STS f
STS 16
STS F
SICK R
MRPC
RTE
Score
SMAR Systsetic
15:54 + 0.25
56.1 0:47
43.77 1.01
34.59 2,58
50. 0.97
41.83 2.23
50.3 0.88
67.24
56.16 + 1.59
46.21
Self-StrAF RPF
13.04 + 0.07
48.19 + 1.03
45.47 + 0.83
34.42 0.73
49.93 + 0.38
36.68 0.64
51 0.3
68.91 0.2
54.42 + 0.49
44.67
Self-StrAE Word
+ 0.12
+ 0.39
48.76
0.86
33.23 0.38
49.91
27.97
0.52
68.13
0.53
Tasttext
25.8 1: 0.16
50.8 1 0.24
29.18 1: 0.22
4.35 = 0.5
32,43 0.1
16.93
0:0.9
41.58 1 0.05
69.1 1: 0.12
54.4 14.4.10..87
36.06
HS-L Word
23.74 1.17
2.44
1.73
34,86
13:46
42.36
68.95
0.95
LS.LSTM BPE
12.88 4: 0.77
39,46 à 2.17
32.22 à: 3.07
9:22 = 0.87
33.78
à: 1.66
14.06
a: 2.01
40.36
0.27
68.21 +: 0.41
53.9 4: 0.96
33.79
RoBERTa
9,92
26.6 4: 6.71
3.81
29:48 3.28
50.88 1.11
38.36
1.9
49.58 0.91
69.19 0.27
54,86 0.49
Table 5: Number of parameters for our models and base- lines. Following convention we exclude the embedding matrix (and LM head if applicable) from the count.
erates over a fixed window size. The Bi-LSTM al- lows us to measure the utility of sequential vs hier- archical information processing. Finally, RoBERTa serves as a suitable Transformer baseline, because it only utilises the MLM objective, rather using NSP as with BERT (Devlin et al., 2019), and this has been shown to be more robust. Furthermore. it does not require additional data labelling, such identifying which sentences follow each other, which provides greater parity with Self-StrAE and Fasttext as both models do not have such additional labels as input. For both models, we set the em- bedding dimensionality to 100 to match StrAE and set the number of attention heads in RoBERTa to 10 in order to match StrAE's channels. We set the number of layers in RoBERTa to 6 as, in data constrained setting there was no additional benefit observed with greater number, and the parameter disparity between StrAE and RoBERTa is already substantial (see Table 5 for parameter counts for both the tree and unstructured baselines). The other hyperparameter details for all baselines can be found in Appendix A. To produce sentence embeddings we take the mean of the word em- beddings with Fasttext and the mean of the final layer token representations for RoBERTa. To pro- duce word embeddings for RoBERTa we follow the lessons from Jawahar et al. (2019); Vulic et al. (2020), which state that lcxical information is con- tained in the lower layers. For cases where a word is broken into multiple subwords, we take the av- erage of the embeddings from layers 0-2. Where a word is present in the vocabulary, we simply use its embedding, as there is context to provide it with. In the case of the Bi-LSTM we produce sentence embeddings by passing the concutenated final hidden states for both directions through a learned linear layer in order to produce a single 100-dimensional embedding. We found this to lead to improved performance compared to simply us- ing the concatenation. The same strategy is used
Model
(Self:]StrAF
IORNN
Tree-I.STM
Ri-LSTM
RoBERTa
Params
430
40,400
260,600
181,800
3,950,232
to produce word embeddings in the case where it is broken into multiple subwords. We contrast these models with StrAE trained on constituency parses with contrastive loss, Self-StrAE trained on the word level using the same vocabulary as before, and finally Self-StrAF starting from the subword level. We take the vocabulary from the same BPE (Sennrich et al., 2016) tokeniser used by RoBERTa (minus the special tokens) with total size of 25000, We train Self-Stra AS with the contrastive objective because this proved significantly more effective.
10 Million Tokens
As shown in Table 4, while StrAE performs best overall, Self-StrAE is able to achieve comparable performance across both the word and sentence level. Fasttext performs best on Simlex, but this is likely to be for the same reasons as IORNN i.c., cap- turing similarity at the expense of relatedness. This is also indicated by its comparative lower Wordsim R performance. On both other lexical semantics tasks StrAE and Self-StrAE both outperform it. Fasttext also struggles on the sentence level. Simi- larly, the Bi-LSTM performs well on lexical seman- tics, but struggles at capturing higher levels. This is evidenced both by the STS results and lower lex- ical performance of the BPE Bi-LSTM compared with Self-StrAE. ROBERTa performs comparably to StrAE on STS, but struggles on the word level. This might simply be because Transformers aren't designed to learn static lexical embeddings, but could also be the result of our data-constrained pre- training setting, as there was significant variability between seeds. Consequently, we conducted a final experiment using a significantly larger pre-train pre-training set to assess the impact of scale.

---

Table 6: WikiText- 103 Results. Higher is better. All tasks that use Spearman's ' p have the result 100 and are marked with f. Score represents the average across all tasks. All models were trained over five random seeds. Only results where there is no standard deviation overlap between model performance are bolded.
Model
Simlex f
Wordsim S
Wordsim R
S1812
STS 16
STS B
SICK R
MRPC
RTE
Score
RoRERTa 1.02
46.38
26.12
35.38
52.64 1.11
50.68
0.43
53.71
Self-StrAE
13.41 1 0.66
47,06 0.61
42,53 1.71
46.64 0.23
52.08 1 0.37
40.59 0.83
51.4 0,4
68.35
0.46
55.87
0.3 46.39
100 Million tokens
Klein and Manning (2004); Petrov et al. (2006), and enabled the development of work on tree- structured models that followed. A recent promi- nent approach is the C-PCFG (Kim al., 2019). Induction Through Representation Learning: DIORA and subsequently S-DIORA (Drozdov et al., 2020, 2019) are models which induce struc- ture using the IORNN. Instead of providing a fixed tree as input, they train by using dynamic program- ming over the set of all possible trees based on how well IORNN is able to reconstruct the input sequence. At test time, they use CKY to extract the highest scoring tree as the parse, which achieved SOTA on unsupervised consituency parsing. While these do learn representations, they solely utilise it to enable unsupervised parsing.
For this experiment, we turned to the WikiText-103 benchmark dataset (Merity al., 2016). WikiText- 103 consists of 103 million tokens, with an average sequence length of 118. Each input sequence corre- sponds to an article rather than a sentence, as in our original dataset. We set the maximum sequence length to 512 and train subword tokeniser on the training set with a maximum vocabulary of 25000, This vocabulary is used for both RoBERTa and Self-StrAE. As shown in Table 6, under this set- ting, RoBERTa improves significantly on the word level. RoBERTa also shows improvement on the sentence level, though these are less pronounced as the model was already performing well on these tasks. Surprisingly, Self-StrAL is able to achieve comparable (and in some cases better) performance than the RoBERTa model, despite having orders of magnitude fewer parameters. We can only at- tribute Self-StrAE's performance to the inductive bias behind it: that the model must perform hierar- chical compositions of its input sequence. Which we believe speaks strongly to its merits.
Learning Task Specific Tree Structures: Inspired by Socher et al. (2011) prior work has sought to learn trees for supervised tasks (Choi al., 2017; Maillard et al., 2017; Yogatama al., 2017), under the assumption that a particular task requires its own form of composition. They achieved success, but all use the Tree-LSTM, which was shown to be largely structure agnostic in the supervised setting (Shi et al., 2018): finding we confirm in this work. The Utility of Structure: Prior work has also sought to augment the Transformer architecture with tree structure. Wang al. (2019b) modify self-attention so that it may only operate within constituents. Zanzotto et al. (2020) provide a mod- ule that represents the parse tree for an input and allows the Transformer to attend to it. Sartran et al. (2022) perform pre-training using linearised trees. In all cases, it was found that structure aided with language modelling perplexity and generalisation. Structure and Representation Learning: This area, the focus of our paper, remains largely un- derexplored. Prior work has solely examined the word level, using dependency parses to define the context neighbourhood for given word. The first work to do this was Levy and Goldberg (2014) and achieved promising results, however, they faced issues with vocabulary size becoming intractable. Vashishth et al. (2019) alleviated this issue through the use of GCNs (Kipf and Welling, 2016).
Summary
Comparison with the unstructured baselines shows that explicitly incorporating hierarchical composi- tions be beneficial for multi-level representation learning. With Self-StrAE we show that these ben- efits do necessitate an external parser for prepro- cessing, and can largely be achieved through an inductive bias for explicit merging.
6 Related Work
Recursive Neural Networks: StrAE belongs to the class of recursive neural networks (RvNNs). First popularised by Socher et al. (2011, 2013), who employed RvNNs to perform fine-grained sen- timent analysis, utilising tree structure to overcome the deficits of bag of words models. These early successes inspired the creation of successor frame- works like the IORNN (Le and Zuidema, 2014) and Tree-LSTM (Tai al., 2015) we used baselines. Learning Tree Structure: The induction of struc- ture has long been a goal of NLP. Early pivotal work on corpus based induction was performed by

---

7 Discussion and Future Work
Ethics Statement
The aim this paper is to examine whether explic- itly incorporating hierarchical compositions can be beneficial. The eventual intended goal is to make machine learning models less resource- and data- intensive. We believe this can provide considerable benefits in making research more accessible.
We establish two findings. Firstly, defining repre- sentation similarity through composition is useful, and secondly, asking model to arrange its own composition is powerful inductive bias. Neither of these findings are limited their application the architecture presented in this paper. The re- quirements are: an explicit merge operation and an objective that optimises representations across all levels. As long as these conditions are met, the findings are in theory applicable to any number of architectures. For future development, the natu- ral next step is to examine what happens when allow for significantly more flexible models to dictate their own compositions. Transformers can naturally adapted to incorporate such bias, and we believe this holds considerable promise for fu- ture work, especially in light recent findings that incorporating compression bottlenecks in Trans- formers is beneficial (Nawrot et al., 2023, 2021). Extensions need not be limited to the Transformer architecture, either. Recurrent and recursive neural networks share many similar features, and given the recent resurgence of RNNs (Orvieto al., 2023; Peng et al., 2023) there may also be promise in extending research in this direction.
Acknowledgements
MO was funded by a PhD studentship through Bluawei-ddibburge Research Lab Project 10410153. VP was funded by research grant from the Edinburgh Lab for Integrated Artificial Intelligence (ELIAI).
We also wish thank Eduardo Ponti, Scraphina Goldfard-Tarrann Ivan Titov, Tom Sherborne, Vivek Iyer, Frank Mollica and Ivan Vegner for their valuable comments and feedback.
Limitations
The particular Self-StrAE presented in this paper is considerably limited model. It is minimally pa- rameterised (see Table 5), locally greedy and makes uncontextualised decisions about which nodes to merge. The mode of composition learns is certain to be suboptimal as a result of this, and it speaks to the strength of the inductive bias that it is able to perform at all. The structure it learns certainly doesn't it resemble syntax
understand it (exam- ple trees can be found in Appendix C), and neither did we expect it to. A significantly more flexible model would likely be required this regard. Even then, possible that the form of composition a model learns with respect to its training objective may deviate substantially from our expectations of how compositional structure should look. Sec- ondly, the aim of this paper is to conduct basic research into the benefits of composition and not outperform the state of the art. We believe there is promise in pursuing further research that may eventually lead to improvements over SOTA, but we leave this to future work.

---

References
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- Pre-training of deep bidirectional transformers for language under- standing. In North American Chapter of the Associa- lion for Computational Linguistics (NAACL), pages 4171-4186.
Eneko Agire, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. study on similarity and relatedness using distri- butional and WordNet-based approaches. In North American Chapter of the Association for Computa- tional Linguisties (NAACL), pages 19-27.
Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. CoRR, abs/1601.J 280.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemBval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop Semantic Evaluation (SemEval-2016), pages 497-511, San Diego, Califor- nia. Association for Computational Linguistics.
Andrew Drozdow, Subendhu Rongali, Yi-Pei Chen, Tim O' O'Gorman, Mohit Tyyer, and Andrew McCallum. 2020. Unsupervised parsing with S-DIORA: Single tree encoding for deep inside-outside recursive au- toencoders. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Process- ing (EMNLP), pages 4832-4845, Online. Association for Computational Linguistics.
Eneko Agirre. Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of the First Joint Conference on Lexical and Compu- tational Semantics Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, SemEval 12, page 385-393, USA. Association for Computational Linguistics.
Andrew Drozdov, Patrick Verga, Mohit Yadav, Mohit lyyer, and Andrew McCallum. 2019. Unsupervised latent tree induction with deep ouside recur- sive autoencoders. CoRR, abss/194.00143
Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Siml e9x999: Evaluating semantic models with (gen- uine) similarity estimation. Computational Linguis- tics, (1(4):665-69)
Steven Bird, Ewan Klein, and Edward Loper. 2009, Nat- ural language processing with Python: analyzing text with the natural language toolkit
Sepp Hochreiter and Jürgen Schmidhuber: 1997. Long short-term memory. Neural computation, 9: 17355880
Philip A. Huebner, Elior Sulem, Fisher Cynthia, and
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomás Mikolov. 2016. Enriching word vectors with subword information. CoRR, 0e//16007446666
Dan Roth. 2021. BabyBERTa: Learning more gram- mar with small-scabe child-directed language. In Pro- ceedings of the 25th Conference on Computational Natural Language Learning, pages 624-646, Online. Association for Computational Linguistics.
Daniel Cer, Mona Diab. Encko Agirre, lñigo Lopez- Gazpio, and Lucia Specia. 2017. cemEval-2011 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Workshop on Semantic Evaluation (SemEval), pages 1-14.
Ganesh Jawahar, Benoft Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Association for Computational Lin- guistics (ACL), pages 3651-3657.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A simple framework for contrastive learning of visual representations. In In- ternational Conference on Learning Representations (ICLR), volume I 119, pages 1597-1607.
Tacuk Kim, Jihun Choi, Daniel Edmiston, and Sang-goo Lee. 2020. Are pre-trained language models aware of phrases? simple but strong baselines for grammar induction. In International Conference on Learning Representations (ICLR).
Jihun Choi, Kang Min Yoo, and Sang-goo Lee. 2017. Learning compose task-specific tree structures.
Yoon Kim, Chris Dyer, and Alexander Rush. 2019. Compound probabilistic context-free grammars for grammar induction. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 2369-23662 2385, Florence, Italy. Asso- ciation for Computational Linguistics.
N. Chomsky, 1956. Three models for the description of language. IRE Transactions on Information Theory, 2(3):113-144
Stephen Crain and Mineharu Nakayama. 1987. Struc- ture dependence grammar formation. Language, 6323):322-543.
Thomas N. Kipf and Max Welling. 2016. Semi- supervised classification with graph convolutional networks. CoRR, abs/1609.0290)
Maric-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06) Genoa, Italy. European Language Resources Associ- ation (ELRA).
Dan Klein and Christopher Manning. 2004. Corpus- based induction of syntactic structure: Models of dependency and constituency. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 485- Barcelona, Spain.

---

Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenen- baum, and Samuel J. Gershman. 2017. Building ma- chines that learn and think like people. Behavioral and Brain Sciences, 40:e253.
Antonio Orvieto, Samuel Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. 2023. Resurrecting recurrent neural net- works for long sequences.
Phong Le and Willem Zuidema. 2014. Inside-outside semantics: A framework for neural models of seman- tic composition. In NIPS 2014 Workshop on Deep Learning and Representation Learning.
Christophe Pallier, aine-Domiiqque Devauchelle, and Stanislas Dehaene. 2011. Cortical representation of the constituent structure of sentences. Proceedings of the National Academy of Sciences, 108(6):2522- 2527.
Omer Levy and Yoav Goldberg. 2014, Dependency- based word embeddings. In Association for Compu- tational Linguistics (ACL)( Volume 2: Short Papers). pages 302-308.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Mattco Grella, Kranthi Kiran GV, Xuzheng He. Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Kr- ishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Xiangru Tang, Bolun Wang. Johan S. Wind, Stansi- law Wozniak, Ruichong Zhang, Zhenyuan Zhang. Qihang Zhao, Peng Zhou, Jian Zhu, and Rui-Jie Zhu. 2023. Rwkv: Reinventing mns for the transformer era.
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Jean Maillard, Stephen Clark, and Dani Yogatama. 2017. Jointly learning sentence embeddings and syntax with unsupervised tree-Istms. CoRR,
Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein, 2006. Learning accurate, compact, and inter- pretable tree annotation. In Proceedings of the 21st International Conference on Computational Linguis- tics and 44th Annual Meeting of the Association for Computational Linguistics, pages 433-440 Sydney, Australia. Association for Computational Linguistics.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language pro- cessing toolkit. In Association for Computational Linguistics (ACL): System Demonstrations, pages 55- 60.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Con- ference on Machine Learning (ICML), pages 8748- 8763. PMLR.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zam- parelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14), pages 216-223, Reykjavik, Iceland. European Lan- guage Resources Association (EIRRA).
Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro. Miloš Stanojević, Phil Blunsom, and Chris Dyer. 2022. Transformer grammars: Augmenting trans- former language models with syntactic inductive bi- ases at seale. arXiv preprint arXiv:2203. 00633.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- els. CoRR, wabs//60900770
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Association for Computational Linguistics (ACL), pages 1715-1725.
Aaron Mueller and Tal Linzen. 2023. How to plant trees in language models: Data and architectural effects on the emergence syntactic inductive biases. ArXiv, abs/2305. 19905.
Haoyue Shi, Hao Zhou, Jiaze Chen, and Lei Li. 2018. On tree-based neural sentence modeling. In Em- pirical Methods in Natural Language Processing (EMNLP), pages 4631-4641.
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher Manning. 2022. Characterizing in- trinsic compositionality in transformers with tree pro- jections.
Yuge Shi, Brooks Paige, Philip Torr, and N. Siddharth. 2020. Relating by contrasting: A data-efficient framework for multimodal generative models. In International Conference on Learning Representa- tions (ICLR).
Piotr Nawrot, Jan Chorowski, Adrian kańcucki, and Edoardo M. Ponti. 2023. Efficiennt transformers with dynamic token pooling.
Piotr Nawrot, Szymon Tworkowski, Michal Tyrolski. Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. 2021. Hierarchical transform- ers are more efficient language models. CoRR, abs/21 10. 13
Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning, 201 I. Supi-super recursive autoencoders for predict- ing sentiment distributions. In Empirical Methods

---

Natural Language Processing (EMNLP), pages 151-161.
Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Wang Ling. 2017. Learning to compose words into sentences with reinforcement learning. In International Conference on Learning Representations (ICLR).
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning. Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Empirical Methods in Natural Language Process- ing (EMNLP), pages 1631-1642.
Fabio Massimo Zanzotto, Andrea Santilli, Leonardo Ranaldi, Dario Onorati, Pierfrancesco Tommasino, and Francesca Fallucchi. 2020. KERMIT: Com- plementing transformer architectures with encoders of explicit syntactic interpretations. In Empirical Methods in Natural Language Processing (EMNLP). pages 256-267.
Kai Sheng Tai, Richard Socher, and Christopher D. Man- ning. 2015. Improved semantic representations from tree-structured long short-term memory networks. Association for Computational Linguistics (ACL),
pages 1556-1566.
Shikhar Vashishth, Manik Bhandari, Prateek Yadav, Piyush Rai, Chiranjib Bhattacharyya, and Partha Talukdar. 2019. Incorporating syntactic and semantic information word embeddings using graph convo- lutional networks. In Association for Computational Linguistics (ACL), pages 3308 -318.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems (NeuRIPS), pages 5998-6008.
Ivan Vulic, Edoardo Maria Ponti, Robert Litschko, Goran Glavas, and Anna Korhonen. 2020. Prob- ing pretrained language models for lexical semantics. CoRR, abs/2010.05731.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Inter- national Conference on Learning Representations (ICLR).
Minjie Wang. Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma. Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. 2020. Deep graph library: A graph-centric, berhy-performant package for graph neural networks.
Tongzhou Wang and Phillip Isola. 2020. Understand- ing contrastive representation learning through align- ment and uniformity on the hypersphere. CoRR, abs/2005.1022. 10242.
Yaushian Wang, Hung- Yi Lee, and Yun-Nung Chen. 2019b. Tree transformer: Integrating tree structures into self-attention. In Empirical Methods in Natural Language Processing and International Joint Con- ference on Natural Language Processing (EMNLP- IJCNLP), pages 1061-1070.
Dongling Xiao, Linzheng Chai, Qian-Wen Zhang, Zhao Yan, Zhoujun Li, and Yunbo Cao. 2022. Cqr- sql: Conversational question reformulation enhanced context-dependent text-to-sql parsers.

---

A Training Data and Hyper-parameeers
length of 23.58. The trees from Self-StrAE have an average depth of 9, compared with 23 (rounding up for simplicity) for right-branching trees, and 5 for balanced binary trees. Self-StrAE trees exhibit a slight preference for right-branching, with each non-leaf node on average having fewer left than right successors 60% of the time.
We trained each StrAE (and the tree baselines) model for 15 epochs (sufficient for convergence) using the Adam optimizer with a learning rate of le-3 for cross entropy and le-4 for contrastive loss, using a batch size of 128. We applied dropout of 0.2 the embeddings and 0.1 on the composi- tion and decomposition functions. The temperature hyper-parameter for the contrastive objective was set to 0.2 and the value 0.1. These settings were obtained by a grid search over the values r= 1.0, 0.1, 0.01, batch size = 128, 256, 512, 768, r = 0.2. 0.4, 0.6, 0.8 and learning rate le-3, 5e-4, and le-4. The Fasttext baseline was trained for 15 epochs with learning rate of Ic-3, and window size of 10. Self-StrAE trained using the same hyperpa- rameters as StrAE, except on Wikitext-109 where we lowered the learning rate to 5e-5 and increased I to 0.6, and decreased r to 0.0001. RoBERTa was trained for 100 epochs, with a 10% of steps used for warmup, a learning rate of 5e-5 and a linear schedule. We used relative key-query posi- tional embeddings. The Bi-LSTM was trained for 15 epochs with learning rate le-3, batch size 128 and dropout of 0.2 applied the embeddings and output layer.
However, the best way to get sense for the kind of structures the model learns is by looking at examples shown on the following pages. Look- ing at the examples Fig. and Fig. 6 the model has learned some sensible pattens. For example, has learned to segment sentences with embed- ded clauses or conjunctions into their constituent parts (e.g. Fig. 5a,e.g and Fig, 6a). However, the trees frequently exhibit attachment errors that we hypothesize are the result of structure being deter- mined by co-occurrence frequency. For example, in Fig. 6a the model merges [will be] as its own constituent rather than the correct parse of [will [be cancelled]]. Which is likely because "will" and "be" co-occur much more frequently than any instance of "be" + passive form of given verb. Similar behaviour can be found all throughout the examples in Fig. 5 and Fig. 6. Given the simplic- ity of the model it is unsurprising that Self-StrAE is unable to learn deeper rules, however, it would be interesting to determine what extent this is also a function of the data it is trained on. Re- cent work has shown that transcribed child directed speech leads transformer models to learn grammar more efficiently (Huebner et al., 2021; Mueller and Linzen, 2023). Transcribed CDS is far less regular than Wikipedia and may cause the model avoid simpler heuristics like bigram frequency.
B Self-StrAE Algorithm
Algorithm 1 Self-StrAE Agglomerative Compose
Input: Node frontier W- composition (o)
pairwise node similarity CSIM(e, e')
I: A + [
b initialise frontier > initialise tree merges
2: T <<< Ø
3: while [A| > 1 do
Finally, there are cases where the model seems be learning totally implausible structures. We are yet to determine the root cause of this, but include examples in Fig. 7 for the sake of transparency.
4: eg e+ t
arg max CSIM(er, 1))) EiRiIEE
choose closest adjacent pair
5:
A A (c7,
& remove from frontier > compute composition
6:
e = efe
D Performance by Structure Type
7:
A - AUr(et)
b insert into frontier at
8:
T T TU((1,++1)
D record merge location
We show in Table 7 comparison of performance for different structure types and objectives used in this work. As discussed earlier, the objectives used here are contrastive (C) and standard cross- entropy (CE), and the structure types explored are the syntactic, purely right branching (RB) and the balanced binary (BB) trees.
9: return Merge order T
C Tree Statistics and Examples
Self-StrAE learns trees which are purely right- branching, balanced binary, or purely random. We parse our development split using Self-StrAR BPE models pre-trained on the 10M corpus. The split itself consists of 40k sentences with an average

---

John
Mary
John
who
Mary
's
cousin
(a) Self-StrAr
(b) CoreNLP
The
book
that
Mary
read
interesting
book
that
Mary
read
interesting
(c) Self-StrAE
(d) CoreNLP
(e) Self-StrAE
(I) CoreNLP
wanted
play
but
it
started
rain
play
(g) Self-StrAE
(h) CoreNLP
Figure 5: Comparison of trees: Self-StrAB vs. CoreNLP

---

the
match will
cancelled If it rains
match
will
cancelled
(a) Self-StrAE
(b) CoreNLP
The sun rises in the
east The
sun
rises
in the
east
(c) Self-StrAE
(d) CoreNLP
Do
you
like
coffee ?
Do
you
like
coffee ?
(e) Self-StrAE
(l CoreNLP
The
book
was
read
by
Mary
The
book
was
read
by
Mary
(g) Self-StrAE
(h) CoreNLP
Figure Comparison of trees: Self-StrAB vs. CoreNLP

---

thc
with
the
telescope
pt
dog
The fat
laz.
cat
sle
pt
The
quick
Irowa
fox
jumped
88
f1
moming
sbot
eleyhant
80y
14
Time
fits
an
armiw
fmitl
flies
like
banana
Figure 7: Incomprehensible Cases

---

Table 7: Comparison Across Structure Type
Model
Simalex
Wordsim
Wordsim
STS
STS
STSB
SICK
MRPC
RTE
SMAL Syntactic
15.54 1 0.25
56.1 0.477
43.77
1.01
34,59 1 2.58
50.4
0.97
41.83 1 2.23
50.3
0.88
67.24
0.39
56.16
1.59
StAA Syntnctic CB
17.57 + 1.08
2.97
36.73 = 6.19
5.02
1.1
25.86 1.3
1,76 1.76+0096 0.95
39.06 1,51
67.48 0.3
53.6 0.46
Syntactic
IORNN Syntactic CT
24.36 0.6
51,53 1 1.59
36.55
1.78
.0.17 1. 0.92
22.68 1 0.98
5.04 1 0.55
38.41 1 0.63
67.24
0.63
54.28 L. 0.55
Syntactic
LSTM Syntactic CE
14.23 2.01
3.22
33.24 3.02
-1.66 1,29
19.38 1.21
6.7 1.04
34.55
68 0.0
52.2 + 1.98
SmAAR
4.6
19.6 9.0
13.4 9.2
3.7
17.9 10.9
3.11 5.36
22.65 2.88
66.32 0.44
50.2 1. 1.77
RB
IORNN RB C
3,66 3.66-20.2 0.21
28.24 0.75
8.74: 3.73
2.92 +: 1.31
18.5 0.95
-2.47 = 1.23
24.6 1.67
67.15 0.26
52.2 :: 0.88
LSTM RB C
9.58 0.76
40.97 0.94
22.87 0.4
6.96 à: 0.77
32 0.43
2,76 0.31
29.65 0.31
68.36 0.39
53.6 4: 0.35
LSTM
8.69 2.69:222221
31.62 3.79
19.43 5.84
12.67 +: 2.85
18:65 3.06
5.87 3.51
0.53 28.67
68.16
0.32
53.12 = 1.31
StAA BB
1.0
43.4 16.9
33.6 12.6
16.6 à:
16:1.466
9 9.1
24.18 0.32
68 1
$3.5 1 0.0
SUAE BB CE
2.15
$2,4 =
35.08 =: 2,42
3:46
19.66 1.34
20.37 = 1.75
24.99 0.43
61.28
0.57
$2.56 0.84
TORNN BB CE
19,02 0:48
$7.14 0.91
44.92 =: 0.98
9.46 0.58
18.11 0.1
8,86 :0.91
24.3 0.93
67.4 0.36
55.36 = 1.06
0.54
LSTM CE
4.67 2.78
21.21 6.27
14,04 4.78
8.13 2.18
12.99 1: 1.77
9,90 9.90
21.15% 1,43
68.0 0,00
53.20 = 0.8


Reviews:
[{"review_id": "PpupBnc7hR", "reviewer": "Reviewer_pRFX", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback and time taken reading the paper. \n\nThe embedding process for Self-StrAE is indeed identical to StrAE. The only difference lies in how the order of compositions (i.e. the tree structure) is defined. Self-StrAE orders compositions according to cosine similarity between adjacent node embeddings in the frontier, merging the argmax at each step. Figure 4 of the paper is intended to show this process with the example sentence ‘Homer ate doughnuts’. Following the example in the figure, the similarity between ‘ate’ and ‘doughnuts’ is greater than that of ‘ate’ to ‘Homer’, so the model first merges ‘ate doughnuts’ into a single embedding using the composition function. At the next step, the model merges ‘Homer’ and ‘ate doughnuts’ into a single embedding and arrives at the root. \n\nWe save the merge history in an adjacency matrix so that by the time the encoder reaches the root node, the adjacency matrix represents the whole tree over the input sequence. We then convert the adjacency matrix into a DGL graph (DGL is a library for graph ML), and pass that as input to the decoder. The decoder operates exactly the same as in vanilla StrAE, only that in this case the input graph is defined by the encoder’s merge history not e.g. a syntactic parser. \n\nWe hope this explanation helped make things clearer, and will update the manuscript with a more detailed description. \n\nThank you also for your pointers to typos and suggestions for improving the text—we will incorporate these in the updated manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work addresses the incorporation of linguistic structural patterns (e.g., constituency) into language representations, by means of an autoencoder model of much smaller scale than current large language models. This is of great interest for the NLP community, not only due to structural information learning being a longstanding goal in NLP and its practical applications, but also due to the lower computational cost of the proposed approach. The text is very well written and straightforward to follow, the experiments have appropriate coverage, and the findings are consistent with the claims and argumentation.", "reasons_to_reject": "Not a reason to reject, but the embedding process for Self-StrAE needs to be better clarified. Is it the same as StrAE?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a method for encoding structural information from sentences using an Autoencoder framework. The objective of the approach is to learn token and sentence representations that reflect the hierarchical composition process of language syntax, thus obtaining more expressive token/sentence embeddings. This is achieved through a combination of a recursive Autoencoder architecture and a structure oriented contrastive loss. Two encoding schemes are proposed: one based on external tree structure supervision (StrAE) and one based on self-clustering (Self-StrAE) of embeddings, which are the main contributions of this work. An additional contribution is the empirical analysis of the contrastive objective over different architectural constraints and different tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "L.554-555: \"explicitly incorporating hierarchical compositions be highly beneficial\" -> \"explicitly incorporating hierarhical compositions *can* be highly beneficial\"\n\nL.631-632: \"when we allow for a significantly more flexible models\" -> \"when we allow for significantly more flexible models\""}}, {"review_id": "guGasHMhfO", "reviewer": "Reviewer_D1Qk", "summary": "", "questions": "1. There is no definition for the notation N when it first appears\n2. In Table 4, could you give the total number of parameters for all the competitors\n3. Sentence embeddings produced by the average output of BERT-like models perform much worse than GolVe (46.35 vs 46.35 on the STS benchmark [1]). Therefore the comparison is not fair enough.\n4. RoBERTa is not designed for context-free word embeddings. Are there any empirical studies to show the way you induce word-level embeddings from language models is feasible?\n5. Compared to a tiny version of BERT (2 layers, 2 heads, 128 dimensions), the performance of RoBERTa in Table 4 is much lower (MRPC: 81.1 vs 69.19; RTE: 57.2: 54.86) [2].\n6. Can you provide another naive baseline in Table 4, i.e., CNN (or) with Fasttext or GloVe? I'd like to see the benefit of introducing explicit compositions\n7. In contrastive loss, the objective is to maximize the similarity between $\\overline{e_i}$ and $\\underline{e_i}$ while minimizing the similarity between other token embeddings (in-batch negatives). Does this way introduce noises? For example, two identical words (or semantically related words) appear in the same sentence. \n8. In Table 3, could you describe the model size of each competitor?\n\n[1] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks\n\n[2] https://github.com/google-research/bert", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback, suggestions, insightful questions, and positive appraisal of our work. We would like to begin our rebuttal by addressing the concerns raised by the reviewer before moving to the questions. \n\nRegarding the use of RoBERTa and unstructured baselines, we note that\n1. **RoBERTa for sentence-level embeddings**:\n    Averaging the final layer (i.e. mean pooling)  is demonstrably the best approach for \n         unsupervised STS with BERT like models, as shown and used by S-BERT [1].\n    At lower dataset sizes (as with our work) averaged GloVe embeddings underperform \n         StrAE and RoBERTa (please see evidence in A3 below). \n2. **Roberta for static embeddings:**\n    We use the same method for extracting static lexical embeddings as established by \n          Vulic et al. 2020 [2].\n     We further provide Fasttext embeddings as a baseline (Table 4 of the paper) which is \n         designed specifically to produce static embeddings. \n\nRegarding the benefit of using StrAE, yes, StrAE does indeed need (much) fewer non-embedding parameters (please see response to A2 below).\n\nAs for the question of scalability to very-large sizes, in principle, especially with Self-StrAE, we do not see any specific reason why our method should _not_ be applicable at scale—the only practical constraints, as with most similar methods, is GPU memory.\nAs for effectiveness at reasoning or general-purpose use, we suspect that being able to do so could be a fruitful avenue for future exploration, and are excited by the potential of developing a more interpretable and low resource method for this purpose.\n\nRegarding the use of StrAE as an auxiliary task for pretrained models, we agree that it is an interesting question. However, our goal in this work is to explore the use and utility of explicit structure, and combining StrAE with pretrained models would make it harder for us to assign credit properly. We do believe it could prove to be an interesting direction for future work, though.\n\nTypos/Grammar/Presentation:\nWe will clarify notation by employing normal-weight lower-case letters as scalars, bolded lower-case letters as vectors and upper-case letters as matrices.\n\nWe now respond to the specific questions raised in the review:\n\n**A1: Notation N**\nApologies for the oversight. The use of N (L147) refers to the square root of the embedding dimension. Each embedding is NxN as we treat them as matrices during composition and decomposition. In our experiments, each StrAE embedding is a 10x10 matrix.  \n\n**A2: Parameter Counts**\nThank you for the suggestion; we agree that these should be included in the paper and will amend the manuscript accordingly. \nWe also need to correct the number of parameters for the RoBERTa baseline, which was reported as 1.5 million (Line 485-487) —the true number is closer to 4 million, and reported in the table below. We will fix the error in the manuscript. The list of parameters for Q2 and Q8 is as follows:\n\n| Model Name      | Parameters excluding embedding matrix (and LM head where applicable) |\n|-----------------|-----------------------------------------------------------------------|\n| StrAE/Self-StrAE| 400                                                                   |\n| IORNN           | 40,400                                                                |\n| Tree-LSTM       | 260,600                                                               |\n| RoBERTa         | 3,950,232                                                             |\n\nWe omit Fasttext from the table, as the number of parameters effectively just corresponds to the size of the embedding matrix. We do not include the embedding matrix in counting parameters in the table, as is standard practice, as it only reflects the type of tokenisation used and not the model complexity. \n\n**A3: RoBERTa vs GLoVe Sentence Embeddings** \nYou are indeed correct that the S-BERT paper notes that mean-pooled embeddings of BERT-like models underperforms GloVe. However, this does not appear to be the case with smaller-scale pre-training data. To demonstrate this, we evaluate Self-Strae, RoBERTa, and GloVe on the STS eval tasks with our 10M and Wiki103 datasets.\n\n| Model                  | STS-12       | STS-16       | STS-B        | SICK-R       |\n|------------------------|--------------|--------------|--------------|--------------|\n| Self-StrAE BPE 10M     | 34.42 +- 0.73| 49.93 +- 0.38| 36.68 +- 0.64| 51 +- 0.3    |\n| RoBERTa 10M            | 29.48 +- 3.28| 50.88 +- 1.11| 38.36 +- 1.9 | 49.58 +- 0.91|\n| GloVe 10M              | 25.08 +- 0.95| 27.11 +- 0.4 | 21.84 +- 0.3 | 46.09 +- 1.22|\n| Self-StrAE BPE Wiki103 | 46.64 +- 0.23| 52.08 +- 0.37| 40.59 +- 0.83| 51.94 +- 0.4 |\n| RoBERTa Wiki103        | 35.38 +- 1.47| 52.64 +- 1.11| 39.74 +- 39.74| 50.68 +- 0.43|\n| GloVe Wiki103          | 36.8 +- 1.28 | 31.11 +- 0.49| 29.83 +- 0.71| 51.35 +- 0.87|\n\nIn both these settings, averaged GloVe vectors underperform those from Self-StrAE and RoBERTa, although the gap with RoBERTa diminishes on Wiki103. As a further sanity check, we have also compared the full 840B GloVe embeddings with RoBERTa base, and can replicate the findings of the S-BERT paper.\n\nIn the context of the data scales investigated in our paper, RoBERTa proves a stronger baseline. We are however happy to include GloVe as a baseline in the updated manuscript, and thank the reviewer for raising the issue, as it is possible other readers might share similar concerns.\n\n\n**A4: RoBERTa and static word embeddings**\nYes indeed; we employ the technique derived in [2] to construct static embeddings from RoBERTa’s contextual embeddings, noting that the chosen method is extensively empirically validated therein.\n\n**A5: Comparison with Tiny BERT**\nThere are two notable differences between our RoBERTa and Tiny-BERT. \n1. Tiny-BERT is fine-tuned on the downstream task while all our models are evaluated frozen.\n2. Tiny BERT is pre-trained on significantly more data: Wikipedia + Books contains 3.3 billion tokens, whereas our two pre-training sets consist of 10 and 103 million tokens respectively. \n\nRegarding MRPC, Table 5 in S-BERT [1] gives the MRPC score for averaged BERT embeddings as 69.45 which corresponds to our results. \n\n**A6: Additional Naive Baseline**\nNote that most of our evaluation is zero-shot—simply computing measures with learned embeddings. To introduce a CNN baseline would require pre-training from scratch (unless we have misunderstood what you were requesting?).\nTo our knowledge, there isn’t an established model/architecture that employs CNNs directly for pre-training on natural language, and the plethora of hyperparameter choices involved in designing something reasonable here would not necessarily tell us if performance was because of/despite the choices made. If there is a specific model you had in mind, please do let us know, and we’d be happy to add it as a baseline.\n\nHowever, to address your more general point about a baseline that does not _explicitly_ model hierarchical composition, we can attempt to answer that by employing a standard Bi-LSTM framework on our data (10M tokens). We provide evaluation results for this below, and will be happy to include this as an additional baseline to Table 4.\n\n|  Model     | Bi-LSTM BPE   | Bi-LSTM WORD  |\n|---------------|---------------|---------------|\n| Simlex        | 12.88 +- 0.77 | 23.74 +- 1.17 |\n| Wordsim S     | 39.46 +- 2.17 | 61.94 +- 2.44 |\n| Wordsim R     | 32.22 +- 3.07 | 41.46 +- 1.73 |\n| STS-12        | 9.22 +- 0.87  | 11.18 +- 1.1  |\n| STS-16        | 33.78 +- 1.66 | 34.86 +- 0.86 |\n| STS-B         | 14.06 +- 2.01 | 13.46 +- 1.11 |\n| SICK-R        | 40.36 +- 0.27 | 42.36 +- 0.19 |\n| MRPC          | 68.21 +- 0.41 | 68.95 +- 0.38 |\n| RTE           | 53.90 +- 0.96 | 54.09 +- 0.95 |\n| Score         | 33.79 +- 0.64 | 39.12 +- 0.34 |\n\n**A7: Noise in Contrastive Loss**\nYes, you are indeed correct to note that the contrastive loss can include some measure of noise through false negatives (FN, i.e. same embedding being counted as negative by appearing elsewhere in the batch).\n\nThe noise, in practice, appears to have minimal effect, and performance is largely unaffected without taking mitigation measures. This is primarily due to the effect of many more _true_ negatives (TN) than false negatives (FN). However, without taking any measures, learning plateaus quicker and results are marginally lower. The issue can be mitigated by masking false negatives, so they don’t affect the softmax and don’t count toward loss. We will add an appendix to the updated manuscript with ablations showing the effect of masking to mitigate this issue. \n\n[1] [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://aclanthology.org/D19-1410) (Reimers & Gurevych, EMNLP-IJCNLP 2019)\n\n[2] [Probing Pretrained Language Models for Lexical Semantics](https://aclanthology.org/2020.emnlp-main.586) (Vulić et al., EMNLP 2020)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation is clearly stated and the paper is easy to follow\n2. Using auto-encoder and contrastive loss to inject structural information is interesting\n3. StrAE can outperform other tree models (IORNN and Tree-LSTM), which shows the effectiveness of the proposed method", "reasons_to_reject": "My main concern is the experiment design of section 5 (Comparison to Unstructured Baselines):\n1. This work averages the RoBERTa final output layer to derive sentence-level embeddings. However, this way yields very bad embeddings that are even worse than averaging GloVe embeddings, as discussed in the paper of Sentence-BERT.\n2. The power of RoBERTa is to produce contextual embeddings, and the work uses static embeddings induced from RoBERTa for comparison, which might result in misleading conclusions\n\nMy another concern is the main benefit of using StrAE but I did not see it in the current version \n1. Does the framework consume fewer parameters or have a faster inference speed\n2. Can we scale up the model to a very large size for having powerful reasoning capability and general-purpose usage\n3. Can we integrate StrAE into pre-trained language models as an auxiliary task\n", "questions_to_authors": "1. There is no definition for the notation N when it first appears\n2. In Table 4, could you give the total number of parameters for all the competitors\n3. Sentence embeddings produced by the average output of BERT-like models perform much worse than GolVe (46.35 vs 46.35 on the STS benchmark [1]). Therefore the comparison is not fair enough.\n4. RoBERTa is not designed for context-free word embeddings. Are there any empirical studies to show the way you induce word-level embeddings from language models is feasible?\n5. Compared to a tiny version of BERT (2 layers, 2 heads, 128 dimensions), the performance of RoBERTa in Table 4 is much lower (MRPC: 81.1 vs 69.19; RTE: 57.2: 54.86) [2].\n6. Can you provide another naive baseline in Table 4, i.e., CNN (or) with Fasttext or GloVe? I'd like to see the benefit of introducing explicit compositions\n7. In contrastive loss, the objective is to maximize the similarity between $\\overline{e_i}$ and $\\underline{e_i}$ while minimizing the similarity between other token embeddings (in-batch negatives). Does this way introduce noises? For example, two identical words (or semantically related words) appear in the same sentence. \n8. In Table 3, could you describe the model size of each competitor?\n\n[1] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks\n\n[2] https://github.com/google-research/bert", "paper_topic_and_main_contributions": "This paper proposes a Structured AutoEncoder framework (StrAE) that uses hierarchical\ncompositions explicitly to produce multi-level representations, and it adopts an encoder-decoder architecture with contrastive loss. Moreover, to avoid using structure information as input, they proposed a variant termed Self-StrAE that is able to learn how to compose each token.\nExperimental results show that the proposed framework outperforms baselines that don’t involve explicit hierarchical compositions, and is comparable to models given informative structure.  ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I feel a bit confused by the notations in the paper and cannot distinguish a scalar, vector, and matrix easily."}, "extra_text": {}}, {"review_id": "gIbcS4ugAt", "reviewer": "Reviewer_xyAG", "summary": "", "questions": "- Q1 training and decoding times are  not given, nor compared with other methods. Can you discuss them? I think it would be a nice addition to the paper.\n- Q2 in Section 4 have you tried  experiments regarding the different tree shapes with really inconsistent trees ? (ie. at the beginning of the training or at each epoch, draw a random tree structure for each sentence) to see how the models behave when the merge are really inconsistent. I disagree over the fact that right-branching or balanced structures are inconsistent, merely linguistically inconsistent.\n  Moreover, the tree shapes produced by Self-StrAE are not discussed enough: how do they look? linguistically plausible, right-branching, balanced... or random?\n- Q3 using the NSP objective during transformer pretraining could enable the use of the CLS token as sentence representative instead of the simplistic average, and could lead to better performance. Have you tried? In other words, maybe the transformer baseline for sentence-level tasks is artificially low.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback, insightful suggestions and comments.  \n\n**A1 Time**:\nThe training and inference speeds are not prohibitively slow. We report below iterations per second on a single A40 GPU over a batch size of 128.\nTraining an epoch takes ~10m for the StrAE variants and ~4m for RoBERTa, although RoBERTA needs to train for significantly longer to maximise performance—needing 100 epochs vs 15 for the StrAE variants. Total training time on 10M tokens data amounted to ~2.5hrs for the StrAE variants and ~7hrs for RoBERTa.\n\nNote that the speed for StrAE variants is dependent on the structure, with the tree depth dictating the number of sequential steps. The results below for StrAE are for the constituency parses, with balanced binary and right-branching being faster and slower respectively.\nSelf-StrAE during encoding has a disadvantage in this regard as it only merges once per step.\nIn practice, this could also be improved by merging the top-k most similar nodes, rather than the argmax at each step.\n\n\n| Model     | Train Speed GPU | Eval Speed GPU | Train Speed CPU | Eval Speed CPU |\n|-----------|-----------------|----------------|-----------------|----------------|\n| StrAE     | 8.55 it/s       | 9.09 it/s      | 2.20 s/it       | **1.39 it/s**      |\n| IORNN     | 7.67 it/s       | 8.69 it/s      | 2.26 s/it       | 1.24 it/s      |\n| Tree-LSTM | 6.38 it/s       | 7.43 it/s      | 2.44 s/it       | 1.23 it/s      |\n| Self-StrAE| 7.59 it/s       | 9.46 it/s      | **1.85 s/it**       | 1.37 it/s      |\n| RoBERTa   | **18.2 it/s**       | **43.54 it/s**     | 11.12 s/it      | 3.37 s/it      |\n\n\n**A2a Truly random trees**: \nYour comment about the kind of inconsistency right-branching and balanced trees provide vs truly random trees is indeed correct.\nWe are currently running experiments to test the performance of random trees, which are slightly complicated by two factors: (i) ensuring that the trees are indeed random, i.e., not inadvertently picking up on low-level patterns, and (ii) time and compute to run the full model as was done with the other tree types. We hope to be able to update this response with results as soon as we have them, and of course will include them in the updated manuscript.\n\n\n**A2b Self-StrAE Trees**: We also wholeheartedly agree these should be discussed in the paper. We will update the manuscript with some qualitative examples and descriptive statistics. It doesn’t seem possible for us to share the qualitative examples in the rebuttal, but we can share the statistics.\n\nThese statistics come from the average across the development split of our 10M token pre-training set, which consists of 40k sequences with an average length of 23.58. Self-StrAE trees have an average depth of 9 levels, compared with 23 (if we round up for simplicity) for right-branching trees, and 5 for balanced binary trees. Self-StrAE trees exhibit a slight preference towards right-branching factor, with each non-leaf node on average having fewer left than right successors 60% of the time. We calculate branching preference by looking at the depth of the subtree for the left and right child of each node. \n\nQualitatively, the trees are certainly not random, nor purely right-branching, and also, perhaps unsurprisingly, not aligned with known types of linguistic trees.\nA qualitative analysis on a small set of example sentences reveals the emergence of some sensible patterns. For example, the model learns to merge determiner-noun, or determine-adjective-noun tuples into a single constituent. It also appears to have learned to segment sentences that contain conjunctions (e.g. The fat cat slept _and_ the angry dog barked), into the two separate sub phrases.\n\nHowever, there are definite limitations to the linguistic plausibility of the trees. We noticed that when the number of intervening adjectives between a determiner and a noun increases, the model struggles to resolve the determiner. For example, in the phrase ‘The fat lazy tired old cat’, the model seeks to merge 'cat' and all the corresponding adjectives, but then fails to merge the determiner and complete the constituent.\n\nGiven the simplicity of using the greedy agglomerative (easy-first) clustering based on cosine similarity to determine merge order, this behaviour is not entirely unexpected. The structure-induction process would need to be made more flexible to better handle long-distance dependencies. As stated in the limitations section, we didn’t expect or really intend for Self-StrAE to learn something resembling tree bank syntax, just the best compression order w.r.t its data and training objective and subject to its limitations. It would be interesting for future work to compare how these patterns change with a more sophisticated technique for determining merge order, or modifications to the training objective. \n\nWe thank the reviewer for the suggestion, and would welcome any feedback on particular behaviours we should probe for that they think might be of interest.\n\n\n**A3 NSP**: The reason we use mean pooling and MLM is due to precedents and findings established in prior work. S-BERT [1] demonstrates that mean pooling is significantly more effective than using CLS for unsupervised STS, and these findings are easy to replicate using pretrained models from Huggingface. It is also the strategy the authors used for their model, which achieved SOTA sentence embeddings. In the RoBERTa paper [2] the authors demonstrate that completely removing NSP as a pre-training objective ‘matches or slightly improves downstream task performance’. Secondly, MLM only means that the RoBERTa baseline has greater parity to the objectives that Self-StrAE and Fasttext are trained with. As neither of these models use additional labels during pretraining. Given these findings, we believe our decisions for the RoBERTa baselines are fair and justified. \n\nThank you for pointing out the missing reference; we agree our method is similar, and will add a discussion to it in the related work section of our updated manuscript. The two papers we believe are most appropriate are [3,4] and we will include them in the updated manuscript.\n\n\n[1] [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://aclanthology.org/D19-1410) (Reimers & Gurevych, EMNLP-IJCNLP 2019)\n\n[2] [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) (Liu et al. 2019)\n\n[3] [An Efficient Algorithm for Easy-First Non-Directional Dependency Parsing](https://aclanthology.org/N10-1115) (Goldberg & Elhadad, NAACL 2010)\n\n[4] [Partial parsing via finite-state cascades](https://www.cambridge.org/core/journals/natural-language-engineering/article/abs/partial-parsing-via-finitestate-cascades/431C1780A3D3BED2E42D79B1D14C6861) (Abney S. 1996)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Again, the paper is really well written: the model is well presented and advocated for, the experimental results look solid.\n- Moreover, the limited number of parameters used in this model, could be of interest outside the \"interpretability\" community, as an example of a resource-friendly architecture (but see question Q1 below)\n", "reasons_to_reject": "I can't see one after my reading (unless training and decoding times are prohibitively slow, see Q1)\n", "questions_to_authors": "- Q1 training and decoding times are  not given, nor compared with other methods. Can you discuss them? I think it would be a nice addition to the paper.\n- Q2 in Section 4 have you tried  experiments regarding the different tree shapes with really inconsistent trees ? (ie. at the beginning of the training or at each epoch, draw a random tree structure for each sentence) to see how the models behave when the merge are really inconsistent. I disagree over the fact that right-branching or balanced structures are inconsistent, merely linguistically inconsistent.\n  Moreover, the tree shapes produced by Self-StrAE are not discussed enough: how do they look? linguistically plausible, right-branching, balanced... or random?\n- Q3 using the NSP objective during transformer pretraining could enable the use of the CLS token as sentence representative instead of the simplistic average, and could lead to better performance. Have you tried? In other words, maybe the transformer baseline for sentence-level tasks is artificially low.\n", "paper_topic_and_main_contributions": "This paper presents an auto-encoder architecture for sentences and words where the hierarchical structure is considered an explicit bias.\nThe model computes two representations for every node of the binary parse tree corresponding to the input sentence: one when traversing the tree bottom-up from leaves to root which is basically a compression of the information contained in the word embeddings following preselected paths, and the other one is computed when traversing back the tree in a top-down fashion where the information compressed at the root level is expanded recursively in the children nodes down to the leaves where tokens are predicted back.\n\nThe model is based on the notions of faithfulness, ie. the representation of a node has only access to its children nodes (resp. parent node) during the bottom-up (resp. top-down) phase. This is in contrast with previous propositions such as IORNNs or Tree-LSTMs.\nThis seems to be a crucial point as is demonstrated by the new training method based on constrastive-learning, where each bottom-up node representation\nmust be more similar to its top-down counterpart than any other node representation.\nStrAE can take really advantage of this form of learning while IORNNs and Tree-LSTMs have more mitigated results.\n\nThe comparisons with IORNNs and Tree-LSTMs are performed on various tasks and show that StrAE outperforms the baselines, and moreover that the advantage is stronger when the parse trees are linguistically motivated foir sentence-level semantics tasks.\nThen the comparisons with word-only encoders, such as FastText and RoBERTa, show that the hierarchical information gives a strong inductive bias: with very few parameters the system is able to outperform the RoBERTa baseline, both in small and large data regimes.\n\n\nThe paper is really well written: the model is well presented and advocated for, the experimental results look solid.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The way Self-StrAE build the tree bottom-up following the similarity is reminiscent of the Easy-First strategy, which can be added to a section for related work.\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an encoder-decoder network to compositionally encode by explicitly factoring in constituency parses for sentences. Results are positive on a number of textual similarity and inference tasks. The reviews converge in their soundness and excitement scores.
Decision: Accept-Main