SlowGuess/NeurIPS_2023_OCR huggingface dataset:
No OneSoft: Ray Chowdhury and Caragea [66] also introduced a form of soft top- $k$  function (OneSoft) for better gradient propagation in BT-RvNN. While we still use that as a baseline model, we do not include OneSoft in EBT-RvNN. This is because OneSoft generally doubles the memory usage and EBT-RvNN already runs well without it. The combination of EBT-RvNN with OneSoft can be studied more in the future, but it is a variable that we do not focus on in this study.
None of the fixes here makes any strict asymptotic difference in terms of sequence length but it does lift a large overhead that can be empirically demonstrated (see Table 1).
4 Beyond Sentence Encoding
As discussed before many of the previous models [9, 10, 31, 66] in this sphere that focus on competency on structure-sensitive tasks have been framed to work only as a sentence encoder of the form  $f: \mathbb{R}^{n \times d} \to \mathbb{R}^d$ . Taking a step further, we also explore a way to use bottom-up Tree-RvNNs for token-level contextualization, i.e., to make it serve as a function of the form  $f: \mathbb{R}^{n \times d} \to \mathbb{R}^{n \times d}$ . This allows Tree-RvNNs to be stackable with other deep learning modules like Transformers.
Here, we consider whether we can re-formulate EBT-RvNNs for token contextualization. In EBT-RvNN<sup>5</sup>, strictly speaking, the output is not just the final sentence encoding (the root encoding), but also the intermediate non-terminal tree nodes. Previously, we ignored them after we got the root encoding. However, using them can be the key to creating a token contextualization out of EBT-RvNNs. Essentially, what EBT-RvNN will build is a tree structure with node representations - the terminal nodes being the initial token vectors, the root node being the overall sentence encoding vector, and the non-terminal nodes representing different scales of hierarchies as previously discussed.
Under this light, one way to create a token contextualization is to contextualize the terminal representations based on higher-level composite representations at different scales or hierarchies of which the terminal representation is a part of. In other words, while we use a bottom-up process of building wholes from parts during sentence encoding, for token contextualization, we can implement a top-down process of contextualizing parts from the wholes that it compose.
A similar idea is used by Teng et al. [82] to recursively contextualize child node representations based on their immediate parent node using another recursive cell starting from the root and ending up at the terminal node representations. The contextualized terminal node representations can then become the contextualized token representations. But this idea requires costly sequential operations.
An alternative - that we propose - is to allow the terminal nodes to attend [1] to the non-terminal nodes to retrieve relevant information to different scales of hierarchies. More precisely, if we want the terminal nodes to be contextualized by the wholes that they compose then we want to restrict the attention to only the parents (direct or indirect). This can be done by creating an attention mask based
5The principles discussed here also apply to them Gumbel Tree RvNNs, BT-RvNNs, and the like.
on the induced tree structures. In practice, we allow every terminal node as queries to attend to every node as keys and values but use a mask to allow attention only if the key represents the same node as that represented by the query or the key represents some parent (direct or indirect) of the node represented by the query. We also implement a relative positional encoding to bias attention [70] - using the difference in heights of the nodes as the relative distance. In essence, we are proposing the use of a form of graph attention network [86]. This attention mechanism can be repeated for iterative refinement of the token representations through multiple layers of message-passing.
In the case of EBT-RvNNs, we can create separate token representations for each beam and then marginalize them based on beam scores. We describe our concrete setup briefly below but more details are presented in Appendix F.
Step 1: First, we begin with beams of representations before they were marginalized. This allows us to access discrete edges connecting parents for every beam. As a setup, we have some  $b$  beams of tree node representations and their structural information (edge connections).
Step 2: We use Gated Attention Unit (GAU) [36], a modern Transformer variant, as the attention mechanism block. We use the terminal node representations as queries  $(Q)$  and all the non-terminal nodes as keys  $(K)$  and values  $(V)$ . We use GAU, like a graph attention network [86], by using an adjacency matrix  $A$  as an attention mask.  $A_{ij}$  is set as 1 if and only if  $Q_{i}$  is a child of  $K_{j}$  based on our tree extraction. Otherwise  $A_{ij}$  is set as 0. Thus attention is only allowed from parents to their terminal children (direct or indirect).
Step 3: We implement a basic relative positional encoding - similar to that of Raffel et al. [64]. The only difference is that for us, the relative distances are the relative height distances.
Step 4: The GAU layer is repeated for iterative refinement of terminal node representations. We repeat for two iterations since this is an expensive step.
Step 5: As before, we marginalize the beams based on the accumulated log-softmax scores after the terminal node representations are contextualized.
Use Case: In theory, the contextualized terminal node representations that are built up in Step 5 can be used for any task like sequence labelling or masked language modeling. At this point, we explore one specific use case - sentence-pair matching tasks (natural language inference and paraphrase detection). For these tasks we have two input sequences that we need to compare. Previously we only created sentence encoding for each sequences and made the vectors interact, but now we can make the whole of two sequences of contextualized terminal-node embeddings interact with each other through a stack of GAU-based self-attention. This is an approach that we use for some of the sentence-matching tasks in Natural Language Processing (Table 3). The models are trained end to end. We discuss the technical details about these architectural setups more explicitly in Appendix F.
5.1 Model Nomenclature
Sentence Encoder models: Transformer refers to Transformers [85]; UT refers to Universal Transformers [13]; CRvNN refers to Continuous Recursive Neural Network [10]; OM refers to Ordered Memory [71]; BT-GRC refers to BT-RvNN implemented with GRC [66]; BT-GRC OS refers to BT-GRC combined with OneSoft (OS) Top-  $K$  function [66]; EBT-GRC refers to our proposed EBT-RvNN model with GRC; GT-GRC refers to Gumbel-Tree RvNN [9] but with GRC as the recursive cell; EGT-GRC refers to GT-GRC plus the fixes that we propose.
Sentence Interaction Models: Sequence interaction models refer to the models in the style described in Section 4. These models use some deeper interaction between contextualized token representations from both sequences without bottlenecking the interactions through a pair of vectors. We use EBT-GAU to refer to the approach described in Section 4. EGT-GAU refers to a new baseline which uses the same framework as EBT-GAU except it replaces the Beam-Tree-Search with greedy STE gumbel-softmax based selection as in [9]. GAU refers to a plain stack of Gated Attention Units [36] (made to approximate the parameters of EBT-GAU) that do not use any Tree-RvNNs and is trained directly on  $<\mathrm{SEP}>$  concatenated sequence pairs.
Table 1: Empirical time and (peak) memory consumption for various models on an RTX A6000. Ran on 100 ListOps data with batch size 1 and the same hyperparameters as used on ListOps on various sequence lengths. (-slice) indicates EBT-GRC without slicing from Fix 2, (512) indicates EBT-GRC with the hidden state dimension  $(d)$  set as 512 (instead of 128).(512,-slice) represents EBT-GRC with 512 dimensional hidden state size and without slicing.
Sequence Lengths
Model	200 – 250	500 – 600	900 – 1000	1500 – 2000
Time (min)	Memory (GB)	Time (min)	Memory (GB)	Time (min)	Memory (GB)	Time (min)	Memory (GB)
OM	8.0	0.09	20.6	0.21	38.2	0.35	76.6	0.68
CRvNN	1.5	1.57	4.3	12.2	8.0	42.79	OOM	OOM
GT-GRC	0.5	0.35	2.1	1.95	3.5	5.45	7.1	21.76
EGT-GRC	1	0.07	2.5	0.3	4.3	0.81	8.5	3.15
BT-GRC	1.1	1.71	2.6	9.82	5.1	27.27	OOM	OOM
BT-GRC OS	1.4	2.74	4.0	15.5	7.1	42.95	OOM	OOM
EBT-GRC	1.2	0.19	3.2	1.01	5.5	2.78	10.5	10.97
– slice	1.2	0.35	3.2	1.95	5.4	5.4	10.3	21.12
(512)	1.2	0.41	3.3	1.29	5.6	3.13	12.1	11.41
(512,– slice)	1.2	1.55	3.3	7.77	5.5	21.02	OOM	OOM
5.2 Efficiency Analysis
In Table 1, we compare the empirical time-memory trade offs of the most relevant Tree-RvNN models (particularly those that are competent in ListOps and logical inference). We use CRvNN in the no halting mode as [66] because otherwise it can start to halt trivially because of limited training data. For the splits of lengths  $200 - 1000$  we use the data shared by Havrylov et al. [31]; for the  $1500 - 2000$  split we sample from the training set of LRA listops [81].
We can observe from the table that EBT-GRC achieves better memory efficiency among all the strong RvNN contenders (GT-GRC and EGT-GRC fail on ListOps/Logical inference) except for OM. However, OM's memory efficiency comes with a massive cost in time, being nearly 8 times slower than EBT-GRC. Compared to BT-GRC OS's 43GB peak memory consumption in 900-1000 sequence length from [66], the memory consumption of EBT-GRC is reduced to only 2.8GB. Even compared to BT-GRC, the reduction is near ten times. EBT-GRC even outperforms the original greedy GT-GRC used in Choi et al. [9]. Removing the slicing from the full model EBT-GRC (i.e., -slice) can substantially increase the memory cost. This becomes most apparent when training with higher hidden state size (compare (512) vs. (512,-slice)). This shows the effectiveness of slicing.
Hyperparameters are presented in Appendix G, architecture details are presented in Appendix F, task details are provided in Appendix B and additional results (besides what is presented below) in logical inference and text classification are provided in Appendix C.
List Operations (ListOps): The task of ListOps consists of hierarchical nested operations that neural networks generally struggle to solve particularly in length-generalizable settings. There are only a few known contenders that achieve decent performance in the task [31, 10, 71, 66]. For this task we use the original training set [57] with the length generalization splits from Havrylov et al. [31], the argument generalization splits from Ray Chowdhury and Caragea [66], and the LRA test set from Tay et al. [81]. The different splits test the model in different out-of-distribution settings (one in unseen lengths, another in an unseen number of arguments, and another in both unseen lengths and arguments). Remarkably, as can be seen from Table 2, EBT-GRC outperforms most of the previous models in accuracy - only being slightly behind OM for some argument generalization splits. EBT-GRC-Slice represents the performance of EBT-GRC without slicing. It shows that slicing in fact improves the accuracy as well in this context but even without slicing the model is better than BT-GRC or BT-GRC OS.
Table 2: Accuracy on ListOps. For our models, we report the median of 3 runs. Our models were trained on lengths  $\leq 100$ , depth  $\leq 20$ , and arguments  $\leq 5$ . * represents results copied from [71]. We bold the best results that do not use gold trees. Subscript represents standard deviation. As an example,  $90_{1} = 90 \pm 0.1$
Model (Lengths) (Arguments)	near-IID ≤ 1000 ≤ 5	Length Gen.	Argument Gen.	LRA 2000 10
200-300 ≤ 5	500-600 ≤ 5	900-1000 ≤ 5	100-1000 10	100-1000 15
With gold trees
GoldTreeGRC	99.9.2	99.9.9	99.81	100.5	81.228	79.514	78.529
Baselines without gold trees
Transformer *	57.44	—	—	—	—	—	—
UT *	71.578	—	—	—	—	—	—
GT-GRC	75.4.6	47.78.4	42.72.8	37.5337	50.915	51.416	45.312
EGT-GRC	84.219	51.337	42.935	34.435	44.717	40.816	34.414
OM	99.9.3	99.6.7	92.413	76.313	83.224	76.338	79.318
CRvNN	99.72.8	98.811	97.223	94.949	66.640	43.738	55.3844
BT-GRC	99.42.7	96.810	93.622	88.427	75.228	59.179	63.457
BT-GRC OS	99.65.4	97.235	94.865	92.286	73.364	63.192	66.1101
EBT-GRC	99.90.3	99.72.4	99.51	99.35	82.513	79.68.7	79.36.5
EBT-GRC - Slice	99.73	98.612	98.417	98.614	79.320	74.437	75.525
Logical Inference and Sentiment Classification: We show the results of our models in formal logical inference (another dataset where only RvNN-like models have shown some success) and sentiment classification in Appendix C. There we show that our EBT-GRC can easily keep up on these tasks with BT-GRC despite being much more efficient.
NLI and Paraphrase Detection: As we can see from Table 3, although EBT-GRC does not strictly outperform BT-GRC or BT-GRC OS, it remains in the same ballpark performance. The sentence interaction models, unsurprisingly, tend to have higher scores that sentence encoder modes because of their more parameters and more interaction space. We do not treat them commensurate here. Among the sequence interaction models, our EBT-GAU generally outperforms both baseline models in its vicinity - GAU and EGT-GAU. Even when used in conjunction with a Transformer, beam search still maintains some usefulness over simpler STE-based greedy search (EGT-GAU) and it shows some potential against pure Transformer stacks as well (GAU).
We identify a memory bottleneck in a popular RvNN framework [9] which has caused BT-RvNN [66] to require more memory than it needs to. Mitigating this bottleneck allows us to reduce the memory consumption of EBT-RvNN (an efficient variant of BT-RvNN) by 10-16 times without much other cost and while preserving similar task performances (and sometimes even beating the original BT-RvNN). The fixes also equally apply to any model using the framework including the original Gumbel Tree model [9]. We believe our work can serve as a basic baseline and a bridge for the development of more scalable models in the RvNN family and beyond.
Although our proposal improves upon the computational trade-offs over some of the prior works [10, 9, 66, 71], it can be still more expensive than standard RNNs although we address this limitation, to an extent, in our concurrent work [65]. Moreover, our investigation of utilizing bottom-up Tree-RvNNs for top-down signal (without using expensive CYK models [15, 16]) is rather preliminary (efficiency being our main focus). This area of investigation needs to be focused more in the future. Moreover, although our proposal reduces memory usage it does not help much on accuracy scores compared with other competitive RvNNs.
Table 3: Mean accuracy and standard deviation on SNLI [6], QQP, and MNLI [89]. Hard represents the SNLI test set from Gururangan et al. [28], Break represents the SNLI test set from Glockner et al. [22]. Count. represents the counterfactual test set from Kaushik et al. [38].  $\mathrm{PAWS}_{QQP}$  and  $\mathrm{PAWS}_{WIKI}$  are adversarial test sets from Zhang et al. [97], ConjNLI is the dev set from Saha et al. [67], NegM,NegMM,LenM,LenMM are Negation Match, Negation Mismatch, Length Match, Length Mismatch stress test sets from Naik et al. [56] respectively. Our models were run 3 times on different seeds. Subscript represents standard deviation. As an example,  $90_{1} = 90\pm 0.1$
Models	SNLI Training	QQP Training
IID	Hard	Break	Count.	IID	\( PAWS_{QQP} \)	\( PAWS_{Wiki} \)
(Sequence Encoder Models)
CRvNN	85.32	70.64	55.317	59.86	84.83	34.87	46.66
OM	85.52	70.63	67.49	59.92	84.60	38.17	45.68
BT-GRC	84.91	705	5114	594	84.75	36.917	46.412
BT-GRC OS	84.91	70.36	53.2910	58.63	84.22	37.18	46.36
EBT-GRC	84.74	69.98	55.620	58.11	84.32	36.95	47.55
(Sequence Interaction Models)
GAU	872	74.24	69.4034	66.42.5	83.61	38.614	47.31
EGT-GAU	87.10	74.84	66.112	66.22	83.54	39.431	49.313
EBT-GAU	87.52	75.73	7026	67.64	83.92	42.335	47.28
MNLI Training
Models	Match	MM	ConjNLI	NegM	NegMM	LenM	LenMM
(Sequence Encoder Models)
CRvNN	72.24	72.65	41.710	52.86	53.84.2	6244	63.347
OM	72.53	732	41.74	50.97	51.713	56.533	57.0631
BT-GRC	71.62	72.31	40.76	53.737	54.843	64.76	66.45
BT-GRC OS	71.71	71.92	41.29	53.22	54.25	65.613	66.79
EBT-GRC	72.12	721	40.930	52.3323	53.2822	64.9210	66.410
(Sequence Interaction Models)
GAU	76.43	76.52	53.512	48.211	48.2411	69.620	70.622
EGT-GAU	75.12	75.53	53.14	48.714	48.614	69.813	70.611
EBT-GAU	76.51	76.92	53.318	49.24	493	71.618	72.519
This research is supported in part by NSF CAREER award #1802358, NSF IIS award #2107518, and UIC Discovery Partners Institute (DPI) award. Any opinions, findings, and conclusions expressed here are those of the authors and do not necessarily reflect the views of NSF or DPI. We thank our anonymous reviewers for their constructive feedback.
[1] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1409.0473.
[2] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/01386bd6d8e091c2ab4c7c7de644d37b-Paper.pdf.
A Appendix Organization
In Section B, we describe the settings of all the tasks and datasets that we have tested our models on. In Section C, we provide additional results on logical inference and sentiment classification. Then, in Section D, we present an extended survey of related works. In Section E, we present the pseudocode for EBT-RvNN. In Section F, we detail our architecture setup including the sequence-interaction models. In Section G, we provide our hyperparameters.
B Task Details
ListOps: ListOps was introduced by Nangia and Bowman [57] and is a task for solving nested lists of mathematical operations. It is a 10-way classification task. Similar to Chowdhury and Caragea [10], we train our models on the original training set with all samples  $\geq 100$  sequence lengths filtered out. We use the original development set for validation. We test on the following sets: the original test set (near-IID split); the length generalization splits from Havrylov et al. [31] that include samples of much higher lengths; the argument generalization splits from Ray Chowdhury and Caragea [66] that involve an unseen number of maximum arguments for each operator; and the LRA split (which has both higher sequence length and higher argument number) from Tay et al. [81].
Logical Inference: Logical Inference was introduced by Bowman et al. [7] and is a task that involves classifying fine-grained inferential relations between two given sequences in a form similar to that of formal sentences of propositional logic. Similar to Tran et al. [84], our models were trained on splits with logical connectives  $\leq 6$ . We show the results in OOD test sets with logical connections 10-12. We use the same splits as Shen et al. [71], Tran et al. [84], Chowdhury and Caragea [10].
SST5: SST5 is a fine-grained 5-way sentiment classification task introduced by Socher et al. [79]. We use the original splits.
IMDB: IMDB is a binary sentiment classification task from Maas et al. [49]. We use the same train, validation, and IID test sets as created in Ray Chowdhury and Caragea [66]. We also use the contrast set Gardner et al. [21] and counterfactual set Kaushik et al. [38] as additional test splits.
QQP:  $\mathrm{QQP^6}$  [37] is a task of classifying whether two given sequences in a pair are paraphrases of each other or not. Following prior works Wang et al. [88], we randomly sample 10,000 samples for validation and IID test set such that for each split 5,000 samples are maintained to be paraphrases and the other 5,000 are maintained to be not paraphrases. We also use the adversarial test sets  $\mathrm{PAWS}_{QQP}$  and  $\mathrm{PAWS}_{WIKI}$  form Zhang et al. [97].
SNLI: SNLI [6] is a natural language inference (NLI) task. It is a 3-way classification task to classify the inferential relation between two given sequences. We use the same train, development, and IID test set splits as in Chowdhury and Caragea [10]. Any data with a sequence of length  $\geq 150$  is filtered out from the training set for efficiency. We use also additional test set splits for stress tests. We use the hard test set split from Gururangan et al. [28], the break test set from Glockner et al. [22], and the counterfactual test set from Kaushik et al. [38].
MNLI: MNLI [89] is another NLI dataset, which is similar to SNLI in format. We use the original development sets (match and mismatch) as test sets. We filter out all data with any sequence length  $\geq 150$  from the training set. Our actual development set is a random sample of 10,000 data-points from the filtered training set. As additional testing sets, we use the development set of Conjunctive NLI (ConjNLI) [67] and a few of the stress sets from Naik et al. [56]. These stress test sets include - Negation Match (NegM), Negation Mismatch (NegMM), Length Match (LenM), and Length Mismatch (LenMM). NegM and NegMM add tautologies containing "not" terms - this can bias the models to classify contradiction as the inferential relation because the training set contains spurious correlations between existence of "not" related terms and the class of contradiction. LenM and LenMM add tautologies to artificially increase the lengths of the samples without changing the inferential relation class.
$^{6}$ https://data.quora.com/First-Quora-Dataset-Release-QuestionPairs
Table 4: Mean accuracy and standard deviation on the Logical Inference [7] for  $\geq 10$  number of operations after training on samples with  $\leq 6$  operations, and on SST5 [79] and IMDB [49]. Count. represents counterfactual test split from Kaushik et al. [38] and Cont. represents contrast test split from Gardner et al. [21] The best results are shown in bold. Our models were run 3 times on different seeds. Subscript represents standard deviation. As an example,  $90_{1} = 90\pm 0.1$
Model	Logical Inference Number of Operations	SST5	IMDB
10	11	12	IID	IID	Cont.	Count.
GT-GRC	90.3322	88.4318	85.7024	51.678.8	85.1110	70.6321	81.975
EGT-GRC	75.7961	73.3868	69.6878	51.6314	86.582.7	72.92	81.7614
CRvNN	94.512.9	94.485.6	92.7315	51.7511	91.471.2	77.8015	85.383.5
OM	94.952	93.92.2	93.366.2	52.302.7	91.690.5	76.985.8	83.687.8
BT-GRC	95.042.3	94.293.8	93.362.4	52.324.7	91.291.2	75.0729	82.8623
BT-GRC OS	95.434.5	94.216.6	93.391.5	51.927.2	90.869.3	75.6821	84.7711
EBT-GRC	94.951.5	93.877.4	93.046.7	52.221	91.471.2	76.1617	84.2912
In Table 4, we show that our EBT-GRC model can keep up fairly well with BT-GRC and BT-GRC OS on logical inference [7] and sentiment classification tasks like SST5 [79], and IMDB [21] while being much more computationally efficient as demonstrated in the main paper. Additional comparisons with other models like Transformers and Universal Transformer in logical inference can be found in prior works Shen et al. [71], Tran et al. [84]. They underperform RNNs and RvNNs in logical inference.
D Extended Related Works
RvNN History: Recursive Neural Networks (RvNNs) in the more specified sense of building representations through trees and directed acyclic graphs were proposed in [63, 24]. Socher et al. [77, 78, 79] extended the use of RvNNs in Natural Language Processing (NLP) by considering constituency trees and dependency trees. A few works [98, 80, 43, 99] started adapting Long Shot-term Memory Networks [33] as a cell function for recursive processing. Le and Zuidema [44], Maillard et al. [51] proposed a chart-based method for simulating bottom-up Recursive Neural Networks through dynamic programming. Shi et al. [75], Munkhdalai and Yu [55] explored heuristics-based tree-structured RvNNs.