SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Problem with the String-it Solution: The choice above is arbitrary. While concatenating the top scoring beams (the first beam of every  $BR$ ) can be motivated, there is no reason to disallow beams from different ranks (say  $BR_{1}[i]$  and  $BR_{2}[j]$  where  $i \neq j$ ) from being concatenated together especially if they form a unique beam with higher combined scores than others.
Beam Alignment Solution: Given the above problem, we introduce our solution - the Beam Alignment technique. In this technique, we apply a simple stochastic approach following the idea that "higher scoring beams should have higher chance to exist in the combined beams". Based on this idea, we simply transform each  $(BS_{l}, BR_{l})$  into some  $(BS_{l}', BR_{l}')$  (we explain this below) and then apply the string-it solution on the transformed outputs  $(BS_{l}', BR_{l}')$  to get the input for the next outer recursion. The transformation works as follows:
$$
i d x _ {l} = \text {s a m p l e} (\text {p o p u l a t i o n} = [ 0, 1, \dots , b - 1 ],
$$
$$
\text {d i s t r i b u t i o n} = \text {n o r m a l i z e} \left(B S _ {l}\right), \tag {8}
$$
$$
\mathrm {t i m e s} = b)
$$
That is, first, we sample the indices  $(idx_{l})$  of beams based on their corresponding scores  $(BS_{l})$ . The scores  $BS_{l}$  can be normalized with softmax. We sample  $b$  times with repetition. The next steps are:
$$
B S _ {l} ^ {\prime} = \left[ B S _ {l} [ j ] \text {f o r} j \text {i n} i d x _ {l} \right]; \quad B R _ {l} ^ {\prime} = \left[ B R _ {l} [ j ] \text {f o r} j \text {i n} i d x _ {l} \right] \tag {9}
$$
Thus, beams with higher scores will be sampled more times and in turn will have a higher chance to be present in  $BS_{l}^{\prime}$  and  $BR_{l}^{\prime}$ , and in turn, more likely to be present in the stringed-together beams  $BS_{new}$  and  $BR_{new}$  after applying the string-it solution. As such, the high-level idea of increasing
2That is the beams of sentence encoding for the chunk input  $\in \mathbb{R}^{B\times k\times d_h}$  as created by the inner EBT-RvNN processor.
3 Alternatively one could also try to search for an optimal combination of beams but that can increase compute and may end up losing diversity in the stringed beams.
the chance of higher scoring beams is implemented<sup>4</sup>. One additional detail: in practice, we enforce  $BS_{new}[0]$  and  $BR_{new}[0]$  to be the concatenation of the top beams from the pre-transformed materials  $(BS_{l}, BR_{l})$  so that the best possible stringed beam is guaranteed to be present. We provide additional intuition on beam alignment with added visualization in Appendix I.
3.2 Pre-Chunk Processing
The chunking used in BKT-RvNN does not take into account the input content. This can create bad chunks where the information necessary to process one chunk is in another chunk. For example, imagine what happens if we split MAX(1,2,3,4) into three chunks [MAX(1), [2,3,4], and ()) (where the square brackets indicate chunk boundaries). There would be no information initially for the later chunks to work on. To mitigate that we use an initial (single) linear RNN layer bidirectionally to propagate information outside chunks before starting the RIR loop. Specifically, we use an S4D with Gated Linear Units [30] (described more in Appendix J). Moreover, earlier literature has often used a recurrent layer to initialize the non-terminals before running their Tree-RvNN [12, 84] for more contextualized parsing decisions. So the S4D layer can be seen as serving that purpose too.
3.3 Practical Tips and Points
The chunk size  $(k)$  for BKT-RvNN can effectively work as a trade-off parameter in the RIR framework. Lower chunk size can make the whole pipeline faster by having less work to do for the heavy inner-RvNN, but for the same reason the point of using EBT-RvNN (instead of a simpler RNN model) would start to get lost as we lower the chunk size. In Appendix C, we show that lowering chunk size drastically deteriorates the performance of our model in ListOps. A practical option is to set the chunk size as high as possible while meeting one's practical computational need. We generally set  $k = 30$  for our purposes. We also discuss theoretical ways to extend the framework for language modeling in Appendix H.
RIR Inference: As mentioned before, EBT-RvNN runs within the RIR framework and can be interpreted as just EBT-RvNN with some additional constraints in its tree-search space. Thus, during inference, we can just run EBT-RvNN as it is without RIR given that the core model remains the same - now just being allowed to cover more space and gain enhanced performance. By default, we generally indeed do that during inference even if the EBT-RvNN model is trained under RIR however using RIR inference generally works just as well for less structure-sensitive tasks. We discuss more about this decision in Appendix D.
4.1 Model Nomenclature
RNN-GRC represents an RNN implemented with Gated Recursive Cell (GRC) as the recursive cell  $R$  (GRC was introduced in [82] and discussed in Appendix J); CRvNN refers to Continuous Recursive Neural Network [13]; OM refers to Ordered Memory [82]; BT-GRC refers to BT-RvNN implemented with GRC [77]; BT-GRC OS refers to BT-GRC combined with OneSoft (OS) Top-  $K$  function [77]; EBT-GRC refers to EBT-RvNN model with GRC; S4D refers to stacked bidirectional S4D model implemented similar to S4D-inv in [30]; MEGA refers to a hybrid model combining Transformers and an SSM based on exponential moving average as introduced in [60]; BBT-GRC refers to BBT RvNN [84] but with GRC (it is intended to be a more basic baseline and does not have S4D as an initial layer), RIR-GRC refers to an RIR-based model using a recurrent GRC cell for the inner RvNN (it can be thought to be similar to sliced RNN [101]), RIR-EBT-GRC is our main proposed model. Both of the latter models use an S4D layer before starting the main RIR loop.
4The overall strategy can create some repetitions of beams. However, we do not think this should happen too often because the score distributions tend to have high entropy in practice. Moreover, the original BT-RvNN framework already allows repetitions and still works well. Trying to completely remove repetitions can require some tricky operation that takes up resources without much return. We leave it for future research for further analysis on potential ways to avoid repetitions.
Table 1: Empirical time and (peak) memory consumption for various models on an RTX A6000. Ran on 100 ListOps data with batch size 1 and the same hyperparameters as used on ListOps on various sequence lengths. For the splits of lengths  $200 - 1000$  we use the data shared by Havrylov et al. [38]; for the  $1500 - 2000$  split we sample from the training set of LRA listops [92]
Sequence Lengths
Model	200 – 250	500 – 600	900 – 1000	1500 – 2000
Time (min)	Memory (GB)	Time (min)	Memory (GB)	Time (min)	Memory (GB)	Time (min)	Memory (GB)
RNN-GRC	0.22	0.02	0.54	0.02	1.3	0.03	2.5	0.04
MEGA	0.06	0.07	0.05	0.16	0.05	0.29	0.05	0.72
S4D	0.04	0.03	0.05	0.05	0.04	0.07	0.05	0.14
BBT-GRC	0.02	0.02	0.02	0.02	0.03	0.02	0.03	0.04
RIR-GRC	0.06	0.02	0.08	0.03	0.09	0.04	0.1	0.07
ListOps Competitive
OM	8.0	0.09	20.6	0.21	38.2	0.35	76.6	0.68
CRvNN	1.5	1.57	4.3	12.2	8.0	42.79	OOM	OOM
BT-GRC	1.1	1.71	2.6	9.82	5.1	27.27	OOM	OOM
BT-GRC OS	1.4	2.74	4.0	15.5	7.1	42.95	OOM	OOM
EBT-GRC	1.2	0.19	3.2	1.01	5.5	2.78	10.5	10.97
RIR-EBT-GRC	0.1	0.07	0.2	0.14	0.3	0.23	0.3	0.43
Table 2: Accuracy on ListOps-O. For our models we report the median of 3 runs. Our models were trained on lengths  $\leq 100$ , depth  $\leq 20$ , and arguments  $\leq 5$ . We bold the best results that do not use gold trees. Subscript represents standard deviation. We use the original training set [70] with the length generalization splits from Havrylov et al. [38], the argument generalization splits from Ray Chowdhury and Caragea [77], and the LRA test set from Tay et al. [92]. As an example,  $90_{1} = 90 \pm 0.1$
Model (Lengths) (Arguments)	near-IID ≤ 1000 ≤ 5	Length Gen.	Argument Gen.	LRA 2000 10
200-300 ≤ 5	500-600 ≤ 5	900-1000 ≤ 5	100-1000 10	100-1000 15
MEGA	75.452.6	45.21.5	31.714	24.736	39.533	33.859	25.849
S4D	72.33	31	20.85	14.7	27.3	22.76	17.4
BB-Tree GRC	59.18	43.6	40.4	31.5	45.35	44.5	38.25
RIR-GRC	78.33	41.75	35.55	32.3	43.8	42.75	40.05
OM	99.9	99.6	92.7	76.9	84.15	75.05	80.1
CRvNN	99.82	99.5	98.5	98	65.45	45.1	55.38
BT-GRC OS	99.9	99.5	99	97.2	76.05	67.9	71.8
EBT-GRC	99.9	99.9	99.4	99.5	82.95	79	79.5
RIR-EBT-GRC	99.72	99.15	98.25	97.1	74.9	49.55	61.65
-S4D	99.78	99.15	98.87	98.6	63.95	47.73	48.25
-Beam Align	98.83	91.75	79.05	68.8	59.55	43.65	40.75
For more details on the architectures and experiments, see Appendix A. There are also more ablations in Appendix C.
Efficiency Analysis: In Table 1, we compare the empirical time-memory trade-offs of the most relevant Tree-RvNN models. We use CRvNN in the no halting mode as [77] $^5$ . As can be seen, our RIR-EBT-GRC is multiple times faster than the original BT-GRC models and even EBT-GRC
5Otherwise it fails to learn to halt in the small training set and starts to halt too quickly.
Table 3: Mean accuracy and standard deviation on the Logical Inference for  $\geq 8$  number of operations after training on samples with  $\leq 6$  operations. Our models were run 3 times on different seeds. Subscript represents standard deviation. As an example,  $90_{1} = 90\pm 0.1$
Model	Number of Operations
8	9	10	11	12
MEGA	89.716	83.628.8	76.918	70.4535	63.8536
S4D	87.620	79.436	72.3232	65.0153	59.4838
BBT-GRC	77.715	72.531	67.411	64.014	57.512
RIR-GRC	865.6	64.898.9	47.3527	46.4526	45.9225
OM	97.51.6	96.741.4	94.952	93.922	93.366.2
BT-GRC OS	97.031.4	96.491.9	95.434.5	94.216.6	93.391.5
EBT-GRC	97.123	96.53.1	94.951.5	93.877.4	93.046.7
RIR-EBT-GRC	96.841.4	96.051.3	95.453.4	94.213.7	93.3611
Table 4: Accuracy on LRA [92]. * represents that the results are copied from [30] and † represents that the results are copied from [60]. For our models we report the mean of 2 runs. ListOpsMix combines the training set of ListOps-O and ListOps LRA; it keeps the dev set and test set of LRA. Subscript represents standard deviation. As an example,  $90_{1} = 90 \pm 0.1$
Model	LRA	ListOpsMix
ListOps	Text	Retrieval
\( MEGA✝ \)	63.14	90.43	91.25	—
S4D*	60.183.5	87.342	91.09.1	—
S4D	59.30	88.180.4	91.17.6	59.052.5
BBT-GRC	55.151	85.690.7	90.781.6	55.715
RIR-GRC	20.185.7	86.8941.3	88.2410	58.956.5
RIR-EBT-GRC	59.04	87.971.6	88.995.2	64.13.5
-S4D	59.531	88.131.3	89.743.9	70.4390
while using much less memory. As expected, because of still using a heavy inner loop model, it is not as competitive resource-wise against S4D, MEGA, BBT-GRC, or RIR-GRC. However, in our experiments, we find these competitors to fail on ListOps and logical inference whereas RIR-EBT-GRC still works. We present a pareto frontier analysis in the Appendix F.
List Operations (ListOps): Here we discuss the results of Table 2. This table is based on the original ListOps dataset [70]. To disambiguate it from the ListOps used in LRA [92], we call this one ListOps-O. Similar to [13], we filtered all sequences of length  $>100$  from ListOps-O training set. ListOps-O involves training on much shorter samples than those in LRA ListOps but ListOps-O can be challenging in its own right because this allows investigating length-generalization prowess of neural networks. ListOps requires solving hierarchically nested lists of mathematical operations that neural networks, barring a few exceptions [38, 13, 82, 77], generally struggle to solve. The different splits test the models in different out-of-distribution settings, e.g., unseen lengths, unseen number of arguments, or both. As can be seen from Table 2, RIR-EBT-GRC does fairly well in length-generalization (still getting  $\geq 90\%$ ) despite being perturbed within the RIR framework for efficiency. Its argument generalization performance is, however, hampered. Removing the S4D layer (used for pre-chunk preprocessing) from RIR-EBT-GRC (the -S4D row) does not actually harm length generalization but seems to harm argument generalization. Removing beam alignment from RIR-EBT-GRC (and just keeping the string-it solution) substantially deteriorates the performance showing the effectiveness of beam alignment (see -Beam Align row). MEGA, S4D and RIR-GRC perform reasonably in the near- $\mathrm{IID}^6$  setting compared to LSTMs and Universal Transformers [82] but does not come close to more powerful RvNNs. BBT-GRC also does not work well here.
It is "near" IID because it has some longer sequence data than what is present in the training set. But generally, most data are on the shorter end.
Logical Inference: In Table 3, we show the results of our models in a formal logical inference task [6]. This is another task where only stronger Tree-RvNN based models have been able to perform well. In this task the models are trained on samples with  $\leq 6$  operators and we test their generalization performance on higher operators (similar to [93]). We find that RIR-EBT-GRC can keep up well with the state-of-the-art whereas others like MEGA, S4D, BBT-GRC, or RIR-GRC cannot.
LRA and ListOpsMix: In Table 4 we explore the NLP subset of LRA tasks. We find that BBT-GRC can already perform quite well in LRA without any special initialization (note that BBT-GRC does not use S4D), outperforming all non-hybrid Transformer models [18, 106]. RIR-EBT-GRC also shows high competence in LRA. It starts to get a bit slow on the Retrieval task so we reduced the chunk size to 5 which might reduce its performance there. While RIR-EBT-GRC uses S4D for pre-chunk processing, we also run a variant without any S4D (see row -S4D) and it performs just as well or better. This shows that the S4D layer is not playing a big role here. Curiously enough, RIR-EBT-GRC does worse on ListOps LRA test in the in-distribution setting (in Table 4), than it does in the OOD setting trying to generalize from shorter sequences with lower arguments in Table 2. We hypothesize that this might be because RIR-EBT-GRC learns more easily lower-length samples. To investigate whether RIR-EBT-GRC can gain more on LRA when samples of shorter length are present, we created another ListOps dataset - ListOpsMix which combines the ListOps-O training set and the ListOps LRA training set (ListOps LRA dev and test sets are used for validation and testing, respectively). As we can see RIR-EBT-GRC (with or without S4D) is particularly better at utilizing extra lower-length data than other models; however, it still underperforms compared with what EBT-GRC/OM can get with training purely on ListOps-O.
Other Tasks: We also ran our models in a number of other natural language tasks, e.g., natural language inference and sentiment classification. The results are in Appendix E. Generally RIR-EBT-GRC performs similarly to other RvNN-based models.
In this paper, we introduce the Recursion In Recursion (RIR) framework and beam alignment to make Efficient Beam-Tree RvNNs (EBT-RvNN) more scalable. We also show the power of more basic models like BBT-RvNNs on LRA. Even though they do not outperform S4D or some of the newer state-of-the-art [60, 86, 85, 24], BBT-RvNNs are coming from a completely orthogonal family of models which, in recent times, have received limited attention. Moreover, RIR-EBT-RvNNs provide a better trade-off in maintaining competitive performance in LRA text-related tasks and robust length generalization in structure-sensitive tasks as shown in the Pareto Frontier plots in Appendix F. We intend our results to inspire further exploration of these models. We provide an extended discussion with related works in Appendix G.
Our main method, RIR-EBT-GRC, although much more computationally efficient than other ListOps-competent RvNNs [82, 13, 77], is still quite a bit slower compared to BBT-GRC, RIR-GRC, MEGA, and S4D. Moreover, although it serves as a non-linear recursive model that can both scale to training on  $\geq 1000$  sequence lengths and also length-generalize on ListOps and Logical Inference (from shorter samples), it has still difficulties learning from longer sequences.
[1] Forough Arabshahi, Zhichu Lu, Pranay Mundra, Sameer Singh, and Animashree Anandkumar. Compositional generalization with tree stack memory units. arXiv preprint arXiv:1911.01545, 2019.
A Organization
In Section B, we describe the settings of all the tasks and datasets that we used for model evaluation. In Section C, we provide an extended ablation of RIR-EBT-GRC. In Section D, we discuss in more detail the RIR inference strategy. In Section E, we provide additional results on LRA, sentiment classification, natural language inference, and paraphrase detection. In Section F, we provide a Pareto Frontier analysis of different trade-offs for different models. In Section G, we present an extended survey of related work. In Section H, we discuss ways to use RIR-based models for language modeling. In Section I, we provide more intuition and visualization for beam alignment. In Section J, we detail our architecture setup including the S4D. In Section K, we describe our hyperparameters.
B Details on Evaluation Tasks
ListOps: ListOps was originally introduced by Nangia and Bowman [70]. It is a task for solving nested lists of mathematical operations. It is a 10-way classification task. Like Chowdhury and Caragea [13], we train our models on the original training set with all samples  $\geq 100$  sequence lengths filtered away. We use the original development set for validation. We test on the original test set (near-IID split); the length generalization splits from Havrylov et al. [38] that include samples of much higher lengths; the argument generalization splits from Ray Chowdhury and Caragea [77] that involve an unseen number of maximum arguments for each operator; and the LRA split (which has both higher sequence length and higher argument number) from Tay et al. [92].
Logical Inference: Logical Inference was introduced by Bowman et al. [6]. This task involves classifying fine-grained inferential relations between two given sequences in a form similar to that of formal sentences of propositional logic. Similar to Tran et al. [93], our models were trained on splits with logical connectives  $\leq 6$ . We show the results in OOD test sets with logical connections 8-12. We use the same splits as Shen et al. [82], Tran et al. [93], Chowdhury and Caragea [13].
Long Range Arena (LRA): LRA is a set of tasks designed to evaluate the capacities of neural models for modeling long-range dependencies [92]. We evaluate on the language-related subset of tasks - ListOps LRA, Text LRA (character-level IMDB [61]), and Retrieval LRA (character-level document pair matching task).
SST5: SST5 is a fine-grained 5-way sentiment classification task introduced in Socher et al. [89]. We use the original splits.
IMDB: IMDB is a binary sentiment classification task from Maas et al. [61]. We use the same train, validation, and IID test sets as created in Ray Chowdhury and Caragea [77]. We also use the contrast set Gardner et al. [25] and counterfactual set Kaushik et al. [49] as additional test splits.
QQP:  $\mathrm{QQP^7}$  is a task of classifying whether two given sequences in a pair are paraphrases or not. As standard Wang et al. [95], we randomly sample 10,000 samples for validation and IID test set such that for each split 5000 samples are maintained to be paraphrases and the other 5000 are maintained to be not paraphrases. We also use the adversarial test sets  $\mathrm{PAWS}_{QQP}$  and  $\mathrm{PAWS}_{WIKI}$  form Zhang et al. [102].
SNLI: SNLI [5] is a natural language inference (NLI) task. It is a 3-way classification task to classify the inferential relation between two given sequences. We use the same train, development, and IID test set splits as in Chowdhury and Caragea [13]. Any data with a sequence of length  $\geq 150$  is filtered from the training set for efficiency. We also use additional test set splits for stress tests. We use the hard test set split from Gururangan et al. [34], the break test set from Glockner et al. [27], and the counterfactual test set from Kaushik et al. [49].
MNLI: MNLI [96] is another NLI dataset which is similar to SNLI in format. We use the original development sets (match and mismatch) as test sets. We filter away all data with any sequence length  $\geq 150$  from the training set. Our actual development set is a random sample of 10,000 data points from the filtered training set. As additional testing sets, we use the development set of Conjunctive NLI (ConjNLI) [80] and a few of the stress sets from Naik et al. [69]. These stress test sets include - Negation Match (NegM), Negation Mismatch (NegMM), Length Match (LenM), and Length Mismatch (LenMM). NegM and NegMM add tautologies containing "not" terms - this
<sup>7</sup>https://data.quora.com/First-Quora-Dataset-Release-QuestionPairs
Table 5: Extended Ablation on ListOps-O. For our models, we report the median of 3 runs. Our models are trained on lengths  $\leq 100$ , depth  $\leq 20$ , and arguments  $\leq 5$ . We bold the best results that do not use gold trees. We use the original training set [70] with the length generalization splits from Havrylov et al. [38], the argument generalization splits from Ray Chowdhury and Caragea [77], and the LRA test set from Tay et al. [92]. Subscript represents standard deviation. As an example,  $90_{1} = 90 \pm 0.1$
Model	near-IID	Length Gen.	Argument Gen.	LRA
(Lengths)	≤ 1000	200-300	500-600	900-1000	100-1000	100-1000	2000
(Arguments)	≤ 5	≤ 5	≤ 5	≤ 5	10	15	10
RIR-GRC	78.33	41.75	35.55	32.3	43.8	42.75	40.05
RIR-CRvNN	89.15	40.45	35.2	28.7	45.55	43.4	36.65
RIR-OM	97.62	75.7	42.20	29.7	62.55	47.45	36.20
RIR-EBT-GRC	99.72	99.15	98.25	97.1	74.9	49.55	61.65
-S4D	99.78	99.15	98.87	98.6	63.95	47.73	48.25
-Beam Align	98.83	91.75	79.05	68.8	59.55	43.65	40.75
+Random Align	99.14	96.89	93.3	86.4	67.2	53.2	55.05
+RIR inference	87.53	46.5	42.2	36.9	47.4	44.1	39.75
(beam 5)	99.4	97.8	94.3	89.0	65.6	53.15	48.25
(chunk 20)	98.9	91.75	77.85	63.7	66.8	55.7	50.5
(chunk 10)	89.64	47.5	47.35	39.9	45.9	37.75	33.25
can bias the models to classify contradiction as the inferential relation because the training set has spurious correlations between the existence of "not" related terms and the class of contradiction. LenM and LenMM add tautologies to artificially increase the lengths of the samples without changing the inferential relation class.
C Extended Ablation
In Table 5, we show an extended ablation of RIR-EBT-GRC. As we can see, alternatives to EBT-GRC like CRvNN or Ordered Memory (OM) do not work as well within the RIR framework (see RIR-CRvNN or RIR-OM row) although they can be still better than RIR-GRC. We also ran another ablation for Beam Align where we replaced it with another method - Random Align. In Random Align, instead of sampling beams based on beam scores, we sample them based on a uniform distribution. See the “-Random align” row for the result of this method. As can be seen from the table, it does not perform as well. This shows the effectiveness of the high-level idea behind Beam Alignment over another control test. As discussed in the main paper, we generally switch out of the RIR framework during inference. “+RIR inference” shows the result of using the RIR framework during inference too. The results for “+RIR inference” worsen significantly. Given the efficiency of RIR-EBT-GRC, we also increase its default beam size to 7 from the original beam size of 5 as used in EBT-GRC and BT-GRC models. The row “(beam 5)” shows the result of using beam size 5 with RIR-EBT-GRC. The results worsen a bit. RIR-EBT-GRC uses a chunk size of 30. We also show the performance degradation with lower chunk sizes - see “(chunk 20)” and “(chunk 10)”. These results worsen as is consistent with our discussion in the main paper.
D RIR Inference