Real-World Compositional Generalization with Disentangled Sequence-to-Sequence Learning
Year: 2023

Abstract:
Compositional generalization is a basic mechanism in human language learning, which current neural networks struggle with. A recently proposed Disentangled sequence-to-sequence model (Dangle) shows promising generalization capability by learning specialized encodings for each decoding step. We introduce two key modifications to this model which encourage more disentangled representations and improve its compute and memory efficiency, allowing us to tackle compositional generalization in a more realistic setting. Specifically, instead of adaptively re-encoding source keys and values at each time step, we disentangle their representations and only re-encode keys periodically, at some interval. Our new architecture leads to better generalization performance across existing tasks and datasets, and a new machine translation benchmark which we create by detecting naturally occurring compositional patterns in relation to a training set. We show this methodology better emulates realworld requirements than artificial challenges.1

heading: 1 Introduction
text: The Transformer architecture (Vaswani et al., 2017) and variants thereof have become ubiquitous in natural language processing. Despite widespread adoption, there is mounting evidence that Transformers as sequence transduction models struggle with compositional generalization (Kim and Linzen, 2020; Keysers et al., 2020; Li et al., 2021). It is basically the ability to produce and understand a potentially infinite number of novel linguistic expressions by systematically combining known atomic components (Chomsky, 2014; Montague, 1970). Attempts to overcome this limitation have explored various ways to explicitly inject compositional bias through data augmentation (Jia and Liang, 2016; Akyürek et al., 2021; Andreas, 2020;
1Our code and dataset will be available at https:// github.com/mswellhao/Dangle.
Wang et al., 2021) or new training objectives (Conklin et al., 2021; Oren et al., 2020; Yin et al., 2021). The majority of existing approaches have been designed with semantic parsing in mind, and as a result adopt domain- and task-specific grammars or rules which do not extend to other tasks (e.g., machine translation).
In this work we aim to improve generalization via general architectural modifications which are applicable to a wide range of tasks. Our starting point are Zheng and Lapata (2022) who unveil that one of the reasons hindering compositional generalization in Transformers relates to their representations being entangled. They introduce Dangle, a sequence-to-sequence model, which learns more Disentangled representations by adaptively re-encoding (at each time step) the source input. For each decoding step, Dangle learns specialized source encodings by conditioning on the newly decoded target which leads to better compositional generalization compared to vanilla Transformers where source encodings are shared throughout decoding. Although promising, their results are based on synthetic datasets, leaving open the question of whether Dangle is effective in real-world settings involving both complex natural language and compositional generalization.
We present two key modifications to Dangle which encourage learning more disentangled representations more efficiently. The need to perform re-encoding at each time step substantially affects Dangle’s training time and memory footprint. It becomes prohibitively expensive on datasets with long target sequences, e.g., programs with 400+ tokens in datasets like SMCalFlow (Andreas et al., 2020). To alleviate this problem, instead of adaptively re-encoding at each time step, we only reencode periodically, at some interval. Our decoder is no different from a vanilla Transformer decoder except that it just re-encodes once in a while in order to update its history information. Our second
1711
modification concerns disentangling the representations of source keys and values, based on which the encoder in Dangle (and also in Transformers) passes source information to the decoder. Instead of computing keys and values using shared source encodings, we disassociate their representations: we encode source values once and re-encode keys periodically.
We evaluate the proposed model on existing benchmarks (Andreas et al., 2020; Li et al., 2021) and a new dataset which we create to better emulate a real-world setting. We develop a new methodology for detecting examples representative of compositional generalization in naturally occurring text. Given a training and test set: (a) we discard examples from the test set that contain out-of-vocabulary (OOV) or rare words (in relation to training) to exclude novel atoms which are out of scope for compositional generalization; (b) we then measure how compositional a certain test example is with respect to the training corpus; we introduce a metric which allows us to identify a candidate pool of highly compositional examples; (c) using uncertainty estimation, we further select examples from the pool that are both compositional in terms of surface form and challenging in terms of generalization difficulty. Following these three steps, we create a machine translation benchmark using the IWSLT 2014 German-English dataset as our training corpus and the WMT 2014 German-English shared task as our test corpus.
Experimental results demonstrate that our new architecture achieves better generalization performance across tasks and datasets and is adept at handling real-world challenges. Machine translation experiments on a diverse corpus of 1.3M WMT examples show it is particularly effective for long-tail compositional patterns.

heading: 2 Background: The Dangle Model
text: We first describe Dangle, the Disentangled Transformer model introduced in Zheng and Lapata (2022) focusing on their encoder-decoder architecture which they show delivers better performance on complex tasks like machine translation.
Let X = [x1, x2, ..., xn] denote a source sequence; let fEncoder and fDecoder denote a Transformer encoder and decoder, respectively. X is first encoded into a sequence of N contextualized representations:
N = fEncoder(X) (1)
which are then used to decode target tokens [y1, y2, ..., ym] one by one. At the t-th decoding step, the Transformer takes yt as input, reusing the source encodings N and target memory Mt−1 which contains the history hidden states of all decoder layers corresponding to past tokens [y1, y2, ..., yt−1]:
yt+1,Mt = fDecoder(yt,Mt−1, N) (2)
This step not only generates a new token yt+1, but also updates the internal target memory Mt by concatenating Mt−1 with the newly calculated hidden states corresponding to yt.
Dangle differs from vanilla Transformers in that it concatenates the source input with the previously decoded target to construct target-dependent input for adaptive decoding:
Ct = [x1, x2, ..., xn, y1, ..., yt] (3)
Ht = fAdaptive_Encoder(Ct) (4)
The adaptive encoder consists of two components. Ct is first fed to k1 Transformer encoder layers to fuse the target information:
H̄t = fAdaptive_Encoder1(Ct) (5)
where H̄t is a sequence of contextualized representations [h̄t,1, h̄t,2, ..., h̄t,n, h̄t,n+1, ..., h̄t,n+t]. Then, the first n vectors corresponding to source tokens are extracted and fed to another k2 Transformer encoder layers for further processing:
Ht = fAdaptive_Encoder2(H̄t[: n]) (6)
Finally, the adaptive source encodings Ht together with the target context [y1, y2, ..., yt] are fed to a Transformer decoder to predict yt+1:
yt+1,Mt = fDecoder(y<t+1, {}, Ht) (7)
In a departure from vanilla Transformers, Dangle does not reuse the target memory from previous steps, but instead re-computes all target-side hidden states based on new source encodings Ht.
Similarly to Transformers, Dangle accesses source information at each decoding step via encoder-decoder attention layers where the same encodings Ht are used to compute both keys Kt and values Vt:
Kt = HtW K (8)
Vt = HtW V (9)
Ot = Attention(Qt,Kt, Vt) (10)
where key and value projections WK and W V are parameter matrices; and Qt, Kt, Vt, and Ot are respectively query, key, value, and output matrices, at time step t.

heading: 3 The R-Dangle Model
text: In this section, we describe the proposed model, which we call R-Dangle as a shorthand for Realworld Disentangled Transformer.

heading: 3.1 Re-encoding at Intervals
text: The need to perform re-encoding (and also redecoding) at each time step substantially increases Dangle’s training cost and memory footprint, so that it becomes computationally infeasible for real-world language tasks with very long target sequences (e.g., in the region of hundreds of tokens). Adaptively re-encoding at every time step essentially means separating out relevant source concepts for each prediction. However, the Transformer is largely capable of encoding source phrases and decoding corresponding target phrases (or logical form fragments in semantic parsing), as evidenced by its remarkable success in many machine translation and semantic parsing benchmarks (Vaswani et al., 2017; Keysers et al., 2020; Zheng and Lapata, 2021). This entails that the entanglement problem (i.e., not being able to disassociate the representations of different concepts for a sequence of predictions) does not occur very frequently. We therefore relax the strict constraint of re-encoding at every step in favor of the more flexible strategy of re-encoding at intervals.
Given source sequence X = [x1, x2, ..., xn], we specify P = [t1, t2, ..., tl](ti+1 − ti = o) in advance, i.e., a sequence of re-encoding points with interval o. Then, during decoding, when reaching a re-encoding point t(t = ti), we update source encodings Ht and target memory Mt:
Ht = fAdaptive_Encoder(Ct) (11)
yt+1,Mt = fDecoder(y<t+1, {}, Ht) (12)
where fAdaptive_Encoder denotes the adaptive encoder described in Section 2. For the next time step t(ti < t < ti+1), we fall back to the vanilla Transformer decoder using the source encodings Hti computed at time step ti:
yt+1,Mt = fDecoder(yt,Mt−1, Hti) (13)
Note that we always set t1 to 1 to perform adaptive encoding at the first time step.

heading: 3.2 Disentangling Keys and Values
text: During decoding, Dangle accesses source information via cross-attention (also known as encoderdecoder attention) layers where the same source encodings are used to compute both keys and values. The core design principle underlying Dangle is that learning specialized representations for different purposes will encourage the model to zero in on relevant concepts, thereby disentangling their representations. Based on the same philosophy, we assume that source keys and values encapsulate different aspects of source information, and that learning more specialized representations for them would further improve disentanglement, through the separation of the concepts involved.
A straightforward way to implement this idea is using two separately parameterized encoders to calculate two groups of source encodings (i.e., corresponding to keys and values, respectively) during re-encoding. However, in our preliminary experiments, we observed this leads to serious overfitting and performance degradation. Instead, we propose to encode values once and update keys only during adaptive encoding. We compute source values via the standard Transformer encoder:
Hv = fEncoder(X) (14)
and adaptively re-encode source keys at an interval:
Hkt = fAdaptive_Encoder(Ct) (15)
yt+1,Mt = fKV_Decoder(y<t+1, {}, Hv, Hkt ) (16)
where fKV_Decoder denotes a slightly modified Transformer decoder where source keys and values in each cross-attention layer are calculated based on different source encodings:
Kt = H k t W K (17)
V = HvW V (18)
Ot = Attention(Qt,Kt, V ) (19)
At time step t (where ti < t < ti+1), we perform vanilla Transformer decoding:
yt+1,Mt = fKV_Decoder(yt,Mt−1, Hv, Hkti) (20)
Note that fully sharing values could potentially cause some entanglement, however, we we did not observe this in practice. We also experimented with a variant where keys are shared and values are repeatedly re-computed but empirically observed
it obtains significantly worse generalization performance than the value-sharing architecture described above. This indicates that entanglement is more likely to occur when sharing keys.

heading: 4 A Real-world Compositional Generalization Challenge
text: Models of compositional generalization are as good as the benchmarks they are evaluated on. A few existing benchmarks are made of artificially synthesized examples using a grammar or rules to systematically control for different types of generalization (Lake and Baroni, 2018a; Kim and Linzen, 2020; Keysers et al., 2020; Li et al., 2021). Unfortunately, synthetic datasets lack the complexity of real natural language and may lead to simplistic modeling solutions that do not generalize to real world settings (Dankers et al., 2022). Other benchmarks (Finegan-Dollak et al., 2018; Shaw et al., 2021) focus on naturally occurring examples but create train-test splits based on the properties of their formal meaning representations (e.g., logical forms). However, formal annotations of meaning are not readily available for tasks other than semantic parsing. Since compositional generalization is a general problem, it is desirable to define it on the basis of natural language alone rather than by means of semantic parsing and the availability of formal annotations.
It is fair to assume that a SOTA model deployed in the wild, e.g., a Transformer-based machine translation system, will be constantly presented with new test examples. Many of them could be similar to seen training instances or compositionally different but in a way that does not pose serious generalization challenges. An ideal benchmark for evaluating compositional generalization should therefore consist of phenomena that are of practical interest while challenging for SOTA models. To this end, we create ReaCT, a new REAlworld dataset for Compositional generalization in
machine Translation. Our key idea is to obtain a generalization test set by detecting compositional patterns in relation to an existing training set from a large and diverse pool of candidates. Specifically, we use the IWSLT 2014 German → English dataset as our training corpus and the WMT 2014 German → English shared task as our test corpus (see Section 5 for details) and detect from the pool of WMT instances those that exemplify compositional generalization with respect to IWSLT. This procedure identifies naturally occurring compositional patterns which we hope better represent practical generalization requirements than artificially constructed challenges.
In the following, we describe how we identify examples that demand compositional generalization. While we create our new benchmark with machine translation in mind, our methodology is general and applicable to other settings such as semantic parsing. For instance, we could take a relatively small set of annotated user queries as our training set and create a generalization challenge from a large pool of unlabeled user queries.
Filtering Out-of-Vocabulary Atoms Compositional generalization involves generalizing to new compositions of known atoms. The WMT corpus includes many new semantic and syntactic atoms that are not attested in IWSLT. A large number of these are out-of-vocabulary (OOV) words which are by definition unknown and out of scope for compositional generalization. We thus discard WMT examples with words occurring less than 3 times in the IWSLT training set which gives us approximately a pool of 1.3M examples. For simplicity, we do not consider any other types of new atoms such as unseen word senses or syntactic patterns.
Measuring Compositionality How to define the notion of compositional generalization is a central question in creating a benchmark. Previous definitions have mostly centered around linguistic notions such as constituent or context-free gram-
mars (Kim and Linzen, 2020; Keysers et al., 2020; Li et al., 2021). These notions are appropriate for synthetic examples or logical forms as their underlying hierarchical structures are well-defined and can be obtained with ease.
Since we do not wish to synthesize artificial examples but rather detect them in real-world utterances, relying on the notion of constituent might be problematic. Sentences in the wild are often noisy and ungrammatical and it is far from trivial to analyze their syntactic structure so as to reliably identify new compositions of known constituents. We overcome this problem by devising a metric based on n-gram matching which assesses how compositional a certain example is with respect to a training corpus.
Specifically, we first create a lookup dictionary of atomic units by extracting all n-grams that occur more than 3 times in the training corpus. Given a candidate sentence, we search the dictionary for the minimum number of n-grams that can be composed to form the sentence. For example, for sentence “x1x2x3x4x5” and dictionary (x1, x2, x3x4, x5, x1x2, x3x4x5, ), the minimum set of such n-grams is (x1x2, x3x4x5). A sentence’s compositional degree with respect to the training corpus is defined as the ratio of the minimum number of n-grams to its length (e.g., 2/5 = 0.4 for the above example). We select the top 60,000 non-overlapping examples with the highest compositional degree as our candidate pool. As we discuss in Section 6, compositional degree further allows us to examine at a finer level of granularity how model performance changes as test examples become increasingly compositional.
Estimating Uncertainty Examples with the same compositional degree could pose more or less difficulty to neural sequence models (see last two utterances in Table 1). Ideally, we would like to identify instances that are compositional in terms of surface form and hard in terms of the underlying generalization (see third example in Table 1). We detect such examples using a metric based on uncertainty estimation and orthogonal to compositional degree. We quantify predictive uncertainty based on model ensembles, a method which has been successfully applied to detecting misclassifications and out-of-distribution examples (Lakshminarayanan et al., 2017; Malinin and Gales, 2021).
We follow the uncertainty estimation framework introduced in Malinin and Gales (2021) for se-
quence prediction tasks. Specifically, we train 10 Transformer models with different random initializations on IWSLT (our training corpus), and run inference over the candidate pool created in the previous stage; for each example in this pool, we measure the disagreement between ensemble models using reverse mutual information, a novel measure (Malinin, 2019; Malinin and Gales, 2021) which quantifies knowledge uncertainty, i.e., a model’s uncertainty in its prediction due to lack of understanding of the data rather than any intrinsic uncertainty associated with the task (e.g., a word could have multiple correct translations). We use the token-level approximation of knowledge uncertainty.
We empirically find that the most uncertain examples are extremely noisy and barely legible (e.g., they include abbreviations, typos, and nonstandard spelling). We therefore throw away the top 2,000 uncertain examples and randomly sample 3,000 instances from the next 18,000 most uncertain examples in an attempt to create a generalization test set with diverse language patterns and different levels of uncertainty.
Analysis We analyze the compositional nature of ReaCT by comparing it to several popular benchmarks. Specifically, for all datasets, we count the number of novel test set n-grams that have not been seen in the training. We extract n-grams over words and parts of speech (POS); word-based n-grams represent more superficial lexical composition while n-grams based on POS tags reflect more of syntactic composition.
As shown in Table 2, despite being considerably smaller compared to other benchmarks (see # examples column), ReaCT presents substantially more diverse patterns in terms of lexical and syntactic composition. It displays a much bigger number of novel word n-grams, which is perhaps not surprising. Being a real-world dataset, ReaCT has a larger vocabulary and more linguistic variation. While
our dataset creation process does not explicitly target novel syntactic patterns (approximated by POS n-grams), ReaCT still includes substantially more compared to other benchmarks. This suggests that it captures the complexity of real-world compositional generalization to a greater extent than what is achieved when examples are synthesized artificially. We show examples with novel POS n-gram compositions in Table 3).

heading: 5 Experimental Setup
text: Datasets We evaluated R-Dangle on two machine translation datasets and one semantic parsing benchmark which we selected to maximally reflect natural language variations and real-world generalization challenges. These include: (a) ReaCT, the machine translation benchmark developed in this paper; we used the IWSLT 2014 De→En test set as an in-domain test set and created an outof-distribution test set from the WMT’14 De→En training corpus; (b) CoGnition (Li et al., 2021) is a semi-natural machine translation benchmark focusing on English-Chinese sentence pairs; source sentences were taken from the Story Cloze Test and ROCStories Corpora (Mostafazadeh et al., 2016, 2017), and target sentences were constructed by post-editing the output of a machine translation engine; (c) SMCalFlow-CS (Andreas et al., 2020) is a semantic parsing dataset for task-oriented dialogue, featuring real-world human-generated utterances about calendar management; following previous work (Yin et al., 2021; Qiu et al., 2022), we report experiments on the compositional skills split, considering a few-shot learning scenario (with 6, 16, and 32 training examples). See Appendix A for more details on these datasets.
Models On machine translation, our experiments evaluated two variants of R-Dangle depending on whether keys and values are shared (R-Dangleshr) or separate (R-Danglesep). We implemented all machine translation models with fairseq (Ott et al., 2019). We compared R-Dangle against a vanilla Transformer (Vaswani et al., 2017) and the original Dangle model (Zheng and Lapata, 2022) which used the popular fairseq configuration transformer_iwslt_de_en. We also implemented bigger variants of these models using 12 encoder layers and 12 decoder layers which empirically led to better performance.
R-Dangleshr and R-Danglesep also use a 12-layer decoder. We tuned the number of layers of the adaptive components (k1 = 2 and k2 = 10) on the development set. For R-Danglesep, we adopted a 10-layer value encoder and a 10-layer key encoder (k1 = 2 and k2 = 8), with the top 8 layers in the two encoders being shared. This configuration produced 12 differently parametrized transformer encoder layers, maintaining identical model size to comparison systems.
Previous work (Qiu et al., 2022) has shown the advantage of pre-trained models on the SMCalFlow-CS dataset. For our semantic parsing experiments, we therefore built R-Dangle on top of BART-large (Lewis et al., 2020). We only report results with R-Dangleshr as the R-Danglesep architecture is not compatible with BART. We again set k1 = 2 and k2 = 10. We provide more detail on model configurations in Appendix B.

heading: 6 Results
text: Disentangling Keys and Values Improves Generalization Table 4 reports the BLEU score (Papineni et al., 2002) achieved by the two R-Dangle
variants on ReaCT and CoGnition, across different re-encoding intervals. R-Danglesep is consistently better than R-Dangleshr which confirms that representing keys and values separately is beneficial. We also observe that smaller intervals lead to better performance (we discuss this further later).
Table 5 compares R-Danglesep (with interval 1) against baseline models. In addition to BLEU, we report novel compound translation error rate, a metric introduced in Li et al. (2021) to quantify the extent to which novel compounds are mistranslated. We compute error rate over instances and an aggregate score over contexts. R-Danglesep delivers compositional generalization gains over Dangle and vanilla Transformer models (both in terms of BLEU and compound translation error rate), even though their performance improves when adopting a larger 12-layer network. R-Danglesep achieves a new state of the art on CoGnition (a gain of 0.9 BLEU points over Dangle and 1.5 BLEU points over the Transformer baseline). R-Danglesep fares similarly on ReaCT; it is significantly superior to the Transformer model by 0.9 BLEU points, and Dangle by 0.5 BLEU points. Moreover, improvements on compositional generalisation are not at the expense of in-domain performance (R-Dangle obtains similar performance to the Transformer and Dangle on the IWSLT2014 in-domain test set).
R-Dangle Can Handle Long-tail Compositional Patterns Better We next examine model performance on real-world examples with diverse language and different levels of composition. Specifically, we train R-Danglesep (interval=1) and a Transformer on the IWSLT14 corpus and test on the pool of 1.3M WMT examples obtained after filtering OOV words. Figure 1a plots the difference in BLEU between the two models against compositional degree. This fine-grained evaluation reveals that they perform similarly on the major-
ity of less compositional examples (BLEU difference is around zero), however, the performance gap becomes larger with more compositional examples (higher difference means higher BLEU for R-Danglesep). This indicates that R-Dangle is particularly effective for handling long-tail compositional patterns.
R-Dangle Boosts the Performance of Pretrained Models The “pre-train and fine-tune” paradigm (Peters et al., 2018; Devlin et al., 2019; Raffel et al., 2020; Lewis et al., 2020) has been widely adopted in NLP, and semantic parsing is no exception (Shin et al., 2021; Qiu et al., 2022). We further investigate R-Dangle’s performance when combined with a pre-trained model on the SMCalFlow-CS dataset (across the three cross-domain settings). Table 6 shows that R-Dangleshr boosts the performance of BART-large, which suggests that generalization improvements brought by R-Dangle are complementary to generalization benefits afforded by largescale pre-training (see Zheng and Lapata 2022 for a similar conclusion). The proposed model effectively marries pre-training with disentangled representation learning to achieve better generalization.
In Table 6, we also compare R-Dangle with other top-performing models on SMCalFlow-CS. These include: (a) a sequence-to-sequence model with a BERT encoder and an LSTM decoder using a copy mechanism (BERT2SEQ; Yin et al. 2021); (b) the coarse-to-fine (C2F) model of Dong and Lapata (2018) which uses a BERT encoder and a structured decoder that factorizes the generation of a program into sketch and value predictions; (c) and combi-
nations of these two models with span-supervised attention (+SS; Yin et al. 2021). We also include a T5 model and variant thereof trained on additional data using a model called Compositional Structure Learner (CSL) to generate examples for data augmentation (T5+CSL; Qiu et al. 2022). R-Dangle with BART performs best among models that do not use data augmentation across compositional settings. Note that our proposal is orthogonal to CSL and could also benefit from data augmentation.
Larger Re-encoding Intervals Reduce Training Cost The results in Table 4 indicate that reencoding correlates with R-Dangle’s generalization ability, at least for machine translation. Both model variants experience a drop in BLEU points when increasing the re-encoding interval to 8. We hypothesize that this sensitivity to interval length is task-related; target sequences in machine translation are relatively short and representative of real language, whereas in SMCalFlow-CS, the average length of target sequences (in formal language) is 99.5 and the maximum length is 411. It is com-
putationally infeasible to train R-Dangle with small intervals on this dataset, however, larger intervals still produce significant performance gains.
Figure 1b shows how accuracy and training time vary with interval length on SMCalFlow-CS with the 16-C setting. Larger intervals substantially reduce training cost with an optimal speed-accuracy trade off in between 10 and 50. For instance, interval 40 yields a 4x speed-up compared to interval 10 while achieving 50.3% accuracy. Finding a tradeoff between generalization and efficiency is an open research problem which we leave to future work.

heading: 7 Related Work
text: The realization that neural sequence-to-sequence models struggle with compositional generalization has led to numerous research efforts aiming to precisely define this problem and explore possible solutions to it. A line of research focuses on benchmarks which capture different aspects of compositional generalization. Finegan-Dollak et al. (2018) repurpose existing semantic parsing benchmarks for compositional generalization by creating more challenging splits based on logical form patterns. In SCAN (Lake and Baroni, 2018b) compositional generalization is represented by unseen combinations of seen actions (e.g., JUMP LTURN). Keysers et al. (2020) define compositional generalization as generalizing to examples with maximum compound divergence (e.g., combinations of entities and relations) while guaranteeing similar atom distribution to the training set. Kim and Linzen (2020) design five linguistic types of compositional generalization such as generalizing phrase nesting to unseen depths. In ReaCT, our definition of compositional generalization is dependent on the data distribution of the candidate corpus, which determines what compositional patterns are of practical
interest and how frequently they occur. Another line of work focuses on modeling solutions, mostly ways to explicitly instil compositional bias into neural models. This can be achieved by adopting a more conventional grammar-based approach (Herzig and Berant, 2021) or incorporating a lexicon or lexicon-style alignments into sequence models (Akyurek and Andreas, 2021; Zheng and Lapata, 2021). Other work employs heuristics, grammars, and generative models to synthesize examples for data augmentation (Jia and Liang, 2016; Akyürek et al., 2021; Andreas, 2020; Wang et al., 2021; Qiu et al., 2022) or modifies standard training objectives with new supervision signals like attention supervision or meta-learning (Oren et al., 2020; Conklin et al., 2021; Yin et al., 2021). Our work builds on Dangle (Zheng and Lapata, 2022), a disentangled sequence-to-sequence model, which tries to tackle compositional generalization with architectural innovations. While Dangle is conceptually general, our proposal is tailored to the Transformer and features two key modifications to encourage more disentangled representations and better computational efficiency.

heading: 8 Conclusions
text: In this paper we focused on two issues related to compositional generalization. Firstly, we improve upon Dangle, an existing sequence-to-sequence architecture that generalizes to unseen compositions by learning specialized encodings for each decoding step. We show that re-encoding keys periodically, at some interval, improves both efficiency and accuracy. Secondly, we propose a methodology for identifying compositional patterns in real-world data and create a new dataset which better represents practical generalization requirements. Experimental results show that our modifications improve generalization across tasks, metrics, and datasets and our new benchmark provides a challenging testbed for evaluating new modeling efforts.
Limitations
Our machine translation experiments revealed that optimal generalization performance is obtained with small interval values. However, R-Dangle with small intervals still runs much slower than an equivalent Transformer model. Despite our modifications, large R-Dangle models with small intervals on large datasets remain computationally expensive. In this paper, we only explored a sim-
ple periodic re-encoding strategy. However, more complex and flexible ways of re-encoding could be used to further improve computational efficiency. For instance, we could adopt a dynamic strategy which learns when re-encoding is necessary.

heading: Acknowledgments
text: We thank our anonymous reviewers for their feedback. We also wish to thank Tom Hosking and Biao Zhang for helpful discussions. Finally, we gratefully acknowledge the support of the UK Engineering and Physical Sciences Research Council (grant EP/W002876/1).

heading: A Dataset Details
text: We evaluated our model on two machine translation datasets, and one semantic parsing benchmark which we selected to maximally reflect natural language variations and real-world generalization challenges. We describe these in detail below.
ReaCT is the real-world machine translation benchmark developed in this paper for compositional generalization. The IWSLT 2014 De→En dataset consists of approximately 170K sequence pairs. We used the fairseq script prepare-iwslt14.sh to randomly sample approximately 4% of this dataset as validation set and kept the rest as training set. Following standard practice, we created an in-domain test set, the concatenation of files dev2010, dev2012, tst2010, tst2011, and tst2012. We created an outof-distribution test sets from the WMT’14 De→En training corpus following the uncertainty selection method based on sequences.
CoGnition is another machine translation benchmark targeting compositional generalization (Li et al., 2021). It also contains a synthetic test set to quantify and analyze compositional generalization of neural MT models. This test set was constructed by embedding synthesized novel compounds into training sentence templates. Each compound was combined with 5 different sentence templates, so that every compound can be evaluated under 5 different contexts. A major difference between REACT and CoGnition is the fact that test sentences for the latter are not naturally occurring. Despite being somewhat artificial, CoGnition overall constitutes a realistic benchmark which can help distinguish subtle model differences compared to purely synthetic benchmarks. For example, Zheng and Lapata (2022) showed that their encoder-only Dangle variant performed badly on this dataset in spite of impressive performance on synthetic semantic parsing benchmarks (Kim and Linzen, 2020; Keysers et al., 2020).
SMCalFlow-CS (Andreas et al., 2020) is a largescale semantic parsing dataset for task-oriented
dialogue, featuring real-world human-generated utterances about calendar management. Yin et al. (2021) proposed a compositional skills split of SMCalFlow (SMCalFlow-CS) that contains singleturn sentences from one of two domains related to creating calendar events (e.g., Set up a meeting with Adam) or querying an org chart (e.g., Who are in Adam’s team? ), paired with LISP programs. The training set S consists of samples from single domains while the test set C contains compositions thereof (e.g., create a meeting with Adam and his team). Since zero-shot compositional generalization is highly non-trivial due to novel language patterns and program structures, we follow previous work (Yin et al., 2021; Qiu et al., 2022) and consider a few-shot learning scenario, where a small number of cross-domain examples are included in the training set. We report experiments with 6, 16, and 32 examples.
B Implementation Details
Machine Translation Models We implemented all translation models with fairseq (Ott et al., 2019). Following previous work (Li et al., 2021; Zheng and Lapata, 2022), we compared with the baseline machine translation models Dangle and Transformer using the popular fairseq configuration transformer_iwslt_de_en. We also implemented a bigger variant of these models using a new configuration, which empirically obtained better performance. We used 12 encoder layers and 12 decoder layers. We set the dropout to 0.3 for attention weights and 0.4 after activations in the feedforward network. We also used pre-normalization (i.e., we added layer normalization before each block) to ease optimization. Following Zheng and Lapata (2022), we used relative position embeddings (Shaw et al., 2018; Huang et al., 2020) which have demonstrated better generalization performance.
Hyperparameters for R-Dangle were tuned on the respective validation sets of CoGnition and ReaCT. Both R-Dangleshr and R-Danglesep used a 12-layer decoder. For R-Dangleshr, we tuned the number of layers of the two adaptive components k1 and k2, and set k1 and k2 to 2 and 10, respectively. For R-Danglesep, we shared some layers of parameters between the value encoder and the adaptive key decoder and experimented with different sharing strategies. Finally, we adopted a 10-layer value encoder and a 10-layer key encoder (k1 = 2
and k2 = 8). The top 8 layers in the two encoders were shared. This configuration produced 12 differently parametrized transformer encoder layers, thus maintaining identical model size to the baseline.
Semantic Parsing Models Qiu et al. (2022) showed the advantage of pre-trained sequence-tosequence models on SMCalFlow-CS. We therefore built R-Dangle on top of BART-large (Lewis et al., 2020), which is well supported by fairseq. We used BART’s encoder and decoder to instantiate the adaptive encoder and decoder in our model. For compatibility, we only employ the R-Dangleshr architecture. We also set k1 and k2 to 2 and 10, respectively.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: the limitations section

heading: 3 A2. Did you discuss any potential risks of your work?
text: the limitations section

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? The section 4 and 5

heading: 3 B1. Did you cite the creators of artifacts you used?
text: The section 5
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We were unable to find the license for the dataset we used
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We reuse existing datasets to create our benchmark.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. the section 4
C 3 Did you run computational experiments? the section 5 and 6

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? the section 6
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? the section 5 and appedix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? the section 6

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? the section 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Finding a tradeoff between generalization and efficiency is an open research problem which we leave to future work