Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization
Authors: Lei Lin
Shuangtao Li
Yafang Zheng
Biao Fu
shan liu
Yidong Chen
Xiaodong Shi
Venue: EMNLP 2023 Findings (2023)
Keywords: Compositional Generalization
Seq2Seq Models
Machine Translation
Semantic Parsing

Abstract: Recent studies have shown that sequence-to-sequence (seq2seq) models struggle with compositional generalization (CG), i.e., the ability to systematically generalize to unseen compositions of seen components. There is mounting evidence that one of the reasons hindering CG is the representation of the encoder uppermost layer is entangled, i.e., the syntactic and semantic representations of sequences are entangled. However, we consider that the previously identified representation entanglement problem is not comprehensive enough. Additionally, we hypothesize that the source keys and values representations passing into different decoder layers are also entangled.
Starting from this intuition, we propose \textsc{CompoSition} (\textbf{Compo}se \textbf{S}yntactic and Semant\textbf{i}c Representa\textbf{tion}s), an extension to seq2seq models which learns to compose representations of different encoder layers dynamically for different tasks, since recent studies reveal that the bottom layers of the Transformer encoder contain more syntactic information and the top ones contain more semantic information. Specifically, we introduce a \textit{composed layer} between the encoder and decoder to compose different encoder layers' representations to generate specific keys and values passing into different decoder layers. \textsc{CompoSition} achieves competitive results on two comprehensive and realistic benchmarks, which empirically demonstrates the effectiveness of our proposal. Codes are available at~\url{https://github.com/thinkaboutzero/COMPOSITION}.

Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization
Lei Lin Shuangtao Li Yafang Zheng!? 1.2" Biao Fu
Shan Liu
Yidong Chen Xiaodong Shi
Department of Artificial Intelligence, School of Informatics, Xiamen University Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China {linlei jēstu. xmu edu. cn, cncccheenn mandel j0xmu. bexxmu.eda. edu. cn
Abstract
Known Components
xihuan
Recent studies have shown that sequence-to- sequence (seg2seq) models struggle with com- positional generalization (CG), i.c., the ability systematically generalize unseen compo- sitions of seen components. There is mount- ing evidence that one of the reasons hindering CG is the representation of the encoder upper- most layer is entangled, i.e., the syntactic and semantic representations of sequences are en- tangled. However, consider that the pre- viously identified representation entanglement problem not comprehensive enough. Addi- tionally, we hypothesize that the source keys and values representations passing into differ- ent decoder layers also entangled. Start- ing from this intuition, we propose COMPOSI- TION (Compose Syntactic and Semantic Rep- resentations), an extension to seq2seq mod- els which learns to compose representations of different encoder layers dynamically for dif- ferent tasks, since recent studies reveal that the bottom layers of the Transformer encoder contain more syntactic information and the top ones contain more semantic information. Specifically, we introduce a composed layer between the encoder and decoder com- pose different encoder layers' representations to generate specific keys and values passing into different decoder layers. COMPOSITION achieves competitive results on two compre- hensive and realistic benchmarks, which empir- ically demonstrates the effectiveness of our pro- posal. Codes are available at https: //gtthb.. wwwnkaaouuteeee/COooo
he liked E
Semantic
diushi le gou
lost the dog
23
3
How!
Syntactic
Pattern 2.3: V DET N MOD
Novel Compound
lost the dog he liked Semantic Syntactic
diushi
Xiluaan de gou 2 RD 3
Figure I: Examples from CoGnition (Li et al., 2021) show the workflow of how humans exhibit CG. Suppose interpreters know the translation: 1 [a]] for "lost the dog" and [ XXX for "he liked" (semantic infor- mation). When they first encounter "lost the dog he liked", they can correctly translate 1 ttt # Xx ø6j instead of [ tt # x depending on Pattern 2.3 (syntactic information).
2017). For example, if person knows "the doctor watches movie" (B t2 J and "the lawyer" [ Jifi then is natural for the person to know the translation of "the lawyer watches movie" is # even though they have never seen it before. Such nature is beneficial for generalizing to new compositions of previously observed elements, which is often required in real-world scenarios.
Despite astonishing successes across a broad range of natural language understanding and gen- cration tasks (Sutskever et al., 2014; Dong and La- pata, 2016; Vaswani et al., 2017), neural network models, in particular the very popular sequence-to- sequence (seq2seq) architecture, are argued diffi- cult to capture the compositional structure of hu- man language (Lake and Baroni, 2018; Keysers et al., 2020; Li et al., 2021). A key reason for failure on CG is different semantic factors (c.g., lexical meaning and syntactic patterns) required by CG are entangled, which was proved explicitly or implicitly to exist in the representation of the encoder uppermost layer (encoder entanglement
1 Introduction
A crucial property of human language learning is its compositional generalization (CG) the algebraic ability to understand and produce potentially in- finite number of novel combinations from known components (Fodor and Pylyshyn, 1988; Lake et al.,
"Equal Contribution. Corresponding Author,
'The sentence in "[J" denotes the Chinese translation.

---

problem) by previous studies (Li al., 2019; Rau- nak al., 2019; Russin et al., 2019; Liu et al.,
of the encoder for different tasks (i.e., For A task, the information at encoder layer 0 may be more important, however, for B task, the information at encoder layer 5 may be more important). Addition- ally, the model can select which encoder layer of information is most suitable for itself (that is, which encoder layer of information is the most important) for a particular task. Inspired by that, propose the composed layer (leamable scalars or vectors) generate different specific souree keys and values passing into different decoder layers for different particular tasks, since we argue that the learned scalars or vectors (i.e., different dynamic compo- sition modes) by the model itself during training process can be dynamically adjusted for different particular tasks, and provide way to learn pref- erence of layers in different levels of the encoder for a particular task. Putting everything together, we propose COMPOSITION (Compose Syntactic and Semantic Representations), an extension to seq2seq models that learns to compose the syn- tactic and semantic representations of sequences dynamically for different tasks. COMPOSITION is simple yet effective, and mostly applicable to any seq2seq models without any dataset or task-specific modification.
2020b, 2021; Jiang and Bansal, 2021; Zheng and Lapata, 2022a; Yin et al., 2022; Ruis and Lake, 2022; Li et al., 2022; Cazzaro et al., 2023). In other words, the syntactic and semantic representations of sequences are entangled.
In order to alleviate the encoder entanglement problem, one line of research on CG mainly con- centrate improving the encoder representation separating the learning of syntax and semantics which adopt similar approaches to humans' strate- gies for CG (see Figure 1). Specifically, several works either produce two separate syntactic and se- mantic representations, and then compose them (Li et al., 2019; Russin ct al., 2019; Jiang and Bansal, 2021) or design external modules, and then employ a multi-stage generation process (Liu et al., 2020b, 2021; Ruis and Lake, 2022; Li al., 2022; Caz- zaro et al.. 2023). Moreover, some studies explore bag-of-words pre-training (Raunak al., 2019), newly decoded target context (Zheng and Lapata, 2022a,b) or prototypes token representations over the training set (Yin et al., 2022) to improve the encoder representation. Furthermore, we hy- pothesize that the source keys and values represen- tations passing into different decoder layers also entangled (keys, values entanglement problem), not just the representation of the encoder uppermost layer. We will further illustrate it in Section 5.1.
Experimental results on CFQ (Keysers et al., 2020) (semantic parsing) and CoGnition (Li et al., 2021) (machine translation, MT) empirically show that our method can improve generalization per- formance, outperforming competitive baselines and other techniques. Notably, COMPOSITION achieves 19.2% and 50.2% (about 32%, 20% relative improvements) for instance-level and aggregate-leve error reduction rates on CoGnition. Extensive analyses demonstrate that composing the syntactic and semantic representations of se- quences dynamically for different tasks leads to better generalization results.
Therefore, one natural question can be raised: how to alleviate keys, values entanglement prob- lem. As a remedy, we examine CG from a new perspective to solve it, i.e., utilizing different en- coder layers' information. We conduct preliminary analysis provided in Appendix A, and conclude that the bottom layers of the Transformer encoder con- tain more syntactic information and the top ones contain more semantic information. Inspired by this, we collect representations outputted by each encoder layer instead of separating the learning of syntax and semantics. So one intuitive solution to solve keys, values entanglement problem is to leam different and specific combinations of syntac- tic and semantic information (i.e., representations outputted by each encoder layer) for keys and val- ues of different decoder layers. We argue that an effective composition is to provide different combi- nations for different tasks and a specific combina- tion for a particular task. For example, the model can learn preference of layers in different levels
Related Work
Compositional Generalization. After realizing existing neural models still struggle in scenarios requiring CG (Lake and Baroni, 2018; Keysers et al., 2020; Li et al., 2021), there have been various studies attempt improve the model's ability of CG, including data augmentation (Andreas, 2020:
Akyürek al., 2021; Yang et al., 2022; Li et al., 2023), modifications on model architecture (Li
et al., 2019; Russin et al., 2019; Nye et al., 2020; Liu et al., 2020c, 2021: Zheng and Lapata, 2021;

---

Herzig and Berant, 2021; Chaabouni et al., 2021;
consider that there are also three distinct differ- ences. Firstly, our method exploits information from encoder sub-layers and generates specific keys, values passing into different decoder layers while they do not. Secondly, our method shows the effectiveness of utilizing low-layer encoder rep- resentations while they have the opposite view (see Appendix D). Thirdly, we do not share the same motivation task. Their work focuses how transform information across layers deep neural network scenarios for seq2seq tasks. Our moti- vation is to compose the syntactic and semantic representations of sequences dynamically for CG.
Mittal al., 2022; Zheng et al., 2023), interme- diate representations (Furrer et al., 2020; Herzig et al., 2021), meta-learning (Lake, 2019; Conklin et al., 2021), explorations on pre-trained language models (Furrer et al., 2020; Zhou et al., 2023), aux- iliary objectives (Jiang and Bansal, 2021; Yin et al.. 2023), two representations (Li et al., 2019; Russin al., 2019; Jiang and Bansal, 2021) and enriching the encoder representation (Raunak al., 2019; Zheng and Lapata, b022a,b; Yin al., 2022; Yao and Koller, 2022). One line of research exploring how to alleviate the encoder entanglement problem has attracted much attention. Our work is in line with it, but we examine CG from new perspective, i.e., utilizing different encoder layers' information.
Methodology
We adopt the Transformer architecture (Vaswani et al., 2017) to clarify our method, however, proposed method is mostly applicable to any seq2seq models. In the following, we first intro- duce the Transformer baseline (Section 3.1), and then our proposed COMPOSITION (Section 3.2).
Neural Machine Translation. Recently, CG and robustness of Neural Machine Translation (NMT) have gained much attention from the research community (Cheng et al., 2020; Xu al., 2021; Lake and Baroni, 2018; Li al., 2021), includ- ing pre-training (Raunak al., 2019), data aug- mentation (Guo et al., 2020a), datasets (Li et al., 2021), and enriching semantic information at token- level (Thrush, 2020; Akyurek and Andreas, 2021; Zheng and Lapata, 2022a,b; Yin et al., 2022). Note- worthily, Dankers et al. (2022) argue that MT is suitable and relevant testing ground to test CG in natural language. Different from them, we in- troduce composed layer to compose different en- coder layers' information dynamically, which is inspired by previous studies about analyzing Trans- former (Raganato et al., 2018; Voita et al., 2019).
3.1 Transformer
The Transformer (Vaswani et al., 2017) is designed for sequence to sequence tasks which adopts an encoder-decoder architecture. The multi-layer en- coder summarizes source sequence into con- textualized representation and another multi-layer decoder produces the target sequence conditioned on the encoded representation.
Formally, given a sequence of source sentence X { ..}} and a sequence of target where S, T denote
sentence Y = the number of source and target tokens, respec- tively. D = ((X,Y),.... y).... denotes a training cor- pus, V denotes the vocabulary of D, and 0 de- notes parameters of the Transformer model. The model aims to estimate the conditional probability p(y,,..., 21, ............... Is):
Encoder Layer Fusion. Encoder layer fusion (En- coderFusion) is a technique to fuse all the encoder layers (instead of the uppermost layer) for seq2seq models, which has been proven beneficial, such layer attention (Bapna et al., 2018; Shen al., 2018; Wang al., 2019). layer aggregation (Dou al., 2018; Wang et al., 2018; Dou et al., 2019), and layer-wise coordination (He et al., 2018; Liu et al., 2020a). However, other studies show that exploiting low-layer encoder representations fails to improve model performance (Domhan, 2018). The essence of different EncoderFusion works is to explore different ways to combine information from different encoder layers. Our approach is es- sentially the same as EncoderFusion work, which explores different ways to combine information from different encoder layers, however, we pro- pose a new way to combine them. Meanwhile, we
T+1
p( = II
X;0),
(1)
where t is the index of each time step, Fet denotes prefix of Y and each factor p(u X, ......, ........ St
0) defined as a softmax distribution of V. During training, the model are generally opti- mized with the cross-entropy (CE) loss, which is calculated as follows:
LCE - Σ p(veleeee X;O) (2)

---

Ouiput Probubiitiiss
the Transformer encoder first maps X to embed- dings Ho. and then Ho are fed into a Transformer self-attention sub-layer and feed-forward sub-layer to generate H E I H E R respec- tively, where d denotes the hidden size. Next, each subsequent encoder layer takes the previous layer's output as input. The overall process is as follows:
Linear
Nomn
Add
Multi Hoad Attention
11 = T (Ho),
(3)
Ni
Addi de Normie
H f -Feed-Forrune Forward H.S.).
(4)
FoddForrord Sub-Layer
Liurr
Mx
Blat- Aficctios
H fSelff Attention ( HFF))
(5)
Self-Altattinn Schelayer
11 f ao SA)
(6)
Posttinal Enooding
+
Pesitional Encoding
where 2 S i S M denote i-th encoder layer. Therefore,
Chijaai Embedking
Iibbeiding
collect representations out- putted by each encoder sub-layer Hoollet {IISA H1F H FF The keys and val- ues of multi-head attention module of decoder layer are defined to bc:
Inpuis
Ougputs (stitted night)
Figure 2: Architecture COMPOSITION based on the Transformer. The bright yellow block in the middle denotes the composed layer introduced in Section 3.2. The red line denotes that we collect representations of the same positions for the rest encoder layers.
w H collect Hbby
Σ
2M
During inference, the model predicts the proba- bilities of target tokens in an auto-regressive mode and generates target sentences using a heuristic search algorithm, such beam search (Freitag and Al-Onaizan, 2017).
Hyatu =
Σ
w. Hoolectss
(8)
where wy E wR E R are learnable scalars vectors and mutually different (e.g. w w. wa A w/ and w o w(), which weight each col- lected source representation in dynamic linear manner. Eq. and provide way to learn prefer- ence of sub-layers in different levels of the encoder.
3.2 COMPOSITION
Our proposed COMPOSITION extends the Trans- former by introducing composed layer between the encoder and decoder. Figure shows the over- all architecture of our approach.
Experiments
We mainly evaluate COMPOSITION on two com- prehensive and realistic benchmarks for measuring CG, including CFQ (Keysers et al., 2020) and CoG- nition (Li al., 2021).
3.2.1 Composed Layer
The composed layer is list consisting of 2.N leamn able vectors due 2N source keys, values passing into N' decoder layers, where each vector involves 2M learnable scalars or vectors. M, N denote the number of encoder and decoder layers respectively.
4.1 Experimental Settings
Datasets. CoGnition is a recently released realistic English -> Chinese (En-*Zh) translation dataset, which is used to systematically evaluate CG in MT scenarios. It consists of training set of 196,246 sentence pairs, validation set and test set of 10,000 samples. In particular, it also has dedi- cated synthetic test set (i.e., CG-test set) consisting of 10,800 sentences containing novel compounds, so that the ratio of compounds that are correctly translated can be computed to evaluate the model's
3.2.2 Dynamic Combination
Here, we describe how to use the composed layer to compose collected representations dynamically for generating specific keys and values represen- tations passing into different decoder layers. Let fsolf Attention and freed Forward denote a Trans- former self-attention sub-layer and feed-forward sub-layer respectively. The embedding layer of

---

Compound Translation Error Rate (CTER)
Model
#Params
BLEU 1
VP
PP
Total
A
Transformer
24.7%
24.8%/59.55 5%
35.7%
28.4%/62.99
Transformer-B Rell
35M
30.1%//58.19
27.6%/61.22% 2
38.5%/74.1
32.1 19/64.5
+3.7%/+1.66
59.1
Transforer--maan Small
25M
25.19//56.99 9%
25.6%/660399 3%
39. 19/19/5/500 0%
79 99/64455
+1 +1,59/+ 1.6%
59.0
Transformer- Deep
40M
23.3%
24.1 24.1%/55..0
33.8%//72.6%
27.0%/60.7%
-1.4%
60.1
Bow SeqMix Dungle
35M
22.2% 9%%
24.8%/55..64 66%
2..0%/73..22 2%
27.3%/58. 9%
-1.1%%
35M
24.5%
26.9%
34.4%//7.. 1
28.6%/60.6%
+0.2%
35M
-/-
24.4%/55.5
-5.0%/-7,4
Proto- Proto-Tranen Transformer
42M
14.15//3.5%%
22.1% 99%
28.99/68.21%
21.7%/51 8898
-6. 11.1--11.1
60.1
Transforer++CRgg
25M
--
+
20.2% 3%
-8.2%% 14.66
61.3
R-Danglćsop
70M
+
16.0%/42..1% %
-12.4%/-20.89
63.4
DLCL
35M
-/-
--
-/-
28.4%/67.99 9%%
+0.0%
59.2
COMPOSITION
35M
10.0% %
12.1%/
299.2%/68.0
20.4%/52.0P
8.0%
61.5
COMPOSITION- Rela
35M
15.5% 1356/399.26 2%
22.4%%554405 DS
29. 16/66..3
22.3%/53.559
-6.1 66.1%/-9.499 9.4%
61.6
COMPOSITION-
25M
14.356/40.3% 3%
24.4%
34.5% 4%%
24.4%/57.3%
-4.0% 6.6%
60.1
CoMPOSITION-Depp
40M
11.4%
19.5%
26.7%
19.2%
99.2%
62.0
Table CTERs (%) CoGnition. We report instance-level and aggregate-level CTERs in the CG-test set, separated by "Y". In addition, we also report the commonly used metric BLEU score in MT tasks. "." denotes that the results are not provided in the original paper. Results are averaged over random runs.
CQQ
CoGnition
tokenizer* to preprocess source and target English text.
1he MY Mic
y SuSt cuett TRNEE SErindmar SOAMET
Setup. For CoGnition and CFQ, follow the same experimental settings and configurations of Li et al. (2021) and Zheng and Lapata (2022a) rep- spectively. We implement all comparison models and COMPOSITION with an open source Fairseq toolkit (Ott et al., 2019). More details are provided in Appendix B.
Figure Examples of CFQ and CoGnition.
ability of CG directly. CFQ is automatically gen- erated from a set of rules in a way that precisely tracks which rules (atoms) and rule combinations (compounds) of each example. In this way, we can generate three splits with maximum compound di- vergence (MCD) while guaranteeing small atom divergence between train and test sets, where large compound divergence denotes the test set involves more examples with unseen syntactic structures. We evaluate our method on all three splits. Each split dataset consists of a training set of 95,743, a validation set and a test set of 1 11,968 examples. Figure 3 shows examples of them.
Evaluation Metrics. For CoGnition, pound translation error rate (CTER (Li et al.. 2021)) to measure the model's ability of CG. Specifically, instance-level CTER denotes the ratio of samples where the novel compounds are translated incor- rectly, and aggregate-leve CTER denotes the ratio of samples where the novel compounds suffer at least one incorrect translation when aggregating all 5 contexts. To calculate CTER, Li et al. (2021) manually construct dictionary for all the atoms based the training set, since each atom contains different translations. We also report character- level BLEU scores (Papineni et al., 2002) using SacreBLEU (Post, 2018) as a supplement. For CFQ, we use exact match accuracy to evaluate model performance, where natural language utter- ances are mapped to meaning representations.
Data Preprocess. We follow the same settings Li et al. (2021) and Keysers et al. (2020) to pre- process CoGnition and CFQ datasets separately. For CoGnition, we use open-source Chinese to- kenizer to preprocess Chinese and apply Moses tokenizer? preprocess English, which is the same in Lin al. (2023) and Liu et al. (2023). We em- ploy byte-pair encoding (BPE) (Sennrich et al., 2016) for Chinese with 3.000 merge operations, generating a vocabulary 5,500 subwords. We do not apply BPE for English due to the small vocabu- lary (i.e., 2000). For CFQ, we use the GPT2-BPE
4.2 Model Settings
Machine Translation. We compare our method with previous competitive systems: (1) Trans- former (Vaswami et al., 2017): first proposes a new
htttps://github.comm// //ttps: http://ggttuuu.ccmoeeeeeeeeee smt smt/mosesdecodrr htte///stterrrrrrpssssooneeeeete
hattps//ggittub.cmm/faaeoooo hwtp:///.ccocacccoecoooeeeeeeeee
encoder.py

---

Model
MCD1
MCD2
MCD3
Mean
Model
Alleviate K,
CTER
CTER
LSTM-attention
28.9
5.0
10.8
14.9
Transformer
34.9
8.2
10.6
17.9
Transfrmer
28.45i
Universal Transformer
37.4
18.9
COMPOSITION
(-5.8%)
(-7.8%) 52.0% (-10.9%)
Evolved Transformer
42.4
10.8
20.8
COMPOSITION
20.4% (-800%)
CGPS
13.2
1.6
6.6
7.1
NSEN
5.1
0.9
2.3
2.8
Table 3: CTERs (%) against alleviating E or K, on the
T5-1IB
61.4
30.1
31.2
40.9
CG-test set, where CTERinst and CTERAgEr denote instance-level and aggregate- CTER respectively. E and K, V denote encoder and keys, values entanglement problem respectively.
TB-mo
61.6
31.3
33.3
42.1
RoBERTa
60.6
33.6
43.4
HPD
72.0
66.1
63,9
67.3
Dangle
78.3
59.5
60.4
66.1
RoBERTa+CReg
74.8
53.3
58.3
62.1
COMPOSITION
72.8
53.2
52.2
59.4
tion: introduces attention mechanism (Bahdanau et al., 2015) in LSTM (Hochreiter and Schmid- huber, 1997); (2) Transformer (Vaswani al., 2017); (3) Universal Transformer (Dehghani al., 2019): combines the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of RNNs; (4) Evolved Transformer (So et al., 2019): uses wide depth-wise separable convolu- tions in the early layens of both the encoder and decoder; (5) CGPS (Li et al., 2019): leverages prior knowledge of compositionality with two rep- resentations, and adds entropy regularization to the encodings; (6) NSEN (Freivalds et al., 2019): is derived from the Shuffle-xchange network; (7) T5-1IB (Raffel et al., 2020): treats every natural language processing task as text-to-text problem, and is therefore suitable for the semantic parsing tasks. T5-11B 1B is a T5 model with 11B parameters finetuned CFQ; (8) T5- 1111BBoood (Furrer et al., 2020): shows that using masked language model (MLM) pre-training together with an intermediate representation leads to significant improvements in performance; (9) ROBERTa (Liu et al., 2019); makes use of the RoBERTa-base model as the en- coder and the randomly initialized Transformer decoder trained from scratch, where we use the same experimental settings of (Zheng and Lapata, 2022a); (10) HPD (Guo et al., 2020b): proposes novel hierarchical partially ordered set (poset) decoding paradigm, which consists of three compo- nents: sketch prediction, primitive prediction, and traversal path prediction; (11) Dangle (Zheng and Lapata, 2022a); (12) RoBERTa+CReg (Yin et al., 2023); (13) COMPOSITION: builds on (9) with our method.
Table Exact-match accuracy on different MCD splits of CFQ. Results are averaged over 3 random runs.
encoder-decoder architecture based solely on atten- tion mechanisms; (2) Transformer-Relaa only re- places sinusoidal (absolute) positional embedding with relative one; (3) Transformer-Small:: only de- creases the number of encoder layers and decoder layers 4, 4 respectively; (4) Transformer-Deep only increases the number of encoder layers to 8; (5) Bow (Raunak et al., 2019): uses hag-of-wordh pre-training to improve the representation of the en- coder upmost layer; (6) SeqMix (Guo et al., 2020a): synthesizes examples to encourage compositional behavior; (7) Dangle (Zheng and Lapata, 2022a): adaptively re-encodes (at each time step) the source input disentangle the representation of the coder upmost layer,$ (8) Proto-Transforin ( (Yin al., 2022): integrates prototypes of token rep- resentations over the training set into the source encoding to achieve the goal of categorization; (9) Transformer+CReg (Yin et al., 2023): promotes representation consistency across samples and pre- diction consistency for a single sample; (10) R- Danglesep (Zheng and Lapata, 2022b): disentan- gles their representations and only re-encode keys periodically, at some interval: (11) DLCL (Wang et al., 2019): proposes an approach based on dy- namic lincar combination of layers (DLCL), and is one of the very popular EnocderFusion work. Our method is built on top of (1), i.e., COM- POSITION, COMPOSITION-Rela COMPOSITION- Small and COMPOSITION-Deny We also provide reasons for experiments CoGnition without lan- guage models (see Appendix E).
Semantic Parsing. We compare our method with previous competitive systems: (1) LSTM atten-
4.3 Results on CoGnition
The main results on CoGnition are shown Ta- ble 1. We observe that: (1) COMPOSITION gives
*We use the same variant reported by Zheng and Lapata (2022a) (i.c., Dangle-FncDee Enceee (abs)) with sinusoidal (absolute) positional embedding.
20.4% % CTER Iust. and 52.0% CTER with a

---

Key of Decoder Layer
Value of Decoder Layer
Model
CTERmet
CTERA.c.
4
6- 0.27
0.20
0.19
0.18
0.15
8:12
6 0.00
0.12
0.13
0.14 6.15 0.14
Transformer
28.4%
62.9%
COMPOSITION-SA
22.2% (-6.2%)
53.8% (-9.19)
0.18
3.19
0.20
0:21
0.19
COMPOSITION-PE
22.6% (-5.8%)
55.6% (-7.3%)
Liooder Lave
COMPOSITION-SA & FF
20.4% (-8.0% )
52.0% (-10.9%)
0.07
0.10
0.10
0.09
0.08
0.12
0.10
0:13
0.13
0.13
0.14
0.18
0.19
0.20
0.22
0.23
0.20
0.09
0:13
0.14
0:14
0.14
Table 4: CTERs (%) against composing different source
0.20
0.21
9:20
0.19
0.39
information on the CG-test set.
1 0.00
0.08
0.08
0,08
0,08
0.11
0.29
0:20
0.29
0.19
0.38
0.35
POSITION substantially boosts the performance of RoBERTa (43.4 + 59.4), about 37 % relative im- provements, and is in fact superior to T5-11D BBB and T5-11B-mod. It also outperforms other baseline systems without pre-training except HPD. This re- sult demonstrates that pre-training as a solution to CG also has limitations, and also indicates that COMPOSITION is complementary to pre-trained models; (3) HPD performs better than Dangle, RoBERTa+CReg and COMPOSITION, achieving 67.3 exact match accuracy, which is highly opti- mized for the CFQ dataset. On the contrary, COM- POSITION, RoBERTa+CReg and Dangle are gener- ally applicable any seq2seq models for solving any seq2seq tasks including MT, as mentioned Section 4.3. However, compared with competi- tive performance on CoGnition, the improvements brought by COMPOSITION is relatively moderate, and even worse than Dangle. The underlying rea- son is related ity in natural language is much more complex than the rigid, arithmetic-like operations (Li et al., 2021; Zheng and Lapata, 2022a; Dankers et al., 2022). MT paradigmatically close to the tasks typically considered for testing compositionality in natural language, while our approach is more suitable for dealing with such scenarios.
Figure 4: Learned composition weights (after normal- ized) that each encoder layer (y-axis) attending to keys or values of different decoder layers (x-axis).
significant improvement of 8.0% and 10.9% ac- cordingly compared to the Transformer. More- over, COMPOSITION significantly outperforms most baseline models under the almost same param- eter settings, indicating composing the syntactic and semantic information of sequences dynami- cally for a particular task is more beneficial CG. Although Transformer+CRRg achieves slightly bet- ter performance and contains fewer parameters, is more complex and costly compared with COM- POSITION; (2) COMPOSITION, COMPOSITION- Rela, COMPOSITION-SMi
and COMPOSITION- Deep can deliver various performance improve- ments, demonstrating the general effectiveness of our method; (3) COMPOSITION-Deep performs better than Bow, Dangle and Proto-Transformer, indicating that focusing on alleviating the encoder entanglement problem only can achieve part of goals of CG as mentioned in Section 1. Compared to SeqMix, the improvement of COMPOSITION is more significant (2.3% vs 10.9% Agge) SeqMix utilizes linear interpolation in the input embedding space to reduce representation sparsity, and we suppose that the samples synthesized ran- domly may be unreasonable and harmful to model training: (4) can be seen that Transformer even slightly better than DLCL, indicating DLCL and COMPOSITION do not share the same motivation scenario.
recent finding that compositional-
5 Analysis
In this section, conduct in-depth analyses of COMPOSITION to provide a comprehensive under- standing of the individual contributions of each component. For experiments, we train CoM- POSITION (6-6 encoder and decoder layers) instead of other experimental settings on the CoGnition dataset, unless otherwise specified.
4.4 Results on CFQ
The main results on CFQ are presented in Table 2. We observe that: (1) RoBERTa comparable to T5- IIB, T5-11B-man B-dmod and outperforms other baseline systems without pre-training except HPD, indicat- ing that pre-training indeed benefits CFQ; (2) Com-
5.1 Effects of Specific Keys and Values of Different Decoder Layers
As mentioned in Section and 3.2, we hypothe- size that keys, values entanglement problem ex-
"We implement our approach based on Transformer-Deas for fair comparison with Transsorafrrrrrr...

---

CONUSITTINN
Transformer COMPSITION
Instance- level CTER
25
340 092
18
(0) Compound Length
(by
Length
14.4
Figure 5: InER of COMPOSITION and Transformer over the different compound and context lengths.
wo/ MOD
MOD
ists.? 7 It clear that our hypothesized keys, values entanglement problem an extension to encoder entanglement problem. We show curiosity about whether this problem exists, and COMPOSITION can alleviate it. In this experiment, we investi- gate its influence CoGnition. As shown in Ta- ble 3, observe certain improvements (-5.8% and -8.0% CTERinst -7.8% and -10.9% CTERA Agge) when separately alleviating the encoder or keys, values entanglement problem. 8 suggests that our method can alleviate both problems separately, and learning to compose information of different encoder layers dynamically can improve CG per- formance. Furthermore, the improvement brought from alleviating keys, values entanglement prob- lem is more significant than that brought from alle- viating encoder entanglement problem (52.0% vs 55.1% 55.1% 1% CTER Aggr), demonstrating the reasonable- ness of keys, values entanglement problem.
Figure CTER on compounds w/o and w/ MOD.
values of different decoder layers utilize more than just the information of the encoder topmost layer. More importantly, it also emphasizes our method provides an effective composition of syntactic and semantic information, i.c., specific combination for a particular task. To further demonstrate it, we also provide toy experiment in Appendix C.
5.2 Effects of Composing Information of Encoder Layers or Sub-layers
As mentioned in Section 3, the Transformer en- coder layer consists of two sub-layers. We assume that sub-layers may contain language information in different aspects, which may produce better gen- eralization results. Therefore, we curious about whether composing different encoder layers' sub-layers' information more beneficial to CG. In this experiment, investigate its influence on CoGnition. Specifically, we train COMPOSITION to compose representations outputted by either Eq. 5 or or a combination of both dynamically. Results are presented in Table 4. We observe cer- tain improvements (-6.2% and -5.8% % ITER when separately composing SA- and FF-level rep- resentations, where SA and FF denote representa- tions outputted by Eq. and 6 respectively. Further- more, the combination of both them brings further improvement (-8.0% CTERmsst). which illustrates that the information different encoder sub-layers is complementary and has cumulative gains. It also suggests that syntactic and semantic information brought by SA or FF is similar, but slightly differ- ent (Li et al., 2020), and can improve generalization performance respectively. It can be seen that the results of COMPOSITION-SA and COMPOSITION- FF presented Table are basically the same, and the improvements brought by the combination both them is relatively moderate.
To further illustrate the reasonableness of keys, values entanglement problem and understand how COMPOSITION alleviates it, we visualize the learned composition weights of COMPOSITION after normalized." Specifically, we train COMPO- SITION on CoGnition and then extract W W (see Section 3.2.2) to visualize them. Ideally, each key or value of different decoder layers should pay different attention weights to different encoder lay- ers' information. As shown in Figure 4, the learned composition weights (after normalized) are mutu- ally distinct for keys and values of different decoder layers, which implies COMPOSITION learns differ- ent dynamic composition modes for keys and val- ues every decoder layer respectively. In addition, it also indicates reasonableness of keys, values entanglement problem we proposed, since keys and
"I noteworthy that the representation of the encoder upmost layer serves the same key and value passing into every decoder layer in the Transformer.
"We use one or learnable vectors to generate one or 2.V representations passing into N decoder layers.
WWe only use representations outputted by Eq. for brevity.

---

Source
Transformer
COMPOSITION
The waiter he liked wore cach other's clothes.
T of B Ai
I M ll Di of DE &
(He liked to wear each other's clothes.)
(The waiter he liked wore cach other's clothes.)
The waiter he liked came by and chased the bully off.
Ii FR J HL 1 7
f 43300) HER 1 TE
(The waiter came by and chased the bully off.)
(The waiter he liked came by and chased the bully off.)
The waiter he liked picked up his mail.
3R ol in de Ah
fill Hi xtnn 95 6t # J tt AS
(The waiter liked pick up his mail.)
(The waiter be liked picked up his mail.)
Table 5:
Example translations of Transformer vs COMPOSITION. The bold characters denote the novel compounds and corresponding translations.
5.3 Effects on Compositional Generalization
composing the syntactic and semantic representa-
tions of sequences dynamically.
Compound Length and Context Length. Longer compounds have more complex semantic infor- mation and longer contexts are harder to compre- hend, making them more difficult to generalize (Li et al., 2021). We classify the test samples by com- pound length and context length, and calculate the I In Figure 5, we can observe that COM- POSITION generalizes better as the compound and context grows longer compared to Transformer. In particular, COMPOSITION gives a lower CTER by 11.0% over samples when the context length is more longer than 13 tokens. It suggests that approach can better captures the compositional structure of human language.
Conclusion
In this paper, we examine CG from new perspec- tive, i.e., utilizing different encoder layers' infor- mation. Specifically, we propose an extension to seq2seq models which composes different encoder layers' representations dynamically to generate spe- cific keys and values passing into different decoder layers. Experiments on CoGnition and CFQ have shown the effectiveness of our proposal on CG without any dataset or task-specific modification. To our knowledge, we are the first to point out a new representation entanglement problem and in- vestigate how to utilize information of different encoder layers benefits CG, achieving promising results on two realistic benchmarks. We hope the work and perspective presented in this paper inspire future related work on CG.
Complex Modifier. The postpositive modifier atom (MOD) is used to enrich the information of its preceding word (c.g., he liked in the phrase lost the dog he liked), which challenging to translate due to word reordering from English to Chinese. We divide the test samples into two groups according to compounds with (w/) or without (wo/) MOD. In Figure 6, we observe that the advantage of COMPO- SITION grows larger in translating the compounds with MOD, demonstrating its superiority in pro- cessing complex semantic composition.
Limitations
There are two limitations of our approach. Firstly. compared with competitive performance on CoGni- tion, the improvements brought by COMPOSITION on CFQ is relatively moderate, and even worse than some competitive methods. Hence, COMPOSITION is more suitable for tasks typically considered for testing compositionality in natural language. We strongly recommend researchers pay more atten- tion to tasks evaluating compositionality on natural language. Meanwhile, we regard that designing a more general method that can improve general- ization performance in both synthetic and natural scenarios is promising direction to explore in the future. Secondly, our method is mostly ap- plicable to any seq2seq models which adopt an encoder-decoder architecture instead of encoder- only or decoder-only architecture. However, the methodology of the proposed COMPOSITION is still rather general to any seq2seq models which
Case Study. We present 3 source examples con- taining novel compound the waiter he liked with MOD and 4 atoms, and their translations in Ta- ble 5. For all samples, correct translations denote that the novel compounds are translated correctly, COMPOSITION correctly translates the novel com- pounds across different contexts for all samples, while Transformer suffers from omitting different atoms. For example, the translation of the waiter is omitted in the first example, he liked is omitted in the second example and he omitted in the third example. Our results not only contain the correct compound translations but also achieve better trans- lation quality, while Transformer makes errors on unseen compositions, confirming the necessity of

---

adopt any architecture, since we can use the ran- domly initialized encoder or decoder to constitute the encoder-decoder architecture.
translation models with transparent attention. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 3028- 3033, Brussels, Belgium. Association for Computa- tional Linguistics.
Acknowledgement
Francesco Cazzaro, Davide Locatelli, Ariadna Quattoni, and Xavier Carreras. 2023. Translate first reorder later: Leveraging monotonicity in semantic parsing, In Findings of the Association for Computational Lin- guistics: EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pages 227-238. Association for Computational Linguisties.
We thank all the anonymous reviewers for their insightful and valuable comments. This work is supported by National key R&D Program of China (Grant no. 2000222200111101), the Key Sup- port Project of NSFC-Liaoning Joint Foundation (Grant no. U1908216), and the Project of Research and Development for Neural Machine Translation Models between Cantonese and Mandarin (No. WT135-76).
Rahma Chaabouni, Roberto Dessi, and Eugene Kharitonov. 2021. Can transformers jump around right in natural language? assessing performance transfer from SCAN. In Proceedings of the Fourth BlackboxNLP Workshop Analyzing and Interpret- ing Neural Networks for NLP, pages 136-148. Asso- ciation for Computational Linguistics.
References
Lasha Abzianidze, Johannes Bjerva, Kilian Evang, Hes- sel Haagsma, Rik van Noord, Pierre Ludmann, Duc- Duy Nguyen, and Johan Bos. 2017. The parallel meaning bank: Towards multilingual corpus of translations annotated with compositional meaning representations. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valen- cia, Spain, April 3-7, 2017, Volume 2: Short Papers, pages 242-247. Association for Computational Lin- guistics.
Yong Cheng, Lu Jiang. Wolfgang Macherey, and Ja- cob Eisenstein. 2020. AdvAug: Robust adversarial augmentation for neural machine translation. In Pro- ceedings of the 58th Annual Meeting of the Associa- rion for Computational Linguistics. Association for Computational Linguistics.
Henry Conklin, Bailin Wang, Kenny Smith, and Ivan Titov. 2021. Meta-learning to compositionally gen- eralize. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing, ACL/UCCLP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pages 3322-3335. Association for Computational Linguis- tics.
Ekin Akyürek, Afra Feyza Akyürek, and Jacob An- dreas. 2021. Learning to recombine and resample data for compositional generalization. In 9th Inter- national Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. Openkeview.net.
Verna Dankers, Elia Bruni, and Dieuwke Hupkes. 2022. The paradox of the compositionality of natural lan- guage: A neural machine translation case study. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. Association for Computational Linguistics.
Ekin Akyurek and Jacob Andreas. 2021. Lexicon learn- ing for few shot sequence modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computa- tional Linguistics.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2019, Univer- sal transformers. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Jacob Andreas. 2020. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7556-7566, Online. Association for Computational Linguistics.
Tobias Domhan. 2018 How much attention do you need? a granular analysis of neural machine transla- tion architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1799-1808.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015. San Diego CA, USA, May 7-9, 2015, Conference Track Proceddigg.
Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Pupers). Asso- ciation for Computational Linguistics.
Ankur Bapna, Mia Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. 2018. Training deeper neural machine

---

Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Shuming Shi, and Tong Zhang. 2018. Exploiting deep representations for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4253-4262, Brussels, Belgium. Association for Computational Linguistics.
the Association for Computational Linguistics and the 1 11th International Joins Conference on Natural Longuage Processing, ACI/J/CNLP 2021, (Volume I: Long Popers), Virtual Event, August 1-6, 2021, pages 908-99 9211.21. Association for Computational Linguistics.
Jonathan Herzig, Peter Shaw, Ming- Wei Chang, Kelvin Guu, Panupong Pasupat, and Yuan Zhang. 2021. Un- locking compositional generalization in pre-trained models using intermediate representations. arXiv.
Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Longyue Wang. Shuming Shi, and Tong Zhang. 2019. Dynamic layer aggregation for neural machine translation with routing-by-age aggremeentt In The Thirty-Thha Third AAAI Conference on Artificial Intelligence, AAAT 2019, The Thirty-First Innovative Applications of Artificial In- telligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, pages 86 86-93.44 AAAI Press.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput.
Yichen Jiang and Mohit Bansal. 2021. Inducing trans- former's compositional generalization ability via aux- iliary sequence prediction tasks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Vir- tual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pages 6253-6265. Association for Computational Linguistics.
Jerry A Fodor and Zenon W Pylyshyn. 1988, Connec- tionism and cognitive architecture: A critical analysis. Cognition.
Markus Freitag and Yaser Al-Onaizan. 2017. Beam
Damiel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang. Marc van Zee, and Olivier Bousquet. 2020. Measur- ing compositional generalization: A comprehensive method on realistic data. In 8th International Confer- ence on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
search strategies for neural machine translation. In Proceedings of the First Workshop on Neural Ma- chine Translation, Association for Computational Linguistics.
Karlis Freivalds, Emils Ozolins, and Agris Sostaks. 2019. Neural shuffle-exchange networks- sequence processing in o(n log n) time. In Advances in Neural Information Processing Systems 32: Anmual Confer- ence on Neural Information Processing Systems 2019, NeurlPS 2019, December& 8- 14, 2019, Vancouver. BC, Canada
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Daniel Furrer, Mare van Zec Nathan Scales, and Nathanael Schärli. 2020. Compositional generaliza- tion in semantic parsing: Pre-training vs. specialized architectures. arXiv preprint arXiv:2007.08970
Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of requence-to-sequenee recurrent networks. In Inter- national conference on machine leurning. PMLR.
Demi Guo, Yoon Kim, and Alexander Rush. 2020a. Sequence-level mixed sample data augmentation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
Brenden M. Lake. 2019. Compositional generalization through meta sequence- soreenccett-scuuenee learning. In Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pages
Yinuo Guo, Zeqi Lin, Jian-Guang Lou, and Dongmei Zhang. 2020b. Hierarchical poset decoding for com- positional generalization in language. Advances in Neural Information Processing Systems. Curran Associates, Inc.
Brenden M Lake, Tomer D Ullman, Joshua B Tenen- baum, and Samuel J Gershman. 2017. Building ma- chines that learn and think like people. Behavioral and brain sciences.
Tianyu He, Xu Tan, Yingee Xia, Di He, Tao Qin, Zhibo Chen, and Tie- Yan Liu. 2018 Layer-wise coordina- tion between encoder and decoder for neural machine translation. In Advances in Neural Information Pro- cessing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 7955- 7965.
Bei Li, Ziyang Wang. Hui Liu, Yufan Jiang, Quan Du, Tong Xiao, Huizhen Wang, and Jingbo Zhu. 2020. Shallow-to-dee training for neural machine transla- tion. arXiv preprint arX1v:2010.03337.
Qing Li, Yixin Zhu, Yitao Liang Ying Nian Wu, Song-Chun Zhu, and Siyuan Huang. 2022. Neural- symbolic recursive machine for systematic general- ization. CoRR, Abs/2210.0160/
Jonathan Herzig and Jonathan Berant. 2021. Span- based semantic parsing for compositional general- ization, In Proceedings of the 59th Annual Meeting

---

Yafu Li, Yongjing Yin. Yulong Chen, and Yue Zhang. 2021. On compositional generalization of neural ma- chine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1 1ith International Joint Conference on Natural Language Processing ( (Volume I: Long Papers). Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 19X.vv1997.1169300
Sarthak Mittal, Sharath Chandra Raparthy, Irina Rish, Yoshua Bengio, and Guillaume Lajoie. 2022. Compo- sitional attention: Disentangling search and retrieval. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25- 29, 2022.
Yuanpeng Li, Liang Zhao, Jianyu Wang, and Joel Hest- ness. 2019. Compositional generalization for primi- tive substitutions. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics.
Maxwell I. Nye, Armando
Josh Tenen- baum, and Brenden M. Lake. 2020. Learning com- positional rules via neural program synthesis. In Ad- vances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Zhaoyi Li, Ying Wei, and Defu Lian. 2023. Learning to substitute spans towards improving compositional generalization. arXiv preprint arXiv:2306.00 230.0000000 02840.
Lei Lin, Shuangtao Li, and Xiaodong Shi. 2023. Leapt: Learning adaptive prefix- to- prefix translation for si- multaneous machine translation. In ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling, In Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tion for Computational Linguistics (Demonstrations) Association for Computational Linguisties.
Chenyao Liu, Shengnan An, Zeqi Lin, Qian Liu, Bei Chen, Jian-Guang Lou, Lijie Wen, Nanning Zheng, and Dongmei Zhang. 2021. Learning algebraic re- combination for compositional generalization. In Findings of the Association for Computational Lin- guisties: ACL/JCNLP 2021, Online Event, August 1-6, 2021, Findings of ACL. Association for Compu- tational Linguisties.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Lànguistics, July 6- 6-12, 2002, Philadelphia. PA, USA. ACL.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, WMT 2018, Belgium, Brussels, October 31 November 1, 2018. Association for Computational Linguistics.
Fenglin Liu, Xuancheng Ren, Guangxiang Zhao, Xu Sun, and Liangyou Li. 2020a. Layer-wise cross- view decoding for sequence-to-sequene learning. ArXiv preprint, abs/2005.0808/
Qian Liu, Shengnan An, Jian-Guang Lou, Bei Chen, Zeqi Lin, Yan Gao, Bin Zhou, Namning Zheng, and Dongmei Zhang. 2020b. Compositional generaliza- tion by learning analytical expressions. In Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020, Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res.
Alessandro Raganato, Jörg Tiedemann, et al. 2018. An analysis of encoder representations in transformer- based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP. Analyzing and Interpreting Neural Networks for NLP. The As- sociation for Computational Linguistics.
Qian Liu, Shengnan An, Jian- Mtan-Guang Lou, Bei Chen, Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng, and Dongmmi Zhang. 2020c. Compositional generaliza- tion by learning analytical expressions. In Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 12, 2020, virtual.
Vikas Raunak. Vaibhav Kumar, and Florian Metze. 2019. On compositionality in neural machine translation. ArXiv preprint.
Laura Ruis and Brenden M. Lake. 2022. Improving sys- tematic generalization through modularity and aug- mentation. CoRR, abs/2202.10745.
Shan Liu, Yafang Zheng, Lei Lin, Yidong Chen, and Xiaodong Shi. 2023. A novel pos-guided data aug- mentation method for sign language gloss translation. In CCF International Conference on Natural Lan- guage Processing and Chinese Computing, pages 993-003. Springer.
Jake Russin, Jason Jo, Randall C O''Reii O'Reilly, and Yoshua Bengio. 2019. Compositional generalization in a deep seq2seq model by separating syntax and seman- tics. arXiv preprint arXiv: 1eXiv:1904.09777 1904.09708.

---

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers). Association for Computational Linguistics.
Association for Computational Linguistics: Human Language Technologies: Industry Papers. Associa- tion for Computational Linguisties.
Jingfeng Yang, Le Zhang, and Diyi Yang. 2022. SUBS:
subtree substitution for compositional semantic pars- ing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technolo- gies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022. Association for Computational Linguis- tics.
Yanyao Shen, Xu Tan, Di He, Tao Qin, and Yan Liu. 2018. Dense information flow for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 1294-1303, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Yuekun Yao and Alexander Koller. 2022. Structural gen- models. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022. Association for Computational Linguis- tics.
eralization is hard for
David R. So, Quoe V. Le, and Chen Liang. 2019. The evolved transformer. In Proceedings of the 36th In- ternational Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Califormia, USA.
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems.
Yongjing Yin, Yafu Li, Fandong Meng. Jie Zhou, and Yue Zhang. 2022. Categorizing semantic representa- tions for neural machine translation. In Proceedings of the 29th International Conference on Computa- tional Linguistics, pages 5227-5239. International Committee on Computational Linguistics.
Tristan Thrush. 2020. Compositional neural machine translation by removing the lexicon from syntax. arXiv preprint arXiv:2002.08899
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. Advances in neural information processinnnpoossin systems.
Yongjing Yin. Jiali Zeng. Yafu Li, Fandong Meng, Jie Zhou, and Yue Zhang, 2023. Consistency regular- ization training for compositional generalization. In Proceedings of the s Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), pages 1294-1308.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019. The bottom-up evolution of representations in the trans- former: A study with machine translation and lan- guage modeling objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNILP), pages 4396-4406. Association for Computational I Linguistics.
Daniel Zeman, Martin Popel, Milan Straka, Jan Hajic, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo. Slav Petrov, Martin Potthast, and 2017. Conll 2017 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of the CONLL 2017 Shared Task: Mulsilingual Parsing from Raw Text to Universal Dependencies, Vancouver, Canada, August 3-4, 2017, pages 1-19. Association for Computational Linguistics.
Qiang Wang, Bei Li, Tong Xiao, Jingho Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 18 1810-1822. Florence, Italy. Association for Computational Linguistics.
Hao Zheng and Mirella Lapata. 2021. Compositional generalization via semantic tagging. In Findings of the Association for Computational Linguisties: EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 16-20 November, 2021, pages 1022- 1032. Association for Computational Linguisties.
Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yinqiao Li. and Jingbo Zhu. 2018. Multi-layer representation fusion for neural machine translation. In Proceed- ings of the 27th International Conference on Com- putational Lingnistics, pages 3015-3026, Santa Fe. New Mexico, USA. Association for Computational Linguistics.
Hao Zheng and Mirella Lapata. 2022a. Disentangled se- quence to sequence learning for compositional gener- alization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). Association for Computational Linguistics.
Weiwen Xu, Ai Ti Aw, Yang Ding, Kui Wu, and Shafiq Joty. 2021. Addressing the vulnerability of NMT input perturbations. In Proceedings of the 2021 Conference of the North American Chapter of the
Hao Zheng and Mirella Lapata. 2022b. Real- world compositional generalization with disentan- gled sequence-to- -eeeu-eece learning. arXiv preprint arXiv:2212.05982.

---

EnEi EnZh
EnEt EnZh
88
Precision (96)
Precision (%)
833
81
Encoder Layers
Encoder Layers
Figure 7: Precision (%) against different encoder layers' representations as input the test set of POS tagging task.
Figure 8: Precision (%) against different encoder layers' representations as input the test set of Semantic tag- ging task.
Yafang Zheng. Lei Lin, Zhaohong Lai, Binling Wang, Shan Liu, Biao Fu. Wenhao Rao, Peigen Ye. Yidong Chen, and Xiaodong Shi. 2023. Layer-wise represen- tation fusion for compositional generalization. arXiv preprint 2VXVv23307.10799
low-resource)" by following the same settings of Raganato et al. (2018)." After training the MT models, we freeze the encoder parameters, and only train decoder layer for each task, since we pect the decoder should have overly significant impact on the model's s performance of different tasks. We then analyze the amount of syntactic and semantic information in different encoder layers via evaluating the different encoder layers' perfor- mance of corresponding task. We use the Universal Dependencies English Web Treebank v2.0 (Zeman et al., 2017) for POS tagging (syntactic task) and the annotated data from the Parallel Meaning Bank (PMB) (Abzianidze et al., 2017) for Semantic tag- ging (semantic task). 13 use precision to evalu- ate model performance.
Denny Zhou, Nathanael Schärli, Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans. Claire Cui. Olivier Bousquet, Quae V Le, and Ed H. Chi. 2023. Least-to-mos most prompting enables com- plex reasoning in large language models. In The Eleventh International Conference on Learning Rep- resentations.
A Preliminary Analysis
In this section, we analyze the amount of syntac- tic and semantic information captured by different encoder layers in the Transformer under MT sce- narios. We aim at analyzing the representations learned by different encoder layers of different models through probing the encoder as input repre- sentation for various prediction tasks. We measure the importance of input features for various tasks by evaluating the ability of the decoder. Specif- ically, use fixed encoder representation as input and two different tasks, i.e., -fft-of-Speech (POS) tagging, and Semantic tagging. to evaluate the syntactic and semantic information contained in different encoder layers respectively. The reason is that we assume if the input representation effec- tively captures a property (syntactic or semantic information), then the decoder can casily predict that property.
Results on POS tagging and Semantic tagging are presented in Figure 7 and 8 respectively. We observe that:
For EnEt and EnZh, the performance tends to decrease as the number of layers increase.
For EnEt and EnZh, the performance tends to increase as the number of layers increase.
Therefore, we can conclude that the bottom lay- ers of the Transformer encoder contain more
"We use two datasets with different sizes to analyze in- formation captured by different encoder layers across models with different translation quality and target language.
"The provided datasets freely available https:// statmt. g/wnt1 1/tt8// 18/t a/ttranslation task. html.
To explore the precise effects of information captured by different encoder layers, we train the Transformer on the WMT18 English + Chinese (EnZh, rich-resource), English + Estonian (EnEt,
lWe follow same settings of Raganato (2018) and adopt one attention
and one feed-forward sub-layer to consitute the decoder layer.
We follow the same data preprocess process of Zeman et al. (2017) and Abzianidze et al. (2017).

---

Model
CTERmstt Kintt +
CTERARE
Model
CTER
CTER
Transformer
28.4%
62.9%
Transformer
28.4%
62.9%
Transformer-accu
failed
failed
COMPOSITION-HAN
27.0% (-1.4%)
61.3% (-1.6%)
COMPOSITION
20.4% (-8.0%)
52.0% (-10.9%)
COMPOSITION
20.4% (-8.0%)
52.0% (-10.9%)
Table CTERs (%) against Transformer-accu COM- POSITION on the CG-test set.
Table 7: CTERs (%) against Transformer, COMPOSI- TION and COMPOSITION-Hal the CG-test
syntactic information and the top ones contain more semantic information, and the information
Therefore, we conduct a toy experiment on CoG- nition. Specifically, all encoder sub-layers' infor- encoded by each encoder layer transforms from syntactic to semantic as the number of layers in- crease.
mation is accumulated to serve the same key and value passing into every decoder layer (called Transformer-aceu), rather than composing them dynamically like we do. Results are listed Ta- ble 6. Transformer accu even fails to train. It suggests that even if the syntactic and semantic information of sequences is considered, the inap- propriate combinations will instead bring noise to significantly affect the model's CG performance.
B Experimental Settings
For CoGnition, we set hidden size to 512 and feed- forward dimension to 1,024. The number of en- coder and decoder layers are 6, 6 and the number of attention heads are 4. The model parameters are optimized by Adam (Kingma and Ba, 2015), with
D Effects of Representations from Low-layer Encoder
31 0.9, 32 0.98. The learning rate is set to
5e-4 and the number of warm-steps is 4000. We set max tokens as 199 tokens for iteration. We use one GeForce GTX 2080Ti for training with 100,000 steps and decoding. We report the average performance over 6 random seeds provided in Li et al. (2021). We train all COMPOSITION models from scratch. For CFQ, we use the base RoBERTa with 12 encoder layers, which is combined with Transformer decoder that has decoder layers with hidden size 256 and feed-forward dimension 512. We use separate target vocabulary. The number attention heads are 8. The model parameters are optimized by Adam (Kingma and Ba, 2015), with 31 = 0.9, f2 0.98. The learning rate is set to le-4 and the number of warm-steps is 4000. We set max tokens as 4,096 tokens for iteration. We use one GeForee GTX 2080тi for training with 45,000 steps and decoding. We report the average perfor- mance over random seeds provided in Zheng and Lapata (2022a). We train COMPOSITION built on top of RoBERTa with full parameter fine-tuning.
To verify the low-layer encoder representations are also essential to our approach, we only evaluate approach CoGnition with the collected en- coder representations of the top three layers. Re- sults are presented in Table We can observe that only composing the representations of the top three encoder layers leads to a sharp drop in per- formance (27.0% vs 20.4% CTER but still outperforms the Transformer baseline (27.0% vs 28 4% CTER It further demonstrates the dis- tinct difference between method and the find- ings introduced by previous studies EncoderFu- sion. It also reflects our starting point is correct, i.c.. exploring how to compose syntactic and semantic information. can seen that COMPOSITION'S performance is dramatically reduced given only se- mantic information (the last three encoder layers' information).
Reasons for Experiments on CoGnition without Language Models
C Effects of the Effective Composition
We do not conduct experiments on CoGnition with language models for two reasons. First, CoG- nition is constructed to test CG performance in MT scenarios with simple sentence pairs (see Fig- ure 3), however, language models are trained
As mentioned in Section 3, we introduce the com- posed layer between the encoder and decoder to compose different encoder sub-layers' information dynamically to generate specific keys and values passing into different decoder layers. We show curiosity about whether the composed layer can fuse all encoder sub-layers' information effectively.
[Moo.... M1 are the output of the encoder layers 0 ~ L. The input of keys and values of decoder layer i is z; go + M--1, where 0 < L.

---

vast amounts of multilingual sentences bilingual sentence pairs. is contrary to the compositional generalization task itself, since we can not guar- antee that every sentence in the test set a novel combination from known components for language models. Second, it is unfair to compare large lan- guage models with systems without pre-training. We strongly recommend researchers pay more at- tention conduct experiments CoGniton with- out language models.


Reviews:
[{"review_id": "pHFVRSjv2a", "reviewer": "Reviewer_DCYU", "summary": "", "questions": "A. Do the authors have results for the in distribution splits? Does adding the composed layer hurt the IID performance? \n\nB. Figure 5, is there similar visualization for the standard Transformer for comparison? \n\nC. Do the authors have mean and variance of several runs? for at least main experiments\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reply to Questions and Typos Grammar Style And Presentation Improvements\nThanks for your constructive comments and suggestions, and they are exceedingly helpful to improve our paper. We will incorporate them in the revised paper. In the following, your comments are first stated and then followed by our point-by-point responses.\n> Q1: Do the authors have results for the in distribution splits? Does adding the composed layer hurt the IID performance? \n\n**A1**: Thank you very much for this insightful comment.  For CoGnition, we report BLEU on in-domain test set (ind-test). The detailed results are given below.\n\n|Models|ind-test $\\uparrow$|\n| --- | --- |\n|Transformer|56.4|\n|COMPOSITION|**56.7 (+0.3)**|\n\nResults are averaged over 6 random runs on CoGnition. It is clear that improvements on compositional generalization are not at the expense of in-domain performance (COMPOSITION obtains similar and even slightly better performance than the Transformer on the CoGniton in-domain test set).\n> Q2: Figure 5, is there similar visualization for the standard Transformer for comparison?\n\n**A2**: Thank you very much for this insightful comment. For standard Transformer, the representation of the encoder upmost layer serves as the same key and value passing into every decoder layer. The detailed visualization is shown below in tabular form, where each encoder layer (y-axis) attending to keys of different decoder layers (x-axis).\n\n||1|2|3|4|5|6|\n|---------|---------|---------|---------|---------|---------|---------|\n|**1**|0|0|0|0|0|0|\n|**2**|0|0|0|0|0|0|\n|**3**|0|0|0|0|0|0|\n|**4**|0|0|0|0|0|0|\n|**5**|0|0|0|0|0|0|\n|**6**|1|1|1|1|1|1|\n\nWe omit visualization for each encoder layer attending to values of different decoder layers for brevity, since it is the same as the above table.\n> Q3: Do the authors have mean and variance of several runs? for at least main experiments.\n\n**A3**: Thank you very much for this insightful comment. We have already mentioned in line 263-264 and provided more details in Appendix B. For CoGnition, we report the average performance over 6 random seeds provided in Li et al. (2021). For CFQ, we report the average performance over 3 random seeds provided in Zheng and Lapata (2022). We also provide experimental results for each random seed, the detailed results are as follows.\n\n|CoGnition (seed)|Instance-level CTER|Aggregate-level CTER|BLEU|\n|---------|---------|---------|---------|\n|1|20.2%|52.1%|61.7|\n|7|20.5%|53.2%|61.4|\n|77|19.7%|50.7%|61.9|\n|777|18.4%|49.9%|61.8|\n|7777|21.1%|52.5%|61.3|\n|77777|22.5%|53.4%|60.9|\n\n|CFQ (seed)|MCD1|MCD2|MCD3|Mean|\n|---------|---------|---------|---------|---------|\n|1| 74.7 | 53.3 | 52.2 |60.1|\n|7| 72.1 | 50.9 | 47.9 |57.0|\n|77| 71.7 | 55.3 | 56.4 |61.1| \n\nTherefore, for CoGnition, the mean values of instance-level CTER, aggregate-level CTER and BLEU are **20.4**, **52.0** and **61.5**, respectively, and their variances are **1.6**, **1.6** and **0.1**, respectively. For CFQ, the mean value and variance of exact-match accuracy on Mean MCD splits are **59.4** and **3.0**, respectively.\n\n> Q4: Figure 4 seems to be hard to interpret, it might help readers to understand the visualization better by normalizing the weights; L489-492 refers to qualitative examples in Table 6, which should be Table 5; I found Appendix A provides a nice motivation. The author could consider moving this section to the main text in the later version.\n\n**A4**: Thank you so much for your careful reading! We will follow your suggestions and revise our paper for better readability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper introduces a simple approach that is generally applicable to any seq2seq model. The method does not require a significant change in architecture and demonstrates strong performance for compositional generalization on two tasks, especially in machine translation. \n- The paper is well-motivated. The investigation of the syntactic and semantic entanglement problem is interesting. The related analysis is comprehensive.\n", "reasons_to_reject": "- The in-distribution performance is not reported. It is unclear whether the proposed method favors the specific compositional generalization setups studied in this paper, but may not be generally applicable.\n- The reported results are only one run. It would be better to include mean and variance from several runs, since most models are small size. \n", "questions_to_authors": "A. Do the authors have results for the in distribution splits? Does adding the composed layer hurt the IID performance? \n\nB. Figure 5, is there similar visualization for the standard Transformer for comparison? \n\nC. Do the authors have mean and variance of several runs? for at least main experiments\n", "paper_topic_and_main_contributions": "This paper proposes COMPOSITION, a method that learns to compose representations from different layers of encoder, as a way to mitigate the issue of entangled syntactic and semantic problem in seq2seq models. The proposed method introduces a composed layer to generate keys and values for each decoder layer. The paper demonstrates that this approach improves compositional generalization on machine translation (CoGnition) and semantic parsing (CFQ) tasks. Finally, the paper provides detailed ablation studies and qualitative analysis.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Figure 4 seems to be hard to interpret, it might help readers to understand the visualization better by normalizing the weights\n- L489-492 refers to qualitative examples in Table 6, which should be Table 5. \n- I found Appendix A provides a nice motivation. The author could consider moving this section to the main text in the later version. \n"}}, {"review_id": "6nm0onMHLq", "reviewer": "Reviewer_VRuK", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reply to Reasons To Reject and Typos Grammar Style And Presentation Improvements\nWe sincerely thank the reviewer for carefully reading our paper and pointing out our typos. We will carefully incorporate them in the revised paper. In the following, your comments are first stated and then followed by our point-by-point responses.\n> Q1: The novelty in terms of architecture is limited, despite being pitched as a major point of interest of the paper. As the authors acknowledge (lines 550 sq.), there is significant overlap with prior work. It might have been more straightforward to not present the composition layer as an entirely novel contribution, and instead devote some space to presenting prior work this approach builds upon in §2.1 (esp. Wang et al 2019) clearly delineate in §2.2 how the proposed approach differ from this clearly very related prior work.\n\n**A1**: Thank you very much for this insightful comment. We would like to address the novelty of this submission. There are three distinct differences between our method and EnocderFusion work. **Firstly**, our method exploits information from **all encoder sub-layers** and generates **specialized keys, values passing into different decoder layers** while they do not. **Secondly**, our method shows the effectiveness of utilizing **low-layer encoder representations** while they have the opposite view (see Appendix E). **Thirdly**, we do not share the same motivation or task. Their work focuses on how to transform information across layers in deep neural network scenarios. Our motivation is to learn to composing the syntactic and semantic representations of sequences appropriately for CG. Both our method and DLCL (Wang et.al., 2019) use learnabe vectors or scalars, however, the way we use them is completely different than they do. Importantly, we do not share the same motivation or task. Therefore, we would like to describe our method based on the findings conclued by our prelimary analysis (see Appendix A), not built upon in DLCL (Wang et.al., 2019). To further validate the above differences between our approach and EncoderFusion work, we reproduce DLCL (Wang et.al., 2019), one of the very popular EnocderFusion work, and conduct experiments on CoGnition dataset. The results are reported below.\n\n|Models|Instance-level CTER $\\downarrow$|Aggregate-level CTER $\\downarrow$|BLEU $\\uparrow$|\n|---------|---------|---------|---------|\n|Transformer|28.4%|62.9%|59.5|\n|DLCL|28.4% (+0.0%)|67.9% (+5.0%)|59.2 (-0.3)|\n|COMPOSITION|**20.4% (-8.0%)**|**52.0% (-10.9%)**|**61.5 (+2.0)**|\n\nResults are averaged over 6 random runs on CoGnition. It can be seen that Transformer is even slightly better than DLCL, indicating DLCL and COMPOSITION do not share the same motivation or scenario. DLCL focuses on how to transform information across layers in deep neural network scenarios, while COMPOSITION is to learn to composing the syntactic and semantic representations of sequences appropriately for CG.\n> Q2: On a more fundamental level, the authors posit that transformer encoder representations \"entangle\" semantic and syntax, and claim that their proposal allows to mitigate this problem (lines 103 sq., e.g.). I find support for this claim to be minimal: there is no clear experiment showing that the learned representations disentangle semantic and syntactic factors in any way. Supposedly, appendix D discusses this aspect but only presents an inconclusive supplementary experiment.\n\n**A2**: Thank you very much for this insightful comment. As mentioned, the representation entanglement problem is the syntactic and semantic representations of sequences are twisted inappropriately. Previous work concentrates on **separating the learning of syntax and semantics, and then compose them appropriately**. On the contrary, we have conducted preliminary analysis provided in Appendix A, and concluded that **the bottom layers of the Transformer encoder contain more syntactic information and the top ones contain more semantic information**. Inspired by this, **we collect representations outputted by each encoder layer instead of separating the learning of syntax and semantics**. So one intuitive solution to solve the source keys, values entanglement problems is to learn **different and specific combinations** of syntactic and semantic information (i.e., representations outputted by each encoder layer). Therefore, we propose a composed layer to compose them appropriately to generate specialized source keys and values passing into different decoder layers. As shown in Fig.5, in contrast to the Transformer, COMPOSITION obtains specialized keys and values representations passing into different decoder layers (i.e., **the dots representing keys and values of different decoder layers are in different places**). Notably, three circles represent randomly selected examples of CG-test set.\n\n> Q3: I'm systematically uncertain across all experiments as to which of the following three approaches the authors adopt: (i) fine-tuning a model with a limited number of added trainable parameters (ii) tuning only the newly-added learnable weight vectors, (iii) training a model from scratch.\n\n**A3**: Thank you very much for this insightful comment. For CoGniton, we train COMPOSITION from scratch. For CFQ, we train COMPOSITION built on top of RoBERTa with full parameter finetuing.\n\n> Q4: Some of the baselines are fundamentally unknown to me: I don't know what \"Transformer-rela\" is and no reference is provided for me to look it up. No reference to appendix , where this term is defined, had been provided.\n\n**A4**: Thank you very much for this insightful comment. We have already mentioned in line 304-305 and Appendix C. Transformer-Rela is only replaces sinusoidal (absolute) positional embedding with a relative one.\n\n> Q5: I suggest adopting standard \"matrix cookbook\" notations for all equations. In particular, $W$is generally reserved for weight matrices in ML literature, using this symbol to refer to a scalar is needlessly confusing.\n\n**A5**: Thank you so much for your careful reading! We will follow your suggestions and revise our paper for better readability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors report significant improvement on the synthetic test split of the MT dataset, and some improvement over their baseline encoder on the semantic parsing dataset.\n- The solution put forth is conceptual rather simple, and allows for some interesting manual analyses (in particular, fig 4, lines 371 to 401)", "reasons_to_reject": "**A/** The novelty in terms of architecture is limited, despite being pitched as a major point of interest of the paper. As the authors acknowledge (lines 550 sq.), there is significant overlap with prior work. It might have been more straightforward to not  present the composition layer as an entirely novel contribution, and instead devote some space to presenting prior work this approach builds upon  in §2.1 (esp. Wang et al 2019) clearly delineate in §2.2 how the proposed approach differ from this clearly very related prior work. \n\n**B/** On a more fundamental level, the authors posit that transformer encoder representations \"entangle\" semantic and syntax, and claim that their proposal allows to mitigate this problem (lines 103 sq., e.g.). I find support for this claim to be minimal: there is no clear experiment showing that the learned representations disentangle semantic and syntactic factors in any way. Supposedly, appendix D discusses this aspect but only presents an inconclusive supplementary experiment. \n\n~**C/** Some methodological details are unclear, deferred without notice to the appendix or absent. E.g., :~\n - ~I'm systematically uncertain across all experiments as to which of the following three approaches the authors  adopt: (i) fine-tuning a model with a limited number of added trainable parameters (ii) tuning only the newly-added learnable weight vectors, (iii) training a model from scratch~\n - ~Some of the baselines are fundamentally unknown to me: I don't know what \"Transformer-rela\" is and no reference is provided for me to look it up. No reference to appendix , where this term is defined, had been provided.~\n\n**Edit after rebuttal:** some clarifications made by he authors greatly alleviated my concerns as per this last point.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a modification of the Transformer architecture, so as to allow keys and queries in decoder source attention sublayers to be redefined as weighted sums of encoder sublayers. \nTo motivate this modification, the authors frame their experiments with respect to the task of compositional generalization. They demonstrate the effectiveness of the proposed approach on two datasets (an MT dataset and a semantic parsing dataset).", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I suggest adopting standard \"matrix cookbook\" notations for all equations. In particular, $W$ is generally reserved for weight matrices in ML literature, using this symbol to refer to a scalar is needlessly confusing."}}, {"review_id": "0kiejdhIDk", "reviewer": "Reviewer_5kLD", "summary": "", "questions": "1. what is performance of other LLM look like on the two datasets?\n2. what if the H^{SA} and H^{FF} have different dimensions (which is often the case in some LLMs)? Equation (7) and (8) are not applicable in this case.\n3. what if we apply the listed encoder layer fusion methods on this task, what would be the performance like?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reply to Questions and Reasons To Reject\nThanks for your encouraging words and constructive comments. We sincerely appreciate your time in reading the paper, and our point-to-point responses to your comments are given below.\n> Q1: The novelty of the proposed approach is limited. The difference between the proposed method and related encoder layer fusion approaches is marginal.\n\n**A1**: Thank you very much for this insightful comment. We would like to address the novelty of this submission. There are three distinct differences between our method and EncoderFusion work. **Firstly**, our method exploits information from **all encoder sub-layers** and generates **specialized keys, values passing into different decoder layers** while they do not. **Secondly**, our method shows the effectiveness of **utilizing low-layer encoder representations** while they have the opposite view (see Appendix E). **Thirdly**, we do not share the same motivation or task. Their work focuses on how to transform information across layers in deep neural network scenarios. Our motivation is to learn to composing the syntactic and semantic representations of sequences appropriately for CG. We also provide the experimental results of DLCL (wang et.al., 2019) on the CoGnition dataset to further validate our above differences. Please refer to **A5** for the details.\n> Q2: The compared models are pretty small. Many existing pretrained LLM are not included for comparison. what is performance of other LLM look like on the two datasets?\n\n**A2**: Thank you very much for this insightful comment. We didn’t conduct experiments to evaluate different LLMs’ CG performance on CoGnition for two reasons. **First**, it’s hard to evaluate a broad range of LLMs’ CG performance on CoGnition due to time and resources limits. **Second**, it’s contrary to the compositional generalization task itself, and unfair to compare LLMs to systems without pre-training. Let’s back to the definition of compositional generalization task. Compositional generalization is the algebraic ability to understand and produce a potentially infinite number of **novel combinations** from **known components**, while LLM is a large language model, which are able to access **vast amounts of text data**. If we conduct experiments on LLMs to test their CG performance, one natural question can be raised: **how can we guarantee that every sentence in the test set is a novel combination from known components for LLMs**? \\\nHowever, to validate LLMs' CG performance, we follow your suggestions and test GPT3.5 (also known as ChatGPT). We only choose GPT-3.5 (an effective and generic approach to address a wide range of NLP tasks) due to time and resource limits, and test its compositional generalization performance in the zero-shot manner on the CoGniton dataset. We didn't conduct experiments on CFQ dataset, because it’s hard to design the prompt as the input of GPT3.5 to generate results for CFQ. Specifically, we take *translate from English to Chinese: source sentence* as the input of GPT3.5 to generate the target sentence using greedy decoding. Finally, we obtain 10,554 generated examples, and 246 examples are discarded as a result of the error (i.e., The response was filtered due to the prompt triggering Azure OpenAI’s content management policy). The full experiment results are presented below. \n\n|Models|Instance-level CTER $\\downarrow$|Aggregate-level CTER $\\downarrow$|\n|---------|---------|---------|\n|Transformer|28.4%|62.9%|\n|GPT-3.5|28.1% (-0.3%)|71.9% (+9.0%)|\n|COMPOSITION|**20.4% (-8.0%)**|**52.0% (+10.9%)**| \n\nIt can be seen that Transformer is even slightly better than GPT3.5, indicating GPT3.5 still fails to improve the CG performance. Meanwhile, there have some works investigating whether scaling up model size also improve compositional generalization (Qiu et.al., 2022; Hosseini et.al., 2022; Kim et.al., 2022). They found performance for the largest model size is generally worse than finetuning performance for much smaller models.\n> Q3: Some of the claims are not well justified, making the findings less solid. For example, line 141-143: \"our proposed method is mostly applicable to any Seq2Seq models\". The claim is not justified at all. what if the $H^{SA}$ and $H^{FF}$ have different dimensions (which is often the case in some LLMs)? Equation (7) and (8) are not applicable in this case.\n\n**A3**: Thank you very much for this insightful comment. Yes, you are right to some extent. if the $H^{SA}$ and $H^{FF}$ have different dimensions, our method are not applicable to compose all sub-layers’ information appropriately. However, the original design of our approach is to compose each encoder layer’s information instead of each encoder sub-layer’s information appropriately. The reason why we conduct this experiment is that we are curious about whether composing different encoder layers’ or sub-layers’ information is more beneficial to CG. Results in Table 4 also indicate that COMPOSITION-SA or COMPOSITION-FF can also bring certain improvements compared with Transformer. So the methodology of the proposed COMPOSITION is rather general to many Seq2Seq models.\n> Q4: It indicates that large-scale language models, despite their huge number of parameters and training datasets, still fail to improve the CG performance\". This claim is sufficiently justified due to lack of evaluation.\n\n**A4**: Thank you very much for this insightful comment. We must admit this claim is insufficiently justified on CoGniton to some extent, because it's hard to evaluate a broad range of LLMs’ CG performance on CoGnition due to resource limits. However, there have some works investigating whether scaling up model size also improve compositional generalization (Qiu et.al., 2022; Hosseini et.al., 2022; Kim et.al., 2022). They found performance for the largest model size is generally worse than finetuning performance for much smaller models. We also provide experiments with the GPT3.5 on the CoGnition dataset in **A2** to further validate this argument. Finally ,we will revise these claims to make it clearer and more coherent.\n> Q5: what if we apply the listed encoder layer fusion methods on this task, what would be the performance like?\n\n**A5**: Thank you very much for this insightful comment. Following your suggestions, we reproduce DLCL (Wang et.al., 2019), one of the very popular EnocderFusion work, and conduct experiments on CoGnition dataset. The results are reported below.\n\n|Models|Instance-level CTER $\\downarrow$|Aggregate-level CTER $\\downarrow$|BLEU $\\uparrow$|\n|---------|---------|---------|---------|\n|Transformer|28.4%|62.9%|59.5|\n|DLCL|28.4% (+0.0%)|67.9% (+5.0%)|59.2 (-0.3)|\n|COMPOSITION|**20.4% (-8.0%)**|**52.0% (+10.9%)**| **61.5 (+2.0)** |\n\nResults are averaged over 6 random runs on CoGnition. It can be seen that Transformer is even slightly better than DLCL, indicating DLCL and COMPOSITION do share the same motivation or scenario. DLCL focuses on how to transform information across layers in deep neural network scenarios, while COMPOSITION is to learn to composing the syntactic and semantic representations of sequences appropriately for CG.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The studied problem, compositional generalization in machine translation, is important and critical to machine translation.\n2. The proposed approach is simple and straightforward.\n3. The paper is overall well-written and easy to follow.\n4. The performance boost compared with listed baselines is promising.", "reasons_to_reject": "1. The novelty of the proposed approach is limited. The difference between the proposed method and related encoder layer fusion approaches is marginal. \n2. The compared models are pretty small. Many existing pretrained LLM are not included for comparison. Check [LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) for more details.\n3. Some of the claims are not well justified, making the findings less solid. For example, line 141-143: \"our proposed method is mostly applicable to any Seq2Seq models\". The claim is not justified at all; line 335-337: \"It indicates that large-scale language models, despite their huge number of parameters and training datasets, still fail to improve the CG performance\". This claim is  sufficiently justified due to lack of evaluation.", "questions_to_authors": "1. what is performance of other LLM look like on the two datasets?\n2. what if the H^{SA} and H^{FF} have different dimensions (which is often the case in some LLMs)? Equation (7) and (8) are not applicable in this case.\n3. what if we apply the listed encoder layer fusion methods on this task, what would be the performance like?", "paper_topic_and_main_contributions": "In this paper, the authors proposed method COMPOSITION to improve the compositional generalization capability of transformers. The main idea is inspired by the findings that bottom layers of transformer contains more syntactic information while the top layers contain more semantic information. They proposed a composed layer that combines different encoder layers to improve CG. They conducted experiments on two datasets and demonstrated that the proposed approach outperforms compared baselines.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Please check [LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) for models and corresponding references."}, "extra_text": {}}, {"review_id": "Ym3vkrUlMP", "reviewer": "Reviewer_XNv7", "summary": "", "questions": "A:In the paper, the authors metion the \"source keys and values are entangled\" in many places(e.g., line14-line16, line98-line99, and discussion on Fig.5). However, the detailed discussion on the entanglement of the source keys and values representations(i.e., what is on earth the entanglement of these representations? The authors provide a PCA visualization in Fig.5. Also, what does it represent when those representations are located in the same clusters?) is supposed to be further clarified in avoidance of puzzling readers.\n\nB:The paper lacks the analysis on the extra(**compared with vanilla transformer-based models**) computation cost/ training time/ inference time brought by additionally introducing the composed layer and the different encoding representations passed to the different decoder layers. Could you please provide the analysis for this part?\n\nC:It seems that \"COMPOSITION\" is also generic to the commonly-used pre-trained transformer-based model. Could you please provide the corresponding results of verifying the effectiveness of utilizing the method to boost the performance of more pre-trained models(e.g., T5 or BART) on the CoGnition task?\n\nD:In the design of \"COMPOSITION\", what is the main difference between the $H_i^{SA}$ and the $H_i^{FF}$? Though the authors give the performance(Table.4) of separately using the $H_i^{SA}$ and the $H_i^{FF}$, I am wondering the reason why these two values are orthogonal to each other to some extent.(i.e., the reason why they both play roles in boosting the performance. e.g., are they represent different high-level info or something?).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reply to Questions , Missing References and Typos Grammar Style And Presentation Improvements\nThanks for your constructive comments and suggestions, and they are exceedingly helpful to improve our paper. We will incorporate them in the revised paper. In the following, your comments are first stated and then followed by our point-by-point responses.\n> Q1: In the paper, the authors metion the \"source keys and values are entangled\" in many places(e.g., line14-line16, line98-line99, and discussion on Fig.5). However, the detailed discussion on the entanglement of the source keys and values representations(i.e., what is on earth the entanglement of these representations? The authors provide a PCA visualization in Fig.5. Also, what does it represent when those representations are located in the same clusters?) is supposed to be further clarified in avoidance of puzzling readers.\n\n**A1**: Thank you very much for this insightful comment. As mentioned, the representation entanglement problem is the syntactic and semantic representations of sequences are twisted inappropriately. Previous work concentrates on **separating the learning of syntax and semantics, and then compose them appropriately**. On the contrary, we have conducted preliminary analysis provided in Appendix A, and concluded that **the bottom layers of the Transformer encoder contain more syntactic information and the top ones contain more semantic information**. Inspired by this, **we collect representations outputted by each encoder layer instead of separating the learning of syntax and semantics**. So one intuitive solution to solve the source keys, values entanglement problems is to learn **different and specific combinations** of syntactic and semantic information (i.e., representations outputted by each encoder layer) for keys and values of different decoder layers. Therefore, we propose a composed layer to compose them appropriately to generate specialized source keys and values passing into different decoder layers. As shown in Fig.5, in contrast to the Transformer, COMPOSITION obtains specialized keys and values representations passing into different decoder layers (i.e., **the dots representing keys and values of different decoder layers are in different places**). Notably, three circles represent randomly selected examples of CG-test set.\n\n> Q2: The paper lacks the analysis on the extra(compared with vanilla transformer-based models) computation cost/ training time/ inference time brought by additionally introducing the composed layer and the different encoding representations passed to the different decoder layers. Could you please provide the analysis for this part?\n\n**A2**: Thank you very much for this insightful comment. Following your suggestions, we have investigated its influence on CoGnition, and its results are presented as follows.\n\n|Models|upd/s|token/s|\n|---------|---------|---------|\n|Transformer|4.71(1.00$\\times$)|5986.00(1.00$\\times$)|\n|Dangle|1.23(0.26$\\times$)|2505.58(0.42$\\times$)|\n|COMPOSITION|4.05(0.86$\\times$)|5728.01(0.7$\\times$)|\n\n\"upd/s\" and \"token/s\" are abbreviated notations for \"training updates per second\" and \"generated tokens per second\". We observe that COMPOSITION keeps almost the same training and decoding speed as Transformer. On the contrary, Dangle prohibitively increases the training and decoding cost. Thus, it is clear that our method achieves promising results while brings little increase in computational cost compared with Transformer.\n> Q3: It seems that \"COMPOSITION\" is also generic to the commonly-used pre-trained transformer-based model. Could you please provide the corresponding results of verifying the effectiveness of utilizing the method to boost the performance of more pre-trained models(e.g., T5 or BART) on the CoGnition task?\n\n**A3**: Thank you very much for this insightful comment. We didn’t conduct experiments with commonly-used pre-trained transformer-based model (e.g., T5 or mBART) on CoGnition dataset for two reasons. **First**, it’s contrary to the compositional generalization task itself. Let’s back to the definition of compositional generalization task. Compositional generalization is the algebraic ability to understand and produce a potentially infinite number of **novel combinations** from **known components**, while PLM is a large language model, which are able to access **vast amounts of text data**. If we conduct experiments on PLMs to test their CG performance, one natural question can be raised: **how can we guarantee that every sentence in the test set is a novel combination from known components for LLMs**? **Second**, we want a fair comparison with previous work on CoGnition with a almost simliar parameter settings. \\\nHowever, to validate the general effectiveness of our approach, we follow your suggestions and conduct experiments with mBART  (Liu et al., 2020) and our method built on top of mBART (called COMPOSITION-mBART) on CoGnition dataset. The full experiment results are presented below.\n\n| Models | Params | Instance-level CTER $\\downarrow$ | Aggregate-level CTER $\\downarrow$ |\n| --- | --- | --- | --- |\n| Transformer | 35M | 28.4% | 62.9% |\n| COMPOSITION| 35M | 20.4% | 52.0% |\n| mBART| 611M | 14.9% (-13.5%) | 40.0% (-22.9%) |\n| COMPOSITION-mBART| 611M | **13.9% (-14.5%)** | **37.5% (-25.4%)** |\n\nIt can be seen that COMPOSITION-mBART outperforms mBART and achieves new state-of-the-art performance, demonstrating the general effectiveness of our method. However, compared with the impressive performance for COMPOSITION built on top of Transformer, the improvements brought by COMPOSITION-mBART is relatively moderate. The underlying reason is CoGnition is a synsestic, simple dataset where most sentences in the CG-test set are already seen in the PLMs' pre-training stage, causing less space for improving its CG performance. \\\nTherefore, to further validate the effectiveness of our method, we have conducted experiments with COMPOSITION on COGS (Kim and Linzhen, 2020) dataset (one of representative semantic parsing benchmarks for compositional generalization). Specifically, we build our method on top of Transformer, and use initialized embeddings (on the both source and target side) with Glove (Pennington et al., 2014).  The results are reported below.\n\n| Models | Accuracy $\\uparrow$ |\n| -- | -- |\n| Transformer | 85.5 |\n| Tree-MAML | 66.7 |\n| LexLSTM | 82.1 |\n| T5-base | 85.9|\n| COMPOSITION | **86.1** |\n\nIt is clear that COMPOSITION outperforms most baseline models, indicating its general effectiveness on other compositional generalization tasks. \n> Q4: In the design of \"COMPOSITION\", what is the main difference between the $H_{SA}^{i}$ and $H_{FF}^{i}$? Though the authors give the performance (Table.4) of separately using the $H_{SA}^{i}$ and $H_{FF}^{i}$, I am wondering the reason why these two values are orthogonal to each other to some extent.(i.e., the reason why they both play roles in boosting the performance. e.g., are they represent different high-level info or something?).\n\n**A4**: Thank you very much for this insightful comment. $H_{SA}^{i}$ and $H_{FF}^{i}$ are the outputs of Transformer self-attention and feed-forward sub-layer $i$ (line 198-203), respectively. As mentioned in Appendix A, the information encoded by each encoder layer transforms from syntactic to semantic as the number of layers increase (line 1112-1115). \nThe reason why they both play roles in boosting the performance and these two values are orthogonal to each other to some extent  is information brought by SA or FF is **similar**, but **slightly different** (Li et.al., 2020), respectively. It can be seen that the results of COMPOSITION-SA and COMPOSITION-FF presented in Table 4 are basically the same, and the improvements brought by the combination of both them is relatively moderate.\n> Q5: Just kindly note that there are a bunch of recent published works(e.g., in ACL'2023 proceedings) on compositional generalization which might be helpful for the authors to make the \"Related Works\" section up-to-date and thus make the audience better follow the recent progress of related works. line189: 2M learnable scalars? (I think here should be vectors ($H_{SA}^{i}$ or $H_{FF}^{i}$) instead of scalars?\n\n**A5**: Thank you so much for your careful reading! We will follow your suggestions and revise our paper for better readability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1):From my point of view, the proposed \"COMPOSITION\" is a quite generic method to improve the compositional generalization performance for most transformer-based seq2seq models, which is literally simple and thus requires only little modification on the original design of the transformer model.\n\n(2):The paper is well-motivated and easy-to-follow in most places. Besides, the experimental results and the analysis can fully demonstrate the effectiveness of the proposed method (both in the task performance level, concrete case level (i.e., case studies) and in the learned representation level (i.e., analytical experiments in the **Section 4** and **Appendix**)). And hence I appreciate the rigorousness and the completeness of this paper, which are necessary ingredients of a good scientic paper.", "reasons_to_reject": "This paper also has some weak points.\n\n(1):The paper lacks the analysis on the extra(**compared with vanilla transformer-based models**) computation cost/ training time/ inference time brought by additionally introducing the composed layer and the different encoding representations passed to the different decoder layers.\n\n(2):There are also some minor questions on the understanding of this paper. Please refer to the \"Questions For The Authors\" part (Note: the first weak point is also covered in that part, and hence it is OK for the authors to only respond to the questions in the \"Question\" part for their convenience).", "questions_to_authors": "A:In the paper, the authors metion the \"source keys and values are entangled\" in many places(e.g., line14-line16, line98-line99, and discussion on Fig.5). However, the detailed discussion on the entanglement of the source keys and values representations(i.e., what is on earth the entanglement of these representations? The authors provide a PCA visualization in Fig.5. Also, what does it represent when those representations are located in the same clusters?) is supposed to be further clarified in avoidance of puzzling readers.\n\nB:The paper lacks the analysis on the extra(**compared with vanilla transformer-based models**) computation cost/ training time/ inference time brought by additionally introducing the composed layer and the different encoding representations passed to the different decoder layers. Could you please provide the analysis for this part?\n\nC:It seems that \"COMPOSITION\" is also generic to the commonly-used pre-trained transformer-based model. Could you please provide the corresponding results of verifying the effectiveness of utilizing the method to boost the performance of more pre-trained models(e.g., T5 or BART) on the CoGnition task?\n\nD:In the design of \"COMPOSITION\", what is the main difference between the $H_i^{SA}$ and the $H_i^{FF}$? Though the authors give the performance(Table.4) of separately using the $H_i^{SA}$ and the $H_i^{FF}$, I am wondering the reason why these two values are orthogonal to each other to some extent.(i.e., the reason why they both play roles in boosting the performance. e.g., are they represent different high-level info or something?).", "paper_topic_and_main_contributions": "This paper presents a method, dubbed \"COMPOSITION\", towards the improvement of compositional generalization and specifically, the disentanglement of the information (i.e., the keys and the values) passed between the encoder and the decoder in the transformer sequence-to-sequence model. \n\nThe proposed method is basically simple and reasonable, which is mainly introduced an additional module, composed layer, right between the encoder and the decoder, encouraging the decoder layers to consider the composition of the information outputted by each encoder layers rather than merely the upmost encoder layer (i.e., the original design for the transformer model). The authors also present the intuition of their design, which is to disentangle the semantic information and the syntactic information in the modeling of the natural language. \n\nIn the experiment part, they conduct experiments on two representative and challenging compositional generalization task, CoGnition(machine translation) and CFQ(semantic parsing), and give corresponding case studies to empirically demonstrate the effectiveness of \"COMPOSITION\" (results on CoGnition reach new **SOTA**.). Some necessary analytical results are also available in the paper to help further verify the reasonableness of the initial hypothesis.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line189: 2M learnable scalars? (I think here should be **vectors** ($H_i^{SA}$ or $H_i^{FF}$) instead of **scalars**?)"}, "extra_text": {"Missing_References": "Basically, I think this work is novel and thus **no** new baseline work **must** be appended.\n\nJust kindly note that there are a bunch of recent published works(e.g., in ACL'2023 proceedings) on compositional generalization which might be helpful for the authors to make the \"Related Works\" section up-to-date and thus make the audience better follow the recent progress of related works."}}]

Author rebuttal:

Meta-review: This paper proposes a layer which composes the representations produced by the different layers of the encoder and outputs keys and values which are fed into the different layers of the decoder. The proposed layer is evaluated in the tasks of machine translation and semantic parsing.

The paper received mixed reviews. The reviewers found that that the paper has some merits: (i) the proposed approach is simple and quite general; (ii) the method appears to be empirically strong in the machine translation task; (iii) the considered problem is interesting and deserves further research. However, the reviewers also raised various concerns mainly about the lack of novelty, the lack of large baselines and about claims that appear in the paper and which are not well-justified. For instance, one reviewer stressed that there is no evidence that the proposed method indeed effectively combines semantic with syntactic information. There were also concerns about the presentation of the paper which gives the reader the impression that the proposed method is entirely novel and that there is no overlap with prior work.

Taking author response into account, I think that the paper is interesting and that this line of investigation is worthwhile, but I also agree with some of the concerns that were raised in the reviews.
Decision: Accept-Findings