Permutation Equivariant Models for Compositional Generalization in Language
Authors: Jonathan Gordon
David Lopez-Paz
Marco Baroni
Diane Bouchacourt
Venue:  (2020)
Keywords: Compositionality
Permutation Equivariance
Language Processing

Abstract: Humans understand novel sentences by composing meanings and roles of core language components. In contrast, neural network models for natural language modeling fail when such compositional generalization is required. The main contribution of this paper is to hypothesize that language compositionality is a form of group-equivariance. Based on this hypothesis, we propose a set of tools for constructing equivariant sequence-to-sequence models. Throughout a variety of experiments on the SCAN tasks, we analyze the behavior of existing models under the lens of equivariance, and demonstrate that our equivariant architecture is able to achieve the type compositional generalization required in human language understanding.

Published as a conference paper at ICLR 2020
PERMUTATION EQUIVARIANT MODELS FOR COMPOSITIONAL GENERALIZATION IN LANGUAGE
Jonathan Gordon University of Cambridge jg801 @cam. ac. uk
David Lopez-Paz, Marco Baroni, Diane Bouchacourt Facebook Al Research (dlp, mbaroni, di dianeb] com com
ABSTRACT
Humans understand novel sentences by composing meanings and roles of core language components, In contrast, neural network models for natural language modeling fail when such compositional generalization required. The main contribution of this paper is to hypothesize that language compositionality is a form of for constructing equivariant sequence-to ceeenncetto-sepeee -oseuuenee models. Throughout variety of experiments on the SCAN tasks, we analyze the behavior of existing models under the lens of equivariance, and demonstrate that our equivariant architecture is able to achieve the type compositional generalization required human language understanding.
Based on this hypothesis, we propose a set of tools
INTRODUCTION
When using language, humans recombine known concepts to understand novel sentences. For instance, if one understands the meaning of "run", "jump", and "jump twice", then one understands the meaning of "run twice", even if such sentence was never heard before. This relies on the notion of language compositionaaltyy, which states that the meaning of sentence ("jump twice") is to be obtained by the meaning of its constituents (e.g. the verb "jump" and the quantifying adverb "twice") and the use of algebraic computation verb combined with quantifying adverb results in doing that verb m times) (Kratzer & Heim, 1998).
In the realm of machines, deep learning has achieved unprecedented results in language modeling tasks (Bahdanau et al., 2015; Vaswani et al., 2017). However, these models are sample inefficient, and do not generalize to examples that require the use of language compositionality (Lake & Baroni, 2018; Loula et al., 2018; Dessi & Baroni, 2019). This result suggests that deep language models fail to leverage compositionality; failure remaining to this day roadblock towards true natural language understanding.
Focusing on this issue, Lake & Baroni (2018) proposed the Simplified version of the CommAT Navigation (SCAN), a dataset to benchmark the compositional generalization capabilities of state-of- the-art sequence-to-sequence (seq2seq) translation models (Sutskever et al., 2014; Bahdanau et al., 2015). In a nutshell, the SCAN dataset contains compositional navigation commands such as JUMP TWICE AFTER RUN LEFT, to be translated into the sequence of actions LTURN RUN JUMP JUMP.
Using SCAN, Lake & Baroni (2018) demonstrated that seq2seq models fail spectacularly at tasks requiring the use of language compositionaliiy. Following our introductory example, models trained on the three commands JUMP, RUN and JUMP TWICE fail to generalize to RUN TWICE. Most recently, Dessi & Baroni (2019) showed that architectures based on temporal convolutions meet the same fate.
SCAN did not only reveal the lack of compositionality in language models, but it also became the blueprint to build novel language models able to handle language compositionality. On the one hand, Russin et al. (2019) proposed a seq2seq model where semantic and syntactic information are represented separately, in a hope that such disentanglement would elicit compositional rules. However, their model was not able to solve all of the compositional tasks comprising SCAN, On the other hand, Lake (2019) introduced a meta-learmin learning approach with excellent performance in multiple
Work conducted during an internship Facebook AI Research

---

Published as a conference paper at ICLR 2020
SCAN tasks. However, their method requires substantial amounts of additional supervision, and complex meta-learning procedure fand-engineered for each task.
In this paper, take a holistic look at the problem and connect language compositionality in SCAN to the disparate literature in models equivariant to certain group symmetries (Kondor, 2008; Cohen & Welling, 2016; Ravanbakhsh et al., 2017; Kondor & Trivedi, 2018). Interesting links have recently been proposed between group symmetries and the areas of causality (Arjovsky et al., 2019) and disentangled representation learning (Higgins et al., 2018), and this work proceeds in similar fashion. In particular, the main contribution of this work is not to chase performance numbers, but to put forward the novel hypothesis that language compositionality can be understood as a form of group-equivarre equivarivvve (Section 3). To sustain our hypothesis. we provide tools to construct seq2seq models equivariant when the group symmetries are known (Section 4), and demonstrate that these models solve all SCAN tasks, except length generalization (Section 6).
2 THE SCAN COMPOSITIONAL TASKS
The purpose of the Simplified version of the CommAl Navigation (SCAN) tasks (Lake & Baroni, 2018) is to benchmark the abilities of machine translation models for compositional generalization. Following prior literature (Lake Baroni, 2018; Baroni, 2019; Russin et al., 2019; Andreas, 2019), compositional generalization is understood as the ability to translate novel families of sentences, when this requires leveraging the compositional structure in language.
The SCAN dataset contains compositional navigation commands in English (the input-language) paired with a desired action sequence (the output-language). For instance, the input-language sentence JUMP TWICE AND RUN LEFT is paired to the output-language actions sequence JUMP JUMP LTURN RUN. The rest of our exposition uses SMALL CAPS to denote examples in the input-language. and LARGE CAPS to denote examples in the output-language. Appendix A contains full description of the grammar generating the SCAN language.
To evaluate the compositional generalization abilities of sequence-to-sequeee (seq2seg) machine translation models (Sutskever et al., 2014; Bahdanau et al., 2015), Lake & Baroni (2018) proposes four main tasks based on SCAN:
1. Simple task: data pairs are randomly split into training and test sets. No compositional generalization is required.
2. Add jump task: the only command in the training set containing the verb JUMP is the command JUMP. All commands not containing JUMP are in the training set (for instance, RUN TWICE, and WALK RIGHT THRICE AND LOOK LFFT). The test set contains all commands containing JUMP (for instance, JUMP TWICE, and RUN LEFT AND JUMP RIGHT). To succeed in this task, models must learn that JUMP is a verb, and that any verb can be composed with an adverbial number to be repeated a number of times.
3. Around right task: the phrase AROUND RIGHT is held out from the training set; however, both AROUND and RIGHT are shown in all other contexts (for example, AROUND LEFT or OPPOSITE RIGHT). To succeed at this task, models must learn that both RIGHT and LEFT are directions, and can be combined with AROUND and OPPOSITE.
4. Length generalization task: the training set contains pairs such that the length of the action sequence in the output-language is shorter than 24 actions. The test contains all pairs with action sequences of a length greater or equal than 24 actions. The type of compositional ability required succeed at this task is more difficult to sketch out, as we discuss in Section 6.2.
Lake & Baroni (2018) use these four tasks to demonstrate that sate-of-the-at seq2seq translation models (Bahdanau et al., 2015) succeed at Simple task, but fail at the other three tasks requiring compositional generalization. Convolutional architectures (Dessi & Baroni, 2019) achieve only slightly better performance, and state-of- of-te-of-the-arr art methods specially developed to address SCAN tasks fall short from the best achievable performance (Russin al., 2019), or call for substantial amounts of additional supervision (Lake, 2019).
"Code available http://tthccnnncccccceee///mmmmm
2

---

Published as a conference paper at ICLR 2020
In the following, we take holistic look the language compositionality problems in SCAN, and highlight their connection to equivariant maps in group theory.
3 SCAN COMPOSITIONALITY AS GROUP EQUIVARIANCE
This section puts forward the hypothesis that:
Models achieving the compositional generalization required in certain SCAN tasks equivariant with respect to permutation group operations in the input and output languages.
To unfold the meaning of our hypothesis, must revisit some basic concepts in group theory. A discrete group is a set of elements 1.1..... equipped with a binary group operation "." satisfying the four group axioms (closure, associativity, identity, and invertibility). The sequel focuses on permutation groups G, whose elements are permutations of a set X, and whose binary group operation composes the permutations contained in G. The set of all permutations of X' is a group, but not all subsets of permutations of X satisfy the four group axioms, and therefore they do not form group. For each element g E G. we define the group operation Tg X + X' the map applying the permutation g to the element x E X, to obtain T. Armed with these definitions, we are ready to introduce the main object of study this paper: equivariant maps.
Definition 1 (Equivariant map). Let X' and y be two sets. Let G be group whose group operation on X' is denoted by T, X + X. and whose group operation on y is denoted by g : y. Then, c X y is an equivariant map if and only if 4 (Tzz) T o (xx for all x E X' and g G.
The operation groups (Tg, Thy) defined above operate entire sequences, enormous space when we consider those sequences to be language sentences. In the following two definitions, we relax group operations and equivariant maps operate at word level.
(Local group operations). Let be a set of sequences (or sentences), where each sequence X' contains elements x; V from vocabulary set V, for all E x. Let G be group with associated group operation Ty : X' >> X. Then, we say that T, is a local group operation there exists group operation g... V x V such that Tyx (Tg.. T1.-...... , Ta x L. for all X.
Definition 2
When understanding sequences as language sentences, the group operation Tg.. would be a permuta- tion of the words from the language vocabulary. Such operation can be implemented in terms of permutation matrix, a [V] x [V] matrix with zero/one entries where each row and each columa sum to one. Finally, we leverage the definition of local group operations to define locally equivariant maps.
Definition 3 (Locally equivariant map). Let and y be two sets of sequences. Let G be a group whose group operation on X' is local in its vocabulary, denoted by T. X' + X, and whose group operation on y is local in its vocabulary and denoted by T% 3² x y/ Then, we say that &b X + y an equivariant map if and only (TTgar) = T' for all x E X' and E G.
Tg
RUN LEFT AND WALK
SUN LEFT APTER WALK
WALK LEFT JUMP LEFT AND RUN
Tb
T
LTURN WALK
JUMP
RUN
LEURN
(a)
(c)
Figure I: (a) Commutative diagram for equivariance. (b) Local equivariance enables generalization to verb replacement in SCAN. (c) Local equivariance does not enable generalization to conjunction replacement in SCAN.
*Standard terminology in group theory denotes the elements of group "to act on" elements of a set. However, as the output language in SCAN is in navigation "actions", we use the "operate" terminology for group elements avoid ambiguity.
3

---

Published as a conference paper at ICLR 2020
Now, how do equivariances and local equivariances manifest themselves in the world of SCAN? To assist our examples, the commutative diagram in Figure la summarizes the group theory notations introduced far. In Figure 1b and Figure 1c, we parallel these notations to two different examples of compositional skills required to solve SCAN: verb and conjunction replacement. In the SCAN domain, X' is the set of sentences in the input-language, and ) the set of sentences in the output- language. Furthermore, let & be a locally equivariant SCAN translation model, and let G be group with associated local group operations that permutes words in the input- and output- languages.
On the one hand, we observe in Figure lb that local equivariance enables compositional generalization in the case of verb replacement. This is because replacing one verb in the input-language can be implemented in terms of a local group operation. In turn, this input-verb replacement corresponds deterministically to a second local group operation that replaces the corresponding verb in the output-language. The same would apply to a SCAN task where we are interested in generalizing to the replacement of LEFT and RIGHT. As such, a translation model with these compositional generalization capabilities must be locally equivariant.
On the other hand, we observe in Figure Ic that local equivariance insufficient to enable composi- tional generalization in the case of conjunction replacement. This is because no local group operation in the laguut-angguge would be able to implement the necessary changes induced by the replacement of AND by AFTER the imput-language. In such cases, refer to the equivariance global equivariance. In particular, we can see how blocks of multiple words in the output-language swap their relative location. Local equivariances also insufficient to enable compositional generalization in the Length generalization SCAN task and we elaborate on this in Section 6.2.
In the following section, we propose a set of tools to implement equivariant seq2seq translation models, and propose particular architecture with which we conduct our experiments.
4 IMPLEMENTING AN EQUIVARIANT
MODEL
We now implement our proposed equivariant seq2seq model, following the encoder-decoder architec- ture illustrated in Figure 2. Readers unfamiliar with group theory may parse Figure 2 by temporarily discarding the "G- prefixes, and realize that each depicted module is well-known building block of recurrent neural network models.
G-RNN
fu
G-RNN
h
G-Deende
e(w,)
G-Embed
G-Attention
G-Conv
G-Embed
e(t 1]
(a) G-Equivariant Encoder
(b) G-Equivariant Decoder
Figure 2: Architecture of our fully-equivariant seq2seq model. Variables shaded in gray mappings G R implemented as |G] x K matrices. Encoder and decoder meet at ha = hL,-
To make our model equivariant, we will make intense use of group convolutions.
Definition 4 (Group convolution (Kondor & Trivedi, 2018)). Let G be a discrete group. Let f G -> RK be an input function. Let go- = {v² G + R | be set of learnable filter functions. Then, each scalar real entry from the result of G-convolving f and &e is given by |G] x K" matrix with entries
K
G-Conv(J:
Σ
Σ
(1)
kEdom A=1
for all g G and i E (1). As shown by (Kondor & Trivedi, 2018). the G-Conv layer is equivariant wrt the operations of G. We apply this definition in two ways: (i) "convolving" words with learaalle filters generate equivariant embeddings. Later, when introduce our notations,
4

---

Published as a conference paper at ICLR 2020
discuss how words may be viewed as functions to fit the definition. And (ii) convolving two group representations, in which case dfm(f) = G.
We note that there are several additional methods proposed in the literature for constructing per- mutation equivariant layers (e.g., Zaheer et al., 2017: Ravanbakhsh et al., 2017). However, as demonstrated by Kondor & Trivedi (2018); Bloem-Ree Reddy & Teh (2019), the above form is very general and subsumes most alternatives. Further, while layers based on weight-sharing may be more efficient than the general form of Definition 4, the parameter tying restricts the capacity of the layer. For example, the permutation equivariant layer of Zaheer et al. (2017) requires weight matrices that are restricted form XI + (11)²² with learnable parameters À and y. This layer has fewer learnable parameters than the convolutional form of Definition 4. Thus, for reasons of generality and capacity, we employ the general and expressive convolutional form of Definition 4 for our permutation equivariant layers.
Equivariant with respect what group? The previous G-Con layer requires choosing a discrete group G. As hinted in Section 3, we will choose to contain [G] permutations of language vocabularies, e.g. products of cyclic groups the set of all permutations has a size of n!. However, it suffices to consider subgroups containing permutations such that every word can be reached by composing elements of the subgroup. For example, while the group of permutations on the four verbs in SCAN consists of 24 elements, it will suffice to choose G as the circular shift group the four verbs, which is subgroup of four elements. Following standard notation in group theory. we write g h to denote the composition of two group elements g, h G, and g denote the inverse element of g-
sets of words. Note that for a vocabulary size of [V]
As final preliminaries, denoting [V] = (1...., V), we represent a word w in the input-language by the function : [V] + {0, 1}. where Σn [v w(v) = 1, and similarly by using w E for the output-language. These notations are functional representations of word one-hot encodings that will play well with our notations. Note that this representation equivalent to one-hot vectors, and in what follows we use the shorthand w for the one-hot vector representation of words.
To avoid notational clutter, we use 9 to denote the permutation inttiattioommaiiixxxxi -ww.rm---eaeeeenttiieee representation of the corre- sponding group element. Thus, the group operation on word gu can be implemented as matrix multiplication between the permutation matrix y and the one-hot vector w. Note that this operation results in another one-hot vector, i.e. another word in the vocabulary. Similarly, the binary group operation can be written as matrix multiplication gh between two group members g, G. Here too, multiplication of permutation matrices results in permutation matrix, so gh G.
We now describe each the components in our G-equivariant translation model, by following the transformation process of an input sequence (Wi,.... Wr, ) (in SCAN, navigation command in English) into its output translation = (wi,.... w.,) (in SCAN, a sequence of actions).
4.1 G-EQUIVARIANT ENCODER
Upon arrival, the lanutalanguage sentence (Wi..... WL.) is sent G-equivariant encoder. The first step in the encoding process is to transform each input word w, into a permutation equivariant embedding e(wy). As mentioned before, each word w; is represented by the one-hot vector w, - +0, {0. The corresponding embedding is obtained by applying set of K 1-dimensional learnable filter functions {v' [V] RRJ.K in group convolution (throughout the section, we use K everywhere to ease notation). Using Definition 4, the embedding, which we call G-Embed, is then represented as a matrix TR x where
e(w) g.a G-Emmed w; )).. (t
(2)
for all g E G and i (1...., K}. Note that since w is one-hot vector, G-Embed is a particularly
simple instantiation of Definition 4, as summation over (f) consists of only single term. The corresponding embedding is function e(w;) R - matrix, where each row corresponds to the embedding of the word gw for a particular E G. This layer can be implemented by defining s with standard deep learning embedding modules."
which can be represented
|G] K
".g. PyTorch NN. E..EDEEDDING
5

---

Published as a conference paper at ICLR 2020
Importantly, note that for this layer, both a and w are functions on [V] However, the resulting embedding e(wr) is a function on the group G. Therefore, in all subsequent computations we will require the learnable filters & to also be functions on G.
We illustrate this layer with an example. Let G be the cyclic group that permutes the words LEFT and RIGHT. We can think of 91 as the identity, and 92 as permuting the words LEFT and RIGHT (leaving all other words unchanged). In this case, embedding LEFT results in the 2 x K matrix (#(LEFT)" (RIGHTT)" while embedding JUMP results in [((UM)))" ((JUMP)T since both and Ø2 act as the identity permutation for JUMP.
Next, the word embedding e(w,) is sent to a permutation equivariant Recurrent Neural Network (G-RNN). The cells of a G-RNN mimic those of a standard RNN, where linear transformations are replaced by G-Convs (Definition 4). This cell receives two inputs (the word embedding e(w, and the previous hidden state 14) and returns one output (the current hidden state hç), all three being functions - IR K parametrized as |G] K matrices. More specifically:
h4 C-RNN(<(w))), 11-1) of (G-Conv(h 1hx) G-Conv(e(w)); th)))
where uh,, th: G R are learnable filters (represented as |G] K matrices), and is point-wise activation function.
The cell G-RNN is equivariant because the sum of two equivariant representations is equivariant (Cohen & Welling, 2016), and the pointwise transformation of an equivariant representation is also equivariant. To initialize the hidden state, we set ho = J. We note that our experiments use the equivariant analog of LSTM cells (Hochreiter & Schmidhuber, 1997), which we denote G-LSTM, since these achieved the best performance. We include the architecture of G-LSTM cells in Appendix B.
This completes the description of our equivariant encoder, illustrated in Figure 2a.
4.2 G-EQUIVARIANT DECODER
Once the entire input-language sentence (W1, .... (..,.... WL has been encoded into the hidden representations h = (hi..... h..,,.,..,,,,,, ), we are ready to start the decoding process that will produce the (wi,...,ū1 .L,,).
output-language translation
As illustrated in Figure 2b, our equivariant decoder is also run by an equivariant recurrent cell G-RNN. We denote the hidden states of the recurrent decoding process by h, where ho = hL.- At time t, the two inputs to the decoding G-RNN cell are the previous hidden state h as well as an attention ā over all the encoding hidden states h. (Once again, all variables are mappings G + RR implemented as |G| x K matrices.)
Attention mechanisms (Bahdanau et al., 2015; Vaswani et al., 2017) have emerged as a central tool in language modelling. Fortunately, attention mechanisms typically implemented as linear combina- tions, and a linear combination of equivariant representations itself an equivariant representation. We now leverage this fact to develop an equivariant attention mechanism. Given all the encoder hidden states h, as well the previous decoding hidden state h propose the equivariant analog of dot-product attention (Luong et al., 2015) as
G-AAtention 11) anzhi where
(4)
exp Be,
Ot.
and Br.j Σ (9) hz(9) DEG
(5)
k== exp Br.k
Following Figure 2b, the attention &e and a G-embedding e(w;-1) for the previous output word are concatenated and sent to a G-Convolution.' The concatenation with e(w;-1) provides the decoder with information regarding the previously embedded word. In practice, during training we use teacher-foreing (Williams & Zipser, 1989) to provide the decoder with information about the correct output sequences. This process returns final hidden representation G + k "oo embed we use special "start of sentence" symhol, appended the output language vocabulary.

---

Published as a conference paper at ICLR 2020
As final step in the decoding process, we need to convert Φ into a collection of logits the lanuu-language vocabulary. Then, sampling from the categorical distribution induced by these logits at time (or taking the maximimum) will produce the word w to be appended in the output-language translation, y. This final decoding module can be implemented as follows:
G-Decode(ds ()
Σ
(6)
BEG km
where 3 [V] R$ are the learnable parameters of this layer (represented by [V] x K matrix).
Recall that (h) E R is the final-layer representation for the group element h, and that h- the inverse element of h G applied to the output word i (represented as one-hot vector), which results in another word the output language. Thus, v is a learnable embedding of the output words into RKK This layer is evaluated at every vē the output vocabulary produce scalar. The resulting vector of logits represents a categorical distribution over the output vocabulary. While similar, this layer is not group convolution (Definition 4). Rather, equivariance for this module is achieved via parameter-sharin sharing (Ravanbakhsh et al., 2017).
"w is
This completes the description of our equivariant decoder, illustrated in Figure 2b. Composing the equivariant encoder and decoder results in our complete since all operations in this model are equivariant, the complete model is itself also equivariant the group G (Kondor& & Trivedi, 2018). In Section 6, we provide further implementation details for our model, and detail our empirical evaluation of its equivariant properties and their relation to the SCAN tasks described in Section 2.
model. Importantly.
RELATED WORK
In this section we review state-of-the methods to address SCAN compositional tasks. We focus on two recent models that we will compare to in our experiments.
On the one hand, the syntactic attention model of Russin et al. (2019) builds on the idea that composi- tional generalization can be achieved by language models given the correct architectural organization. Borrowing inspiration from neuroscience, Russin et al. (2019) argue that compositionality might arise when using separate processing channels for semantic and syntactic information. In their model, the attention weights depend on a recurrent encoding of the input sequence, which they refer to as the syntactic representation. The attention weights are then applied to separate, context- independent embeddings of the words in the input sequence, which intend to model a semantic representation. We find (Russin et al., 2019) interesting from a group equivariance perspective, since one way to enforce equivariance is to use an invariani representation (about syntax) together with an additional representation (about semantics) that maintains the information about the original "sentence pose".
On the other hand, the meta-learning (Thrun & Pratt, 2012; Schmidhuber, 1987) approach of Lake (2019) is model that learns to generalize. In particular, Lake (2019) designs one specific and complex meta-learning procedure for each SCAN task, where a distribution over tasks is provided to the learner (Finn et al., 2017; Gordon et al., 2018). For example, in the Add jump and Around right tasks, the meta-learning procedure of Lake (2019) samples permutations from the relevant groups (the permutation groups on the verbs and set of directions, respectively). This interpreted as lata-augmentation, valid procedure for encouraging equivariance (Cohen & Welling, 2016; Andreas, 2019; Weiler et al., 2018). However, at test-time, Lake (2019) sets the context set to the correct mapping between the permuted commands and their corresponding actions. For example, in the Add jump task, the context set for meta-testing would consist of the following pairs: | ((WAL,, WALK), (RUN, RUN), (LOOK, LOOK), (JUMP, JUMP) This is equivalent providing the model with one-to-one information regarding the correct command to-adctiiooooionnn mapping for the permuted words.
6 EXPERIMENTS
We now evaluate the empirical performance of our equivariant seq2seq model (described in Section 4) on the four SCAN tasks (described in Section 2). We compare our equivariant seq2seq to regular seq2seq models (Lake & Baroni, 2018), convolutional models (Dessi & Baroni, 2019). the syntactic attention model of Russin et al. (2019), and the meta-learning approach of Lake (2019). The compared seq2seq models use bi-directional, single-layer LSTM cells with 64 hidden units. For the equivariant
7

---

Published as a conference paper at ICLR 2020
Model
Simple
Add Jump
Around Right
Length
seq2seq (Lake & Baroni, 2018)
99.7
1.2
NA
13.8
CNN (Dessi & Baroni, 2019)
100.0
69.2 + 9.2
56.7 + 10.2
0.0
Syntactic Attention (Russin et al., 2019)
100.0
91.0 + 27.4
28.9 + 34.8
15.2 + 0.7
Meta seq2seq (Lake, 2019)
NA
99.9
99.9*
16.64
seq2seq (comparable architecture)
100.0
0.02 + 2e-2
12.4 + 2.3
Equivariant seq2seq (ours)
100.0
99.1 + 0.04
92.0 + 0.24
15.9 = 3.2
Table 1: Test accuracies for four SCAN tasks, comparing our equivariant seq2seq to the state-of-the-art.
seq2seq models, we use the cyclic permutation group on the verbs for the Add jump task, and the cyclic permutation group on directions for the Around right task. For Length, we use the product of those groups. Our model knows that the same group operates on both the input- and output- languages. However, it does not receive information regarding the correspondence between commands and actions in the set of words being permuted in the input / output languages. This is in contrast to Lake (2019), where (as stated in Section 5), it is necessary to provide the model with explicit information regarding the correct command-to-action mapping test-time.
Training procedures match those of Lake & Baroni (2018) where possible. We train models for 200k iterations, where each iteration consists of minibatch of size using the Adam optimizer to perform parameter updates with default parameters (Kingma & Ba, 2015) with learning rate of le-4. We use teacher-forcing (Williams & Zipser, 1989) with ratio of 0.5, and early-stopping based on a validation set consisting on 10% of the training examples. As in previous works, we compute test accuracies by counting how many exact translations each model provides, across the test set associated to each task.
6.1 RESULTS
Table I summarizes the results of our experiments. First and as expected, all models achieve excellent performance on the Simple task, which does not require any form of compositional generalization.
Second, our equivariant seq2seq model performs very well the Add jump and Around right SCAN tasks, which are the two tasks satisfying our local equivariance assumption from Definition 3. Our equivariant seq2seq model significantly outperforms the regular seq2seq (Lake & Baroni, 2018) and convolutional (Dessi & Baroni, 2019) models, well the state-of-the-art methods of Russin et al. (2019) and Lake (2019). This result is an encouraging piece of evidence supporting our main hypothesis from Section 3. Next, let us compare the results of our equivariant seq2seq model with the previous state-of-the-ard Russin et al. (2019); Lake (2019) in more detail.
On the hand, the syntactic attention model of Russin et al. (2019) achieves significant improve- ments over baselines methods at the Add jump SCAN task. However, it does not fare so well on the Around right task. Furthermore, its performance has high variance. Although we here report the numbers from Russin et al. (2019), we observed such high variance in our own implementation as well, where the model often achieved 0% test accuracy. We hypothesize that modeling the invariance of the syntactic attention directly would result in improved performance and stability. This can be achieved, for instance, by replacing all verbs in the syntactic module by shared word. As expected, by explicitly exploiting equivariance, our model outperforms Russin et al. (2019) on the Add jump and Around right SCAN tasks, also being much more robust.
On the other hand, the meta-learning model of Lake (2019) achieves excellent performance on the local equivariance tasks Add jump and Around right. This additional evidence supporting the use- fulness of local equivariance. In contrast to our model, Lake (2019) requires (i) complicated model and training procedure tailored to each task, (ii) providing the model with the correct permutation of words, equivalent to telling the model the "true" mappings between the input and output words, and (iii) augmenting the set of words being permuted, to ensure enough diversity in the training distribution (for instance, adding additional directions beyond RIGHT and LEFT).
8

---

Published as a conference paper at ICLR 2020
6.2 ON THE DIFFICULTY OF LENGTH GENERALIZATION
As seen in Table I, length generalization remains a tough challenge in SCAN. While generating long sequences is known challenge in seq2seq models (Bahdanau et al., 2015), we believe that this not the main issue with our equivariant seq2seq model, as it is able to produce long translations when these appear in the training set (as are the other models). Therefore, this not a capacity problem, but one of not being able to express the Length generalization SCAN task in terms of local equivariances on both input- and output- languages. We hypothesize that this the very reason why (Russin et al., 2019; Lake, 2019) also fail on this task.
However, we suspect that some forms of local equivariance on the input language, but global equivariance the output language, may help. For example, RUN TWICE, RUN THRICE and RUN AROUND LEFT TWICE are all input commands contained in the training set of the length task. A trained seq2seq model is able to execute them, but fails on the unseen test command RUN AROUND LEFT THRICE, suggesting that the network did not correctly understand the relationship between TWICE and THRICE. Using network that is explicitly equivariant to the permutation of TWICE and THRICE should be able to generalize correctly on RUN AROUND LEFT THRICE. However, while the THICE-THRICE permutation is a local group operation (Definition 2), the corresponding operation on the output language, which is to repeat the same action sequence multiple times, is a global group operation. Similarly, permuting AND and AFTER in the input sequence using local group operation, while operating globally on the output language by permuting the order of the associated actions, should help succeed on the Length generalization SCAN task. How to formalize the aforementioned global operations on the output language and build the desired equivariant network remains a fascinating open research question that we leave for future work.
7 DISCUSSION AND FUTURE WORK
This work has introduced hypothesis linking between group equivariance and compositional gen- eralization in language. Motivated by this hypothesis, we have proposed an equivariant seq2seq translation model, which achieves state-of-the-ar performance on variety of SCAN tasks.
Our work has several points for improvement. Most importantly, our model requires knowing the permutation symmetries of interest, to be provided by some domain expert. While this is simple to do in the synthetic language of SCAN, it may prove more difficult in real-world tasks. We propose three directions to attack this problem. (i) Group words by their purts-of-speech (e.g., nouns, verbs, etc.), which can done automatically by standard part-of-speech taggers (Märquez & Rodríguez, 1998); (ii) Learn such groupings of words from corpora, for example using the recent work of Andreas (2019); (iii) Most appealingly, parameterize the symmetry group and learn operations end-to-end while enforcing the group structure. For permutation symmetries, the group elements can be parameterized by permutation matrices, and learned from data (Lyu et al., 2019). Our preliminary work in this direction hints that this is fruitful avenue for future research.
A further consideration to address is that of computational overhead. In particular, for the convolu- tional form we use in this work (Definition 4), computational complexity scales linearly with the size of the group. O(G))). This arises from the need to sum over group elements when the representation a function on G. and may be prohibitive when considering large groups. One way of addressing this issue when large symmetry groups are of interest is to consider more efficient computational layers for permutation equivariance (e.g Zaheer et al., 2017; Ravanbakhsh et al., 2017). These methods incur less computational overhead at the cost of restricting the layer capacity. Another interesting option for future research is to consider sub-sampling group elements when performing the summation in Definition 4, which requires further consideration of the consequences of doing so.
Another exciting direction for future research to consider global equivariances. Many operations of interest, e.g. groups operating directly parse trees, can only be expressed as global equivariances. Modeling these equivariances holds exciting possibilities for capturing non-trivial symmetries in language tasks, but also requires more sophisticated machinery than is proposed in this work.
Finally, in further theoretical work, we would like to explore the relation between our equivariance framework and the idea of compositionality in formal semantics (Kratzer & Heim, 1998). On the one hand, the classic idea of compositionality as an isomorphism between syntax and semantics is intuitively related the notion of group equivariance. On the other hand, as shown by the failures

---

Published as a conference paper at ICLR 2020
at the length generalization example, it is still unclear how to apply our ideas to more sophisticated forms of permutation, such as those involving grammatical phrases rather than words. This would also require to extend approach to account for the context-sensitive composition (c.f., the natural interpretation of "run" in "run the marathon" vs. "run the code").
that pervades linguistic
ACKNOWLEDGMENTS
We thank Emmanuel Dupoux and Clara Vania for helpful feedback and discussions.
REFERENCES
Jacob Andreas. Good-enough compositional data augmentation, 2019.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: /907. 002893, 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In Proceedings of ICLR Conference Track, San Diego, CA, 2015.
Published online: http:
cc/ /o/kk php id==icr2211 :main.
Marco Baroni. Linguistic generalization and compositionality in modern artificial neural networks. arXiv preprini 1rX00:1904.00157, 2019.
Benjamin Bloem-Reddy and Yee Whye Teh. Probabilistic symmetry and invariant neural networks. arXiv preprint arXiv:/901.06092. 2019.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999, 2016.
Roberto Dessi and Marco Baroni. CNNs found to jump around more skillfully than RNNs: Com- positional generalization in seq2seq convolutional networks. arXiv preprint 1rXiv:1905.088. 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Vone Volume 70, pp. 1126-1135. JMLR. org, 2017.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Meta- learning probabilistic inference for prediction. arXiv preprint 1rXiv:1805.0990 2018.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards definition of disentangled representations. arXiv preprint 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Imre Risi Kondor. Group theoretical methods in machine learning. Columbia University, 2008.
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups arXiv preprint arXiv:1802.088 1802.08620000 2018.
Angelika Kratzer and Irene Heim. Semantics in generative grammar, volume 1185. Blackwell Oxford, 1998.
Brenden Lake. Compositional generalization through meta requence-to-sequence learning. arXiv preprint arXiv: 10X.0010... 2019.
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional
skills of sequence-to sequence-to-sequenec to-seuuence recurrent networks. In Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholm, Sweden, pp. 2879-2888. 2018.
10

---

Published as a conference paper at ICLR 2020
João Loula, Marco Baroni, and Brenden Lake. Rearranging the familiar: Testing compositional generalization in recurrent networks. pp. 108-114, 01 2018. doi: 10. 10.18653/v1/W18 18- 5413.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04000 2015.
Jiancheng Lyu, Shuai Zhang, Yingyong Qi, and Jack Xin. AutoShufileNe:: Learning permutation matrices via an exact lipschitz continuous penalty in deep convolutional neural networks. arXiv preprint /r0iv: /904.0864, 2019.
Lluís Märquez and Horacio Rodríguez. Part-of-speech tagging using decision trees. In European Conference Machine Learning, pp. 25-36. Springer, 1998.
Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos. Equivariance through parameter-sharing. In Proceedings of the 34th International Conference on Machine Learning-Vors Volume 70, pp. 2892- 2901. JMLR. org, 2017.
Jake Russin, Jason Jo, and Randall C O'Reilly. Compositional generalization in a deep seq2seq model by separating syntax and semantics. arXiv preprint arXiv: (eee::9044900000 2019.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, learning how learn: the meta-meta... hook. PhD thesis, Technische Universität München, 1987.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances neural information processing systems, pp. 3104-3112, 2014.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition, pp. 849-858, 2018.
Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems, pp. 3394-3404, 2017.

---

Published as a conference paper at ICLR 2020
A DETAILS ON THE SCAN DATASET
SCAN is composed from non-recursive grammar, shown in Figure 3. In particular, SCAN consists of all commands that can be generated from this grammar (20,910 command sequences), with their deterministic mapping into actions, as detailed by Figure 4
C + and s
V + D[1] opposite D[2]
D + turn left
C + after S
V + D[1] around D[2]
D - turn right
C + S
V + D
U - walk
+ twice
V - -U
U - look
S + V thrice
D - U left
U + run
D - U right
U + jump
Figure 3: The grammar used to generate commands in the SCAN domain. Indexing notation is used to allow infixing: read D[i] as "the i-th element directly dominated by category D". Image borrowed from Lake & Baroni (2018).
[wakk WALK
[u opposite left] [turn opposite left] [u]]
]ook] LOOK
[u opposite right] = [turn opposite right] [u]
[run]= RUN
[tum around lcft] LTURN LTURN LTURN LTURN
[jump] = JUMP
[tum around right] = RTURN RTURN RTURN RTURN
[turn left] LTURN
[u around left] = LTTURN [u]] LTURN [u] LTURN [u]] LTURN [u]
[turn right]= RTURN
around right] RTURN [u]] RTURN [u] RTURN [u]] RTURN [u]
[u =LTRN [w]
[a twice] = [2=]] [=]]]
[u right] = --TURRN [μ]
Ir thrice] = [=:] [x:] [=:]
[trm opposite left] LTUURN LTURN
[m] and 22] = [x]]] [x]]]
[turn opposite right] = RTURN RTURN
[a] after 22 [x2]] [2r
Figure The SCAN translation mapping. Double brackets denote the interpretation function translating SCAN's command (input language) into the action (output) language (which are denoted by upper-case strings. Image borrowed from Lake & Baroni (2018).
B G-LSTM DETAILS
We provide the equations for implementing our G-LSTM. Given Ct-1 (hidden state and cell- state, respectively), and e(w), (all of which of the form G R.K describe the G-LSTM cell as follows:
i = a [x Uh 8:-1 * abin):
ft = a (æ; 1bfi + 8:-1 yp)
g: tanh (x, 2b + 8:-1 * ggh);
o, (æ, thot 8:-1 bbon)
C: = f. c i, gr:
h o, tanh (ct);
where {ubyk G RK E (.f.ggo):k [i,h}} are the learnale filters of the cell. Here we have used the shorthand
f = (b) Vg G
for two functions on the group.
12


Reviews:
[{"review_id": "SJlQ3BLkqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary\n---\n\n(motivation)\nConsider SCAN, a synthetic task where setences like S1=\"jump twice and run left\" are supposed to be translated into action sequences like A1=JUMP JUMP LTURN RUN. One might replace the word \"jump\" in S1 with \"walk\" then translate to get A2=WALK WALK LTURN RUN. If instead S1 is translated into A1 and then the action JUMP is replaced with the action WALK then we should still get the same A2. Such a translation model is equivariant to permutations of \"jump\" and \"walk\".\n\nThis paper aims to\n1) define a general notion of compositionality as equivariance,\n2) build a model which is compositional in this general sense, and\n3) apply the model to SCAN.\n\n(approach - theory)\nThis work considers this kind of compositionality as equivariance to group actions. Previous work (Kondor & Trivedi, 2018) viewed convolution as equivariance to actions by translation groups. This work views language compositionality as equivariance to actions by permutation groups applied to a set of similar words (e.g. verbs in SCAN).\n\n(approach - model)\nThe paper proposes G-Embed, G-RNN, G-Attention, and G-Conv (not new) layers that are equivariant to word permuatations (e.g., switching \"jump\" and \"walk\"). It then composes these modules in a fairly standard fashion to build a new G-seq2seq model which is invariant to group actions.\n\n(experiments)\nExperiments apply a G-seq2seq model to the SCAN tasks, comparing to strong baselines. G-seq2seq requires slightly more knowledge (a set of related words like verbs) than all the baselines, but less knowledge than Lake 2019.\n1. G-seq2seq outperforms all baselines except Lake 2019 (unfair comparison) on basic compositional tasks (\"Add Jump\" and \"Around Right\").\n2. Like other models, G-seq2seq fails on the \"Length\" task, though it is still among the best performers.\n\n\nStrengths\n---\n\nThe theory of compositionality as invariance to actions by permutation groups is new, interesting, and could turn out to be significant.\n\nThe proposed models are also new, interesting, and could be significant.\n\nExperiments on SCAN verify that the proposed models work about as expected, sometimes beating strong baselines in the process.\n\n\nWeaknesses\n---\n\nIt's hard to know what the impact of this paper will be because 1) it's unclear whether this model can generalize to more useful domains and 2) the presentation may turn some readers away. While neither of these issues can really be solved, I think paper could be substantially better in both aspects. Corresponding suggestions:\n1) How expensive is this? It seems quite expensive because the representation size scales with the number of permutation of the set of words equivariance is with respect to. How will it scale to larger problems in terms of computation/memory costs (especially larger vocab sizes)? What knowledge is required for applying this method to new domains--i.e., how do I choose a set of permutation equivariant verbs in general? More discussion of these issues may help increase the impact of the paper.\n2) See next section.\n\n\nPresentation Weaknesses / Points of Confusion / Missing Details\n---\n\nTo mimic a typical decoder RNN there should be another input which copies the word \\tilde{w}_{t-1} from the previous iteration as input, somehow fused with the attention feature \\tilde{a}_t. How does the G-RNN know what the last word it generated was?\n\nThe notation $g^{-1} w$ in the first equation of section 4.1: I think $\\psi^i$ is supposed to take an integer as input but $g^{-1} w$ is a permutation applied to a function. I'm not sure how to apply permutations to functions like w and it doesn't seem like the output should be an integer in any case so I find this notation confusing.\n\nTaking a step back, I find some of the notation (e.g., previous point) a bit confusing. This makes it hard for me to get the main point. I think the idea is that equivariant models can be achieved by tracking a representation (e.g., via rows of the G-Embed matrix) for (almost?) every member of the acting group.\n\nIt may help the presentation to more frequently demonstrate the general concepts with examples, though doing so may be in conflict with the general nature of the paper's theoretical contribution. I'm sure this is a familiar tradeoff, but from my perspective the paper would probably be more impactful if the presentation leaned more on examples.\n\nEquation numbers would be a really great addition. I found it hard to reference some of the material in writing my review.\n\n\"and the use of algebraic computation\"\n* This seems specific to the chosen example whereas the rest of the sentence is trying to be general.\n\n\nSuggestions\n---\n\n* This seems related to [1], which uses group theory to define a notion of disentangled representation similar to compositionality. That may inspire future work and would be useful to mention in the related work.\n\n* Why didn't performance on SCAN get to 100%? It would be useful to spend some time addressing points of failure for the model other than compositionality.\n\n* The G-RNN doesn't have a bias. It's not necessary, but it may be interesting to describe why this design choice was made.\n\n\n[1]: Higgins, Irina et al. “Towards a Definition of Disentangled Representations.” ArXiv abs/1812.02230 (2018): n. pag. \n\n\nPreliminary Evaluation\n---\n\nQuality: The theoretical contributions make sense and the experiments show they lead to useful models.\nClarity: The technical parts of the paper are somewhat unclear, but the rest of the paper is well written.\nSignificance: As discussed in the Weaknesses section this could turn out to be very significant or not significant at all, but that's true for a lot of good research.\nOriginality: The general notion of equivariant neural networks and good performance on SCAN are novel.\n\nOverall, this is a very clear accept.\n\nPost-Rebuttal Update\n---\n\nThere was a lot of agreement between reviewers, though we came to slightly different conclusions about ratings. Though there is significant uncertainty about the impact of this work, I still think 8: Accept is the most appropriate rating. Overall, the other reviews and author responses only increased my confidence that this paper should be accepted.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyl3-5ICFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an architecture that captures equivariance to certain transformations that happen in text, like synonym words and some simple transformation over word order.  \n\n* General comments: \n  \nIncreasing compositional generalization using equivariance is a very interesting idea. Sections 1-3 are well written and the solution of modeling the translation function as a G-equivariant function is well motivated. \n\nSection 4 is far less clear. In its current form, it is very hard to understand the model construction as well as the design choices. This section should be significantly improved in order for me to increase my score. A direct by-product of the confusing writing is that the experiments cannot  be reproduced.\n\nThe experiments show improvement in one out of four tasks, where the single phrase “Around right” is held from the training set. There are no examples, not qualitative analysis, no ablation experiments. Overall, more evidence needed to convince that the approach is useful. In addition to deeper error analysis, the authors can hold out other phrases (e.g., “around left”, and many others). \n\n* Specific comments which I hope the authors address:\n\n1. To the best of my understanding, the authors do not explicitly specify the group G that they want to be invariant to. Is it a product of a few cyclic groups? (a cycle for each set of words that are interchangeable?)\n\n2. The authors suggest using G-convolution, i.e. the group convolution on G. This is in contrast to the (arguably) more popular choice of using linear layers that are G-equivariant (as in, for example,  deep sets (Zaheer et al. 2017), Deep Models of Interactions Across Sets (Hartford et al. 2018),Universal invariant and equivariant graph neural networks (Keriven and Peyré ) and in general convolutional layers for learning images).\nI have several questions regarding this choice:\n2a. Can the authors discuss the differences/advantages of this approach over the approach mentioned above? It seems like the approach mentioned above will be more efficient (as there is no need to sum over all group elements)\n2b. In order to use G-convolution, one has to use functions defined on G. Can the authors explain how they look on the input words as functions on G?\n2c. How is the G-Conv actually implemented? \n2d. Can the authors provide some intuition to what this operator does? \n\n3. Is the whole model G-equivariant? The authors might want to clearly state this. To the best of my understanding, this is the main motivation of this construction.\n\n4. It might be helpful for readers that are not familiar with deep learning for NLP tasks to provide a visualization of the full model (can be added to the appendix)\n\n5. Why are words represented as infinite one-hot sequences? Don’t we assume a finite vocabulary? This is pretty confusing.\n\n6. As a part of the G-RNN the authors apply a G-conv to the state h_{t-1}. What is the dimension of this hidden state? How does G act on it? \n\n7. Please explicitly state the dimensions of each input/output/parameter in the network (this can be combined with the illustration above illustration)\n\n* Minor comments:\n\nSection 4.1 pointwise activation are in general equivariant only to permutation representations\nPage 2 - typo - ‘all short’-> ‘fall short’ \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkgCK-1RKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work focuses on learning equivariant representations and functions over input/output words for the purposes of SCAN task. Basically, the focus is on local equivariances (over vocabulary) such that the effect of replacing and action verb like RUN in the input with the verb JUMP causes a similar change in the output. However, effects requiring global equivariances like learning relationship between \"twice\" and \"thrice\", or learning relationships between different kinds of conjunctions are not handled in this work. For learning equivariant functions over vocabulary, group convolutions are used at each step over vocabulary items in both the sequence encoder and decoder.  The results on SCAN task are impressive for verb replacement based experiments and improve over other relevant baselines. Also, improvement is shown on another word replacement task (\"around right\"), which requires learning corresponding substitutions in output based on the word changes in the input. As expected, for experiments that require global equivariances or no equivariance (simple, length), the difference ion performance is not very pronounced over other baselines.\nWhile this paper does show that modelling effects of word substitution can be handled by the locally equivariant functions, it still cannot account for more complex generalization phenomena which are likely to be much more prevalent especially for domains dealing with natural language that are other than SCAN. Therefor, I think the applicability of the proposed equivariant architectures is rather limited if interesting.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an equivariant sequence-to-sequence model for dealing with compositionality of language. They show these models are better at SCAN tasks.

Reviewers expressed two major concerns:
1) Limited clarity of section 4 which makes the paper difficult to understand.
2) Whether this could generalize to more complex types of compositionality.

Authors responded by revising Section 4 and answering the question of generalization. While the reviewers are not 100% satisfied, they agree there is enough novel contribution in this paper. 

I thank the authors for submitting and look forward to seeing a clearer revision in the conference.
Decision: Accept (Poster)