user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Permutation Equivariant Models for Compositional Generalization in Language

Paper content:
Humans understand novel sentences by composing meanings and roles of core language components.
In contrast, neural network models for natural language modeling fail when such compositional generalization is required.
The main contribution of this paper is to hypothesize that language compositionality is a form of group-equivariance.
Based on this hypothesis, we propose a set of tools for constructing equivariant sequence-to-sequence models.
Throughout a variety of experiments on the SCAN tasks, we analyze the behavior of existing models under the lens of equivariance, and demonstrate that our equivariant architecture is able to achieve the type compositional generalization required in human language understanding.
When using language, humans recombine known concepts to understand novel sentences.
For instance, if one understands the meaning of "run", "jump", and "jump twice", then one understands the meaning of "run twice", even if such sentence was never heard before.
This relies on the notion of language compositionality, which states that the meaning of a sentence ("jump twice") is to be obtained by the meaning of its constituents (e.g. the verb "jump" and the quantifying adverb "twice") and the use of algebraic computation (a verb combined with a quantifying adverb m results in doing that verb m times) (Kratzer & Heim, 1998 ).
In the realm of machines, deep learning has achieved unprecedented results in language modeling tasks (Bahdanau et al., 2014; Vaswani et al., 2017) .
However, these models are sample inefficient, and do not generalize to examples that require the use of language compositionality (Lake & Loula et al., 2018; Dessì & Baroni, 2019) .
This result suggests that deep language models fail to leverage compositionality; a failure remaining to this day a roadblock towards true natural language understanding.
Focusing on this issue, Lake & proposed the Simplified version of the CommAI Navigation (SCAN), a dataset to benchmark the compositional generalization capabilities of state-ofthe-art sequence-to-sequence (seq2seq) translation models (Sutskever et al., 2014; Bahdanau et al., 2014) .
In a nutshell, the SCAN dataset contains compositional navigation commands such as JUMP TWICE AFTER RUN LEFT, to be translated into the sequence of actions LTURN RUN JUMP JUMP.
Using SCAN, demonstrated that seq2seq models fail spectacularly at tasks requiring the use of language compositionality.
Following our introductory example, models trained on the three commands JUMP, RUN and JUMP TWICE fail to generalize to RUN TWICE.
Most recently, Dessì & Baroni (2019) showed that architectures based on temporal convolutions meet the same fate.
SCAN did not only reveal the lack of compositionality in language models, but it also became the blueprint to build novel language models able to handle language compositionality.
On the one hand, Russin et al. (2019) proposed a seq2seq model where semantic and syntactic information are represented separately, in a hope that such disentanglement would elicit compositional rules.
However, their model was not able to solve all of the compositional tasks comprising SCAN.
On the other hand, Lake (2019) introduced a meta-learning approach with excellent performance in multiple SCAN tasks.
However, their method requires substantial amounts of additional supervision, and a complex meta-learning procedure hand-engineered for each task.
In this paper, we take a holistic look at the problem and connect language compositionality in SCAN to the disparate literature in models equivariant to certain group symmetries (Kondor, 2008; Cohen & Welling, 2016; Kondor & Trivedi, 2018) .
Interesting links have recently been proposed between group symmetries and the areas of causality (Arjovsky et al., 2019) and disentangled representation learning (Higgins et al., 2018) , and this work proceeds in a similar fashion.
In particular, the main contribution of this work is not to chase performance numbers, but to put forward the novel hypothesis that language compositionality can be understood as a form of group-equivariance (Section 3).
To sustain our hypothesis, we provide tools to construct seq2seq models equivariant when the group symmetries are known (Section 4), and demonstrate that these models solve all SCAN tasks, except length generalization (Section 6).
This work has introduced hypothesis linking between group equivariance and compositional generalization in language.
Motivated by this hypothesis, we have proposed an equivariant seq2seq translation model, which achieves state-of-the-art performance on a variety of SCAN tasks.
Our work has several points for improvement.
Most importantly, our model requires knowing the permutation symmetries of interest, to be provided by some domain expert.
While this is simple to do in the synthetic language of SCAN, it may prove more difficult in real-world tasks.
We propose three directions to attack this problem.
(i) Group words by their parts-of-speech (e.g., nouns, verbs, etc.), which can be done automatically by standard part-of-speech taggers (Màrquez & Rodríguez, 1998) ;
(ii) Learn such groupings of words from corpora, for example using the recent work of Andreas (2019);
(iii) Most appealingly, parameterize the symmetry group and learn operations end-to-end while enforcing the group structure.
For permutation symmetries, the group elements can be parameterized by permutation matrices, and learned from data (Lyu et al., 2019) .
Our preliminary work in this direction hints that this is a fruitful avenue for future research.
A further consideration to address is that of computational overhead.
In particular, for the convolutional form we use in this work (Definition 4), computational complexity scales linearly with the size of the group, O(|G|).
This arises from the need to sum over group elements when the representation is a function on G, and may be prohibitive when considering large groups.
One way of addressing this issue when large symmetry groups are of interest is to consider more efficient computational layers for permutation equivariance (e.g Zaheer et al., 2017; .
These methods incur less computational overhead at the cost of restricting the layer capacity.
Another interesting option for future research is to consider sub-sampling group elements when performing the summation in Definition 4, which requires further consideration of the consequences of doing so.
Another exciting direction for future research is to consider global equivariances.
Many operations of interest, e.g. groups operating directly on parse trees, can only be expressed as global equivariances.
Modeling these equivariances holds exciting possibilities for capturing non-trivial symmetries in language tasks, but also requires more sophisticated machinery than is proposed in this work.
Finally, in further theoretical work, we would like to explore the relation between our equivariance framework and the idea of compositionality in formal semantics (Kratzer & Heim, 1998) .
On the one hand, the classic idea of compositionality as an isomorphism between syntax and semantics is intuitively related to the notion of group equivariance.
On the other hand, as shown by the failures at the length generalization example, it is still unclear how to apply our ideas to more sophisticated forms of permutation, such as those involving grammatical phrases rather than words.
This would also require to extend our approach to account for the context-sensitivity that pervades linguistic composition (c.f., the natural interpretation of "run" in "run the marathon" vs. "run the code").
A DETAILS ON THE SCAN DATASET SCAN is composed from a non-recursive grammar, as shown in Figure 3 .
In particular, SCAN consists of all commands that can be generated from this grammar (20,910 command sequences), with their deterministic mapping into actions, as detailed by Figure 4

assistant: text: This work focuses on learning locally equivariant representations and functions over input/output words for the purposes of SCAN task.