Permutation Equivariant Models for Compositional Generalization in Language

Humans understand novel sentences by composing meanings and roles of core language components . In contrast , neural network models for natural language modeling fail when such compositional generalization is required . The main contribution of this paper is to hypothesize that language compositionality is a form of group-equivariance . Based on this hypothesis , we propose a set of tools for constructing equivariant sequence-to-sequence models . Throughout a variety of experiments on the SCAN tasks , we analyze the behavior of existing models under the lens of equivariance , and demonstrate that our equivariant architecture is able to achieve the type compositional generalization required in human language understanding . 1 INTRODUCTION . When using language , humans recombine known concepts to understand novel sentences . For instance , if one understands the meaning of “ run ” , “ jump ” , and “ jump twice ” , then one understands the meaning of “ run twice ” , even if such sentence was never heard before . This relies on the notion of language compositionality , which states that the meaning of a sentence ( “ jump twice ” ) is to be obtained by the meaning of its constituents ( e.g . the verb “ jump '' and the quantifying adverb “ twice '' ) and the use of algebraic computation ( a verb combined with a quantifying adverb m results in doing that verb m times ) ( Kratzer & Heim , 1998 ) . In the realm of machines , deep learning has achieved unprecedented results in language modeling tasks ( Bahdanau et al. , 2015 ; Vaswani et al. , 2017 ) . However , these models are sample inefficient , and do not generalize to examples that require the use of language compositionality ( Lake & Baroni , 2018 ; Loula et al. , 2018 ; Dessì & Baroni , 2019 ) . This result suggests that deep language models fail to leverage compositionality ; a failure remaining to this day a roadblock towards true natural language understanding . Focusing on this issue , Lake & Baroni ( 2018 ) proposed the Simplified version of the CommAI Navigation ( SCAN ) , a dataset to benchmark the compositional generalization capabilities of state-ofthe-art sequence-to-sequence ( seq2seq ) translation models ( Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ) . In a nutshell , the SCAN dataset contains compositional navigation commands such as JUMP TWICE AFTER RUN LEFT , to be translated into the sequence of actions LTURN RUN JUMP JUMP . Using SCAN , Lake & Baroni ( 2018 ) demonstrated that seq2seq models fail spectacularly at tasks requiring the use of language compositionality . Following our introductory example , models trained on the three commands JUMP , RUN and JUMP TWICE fail to generalize to RUN TWICE . Most recently , Dessì & Baroni ( 2019 ) showed that architectures based on temporal convolutions meet the same fate . SCAN did not only reveal the lack of compositionality in language models , but it also became the blueprint to build novel language models able to handle language compositionality . On the one hand , Russin et al . ( 2019 ) proposed a seq2seq model where semantic and syntactic information are represented separately , in a hope that such disentanglement would elicit compositional rules . However , their model was not able to solve all of the compositional tasks comprising SCAN . On the other hand , Lake ( 2019 ) introduced a meta-learning approach with excellent performance in multiple ∗Work conducted during an internship at Facebook AI Research SCAN tasks . However , their method requires substantial amounts of additional supervision , and a complex meta-learning procedure hand-engineered for each task . In this paper , we take a holistic look at the problem and connect language compositionality in SCAN to the disparate literature in models equivariant to certain group symmetries ( Kondor , 2008 ; Cohen & Welling , 2016 ; Ravanbakhsh et al. , 2017 ; Kondor & Trivedi , 2018 ) . Interesting links have recently been proposed between group symmetries and the areas of causality ( Arjovsky et al. , 2019 ) and disentangled representation learning ( Higgins et al. , 2018 ) , and this work proceeds in a similar fashion . In particular , the main contribution of this work is not to chase performance numbers , but to put forward the novel hypothesis that language compositionality can be understood as a form of group-equivariance ( Section 3 ) . To sustain our hypothesis , we provide tools to construct seq2seq models equivariant when the group symmetries are known ( Section 4 ) , and demonstrate that these models solve all SCAN tasks , except length generalization ( Section 6 ) .1 2 THE SCAN COMPOSITIONAL TASKS . The purpose of the Simplified version of the CommAI Navigation ( SCAN ) tasks ( Lake & Baroni , 2018 ) is to benchmark the abilities of machine translation models for compositional generalization . Following prior literature ( Lake & Baroni , 2018 ; Baroni , 2019 ; Russin et al. , 2019 ; Andreas , 2019 ) , compositional generalization is understood as the ability to translate novel families of sentences , when this requires leveraging the compositional structure in language . The SCAN dataset contains compositional navigation commands in English ( the input-language ) paired with a desired action sequence ( the output-language ) . For instance , the input-language sentence JUMP TWICE AND RUN LEFT is paired to the output-language actions sequence JUMP JUMP LTURN RUN . The rest of our exposition uses SMALL CAPS to denote examples in the input-language , and LARGE CAPS to denote examples in the output-language . Appendix A contains a full description of the grammar generating the SCAN language . To evaluate the compositional generalization abilities of sequence-to-sequence ( seq2seq ) machine translation models ( Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ) , Lake & Baroni ( 2018 ) proposes four main tasks based on SCAN : 1 . Simple task : data pairs are randomly split into training and test sets . No compositional generalization is required . 2 . Add jump task : the only command in the training set containing the verb JUMP is the command JUMP . All commands not containing JUMP are in the training set ( for instance , RUN TWICE , and WALK RIGHT THRICE AND LOOK LEFT ) . The test set contains all commands containing JUMP ( for instance , JUMP TWICE , and RUN LEFT AND JUMP RIGHT ) . To succeed in this task , models must learn that JUMP is a verb , and that any verb can be composed with an adverbial number to be repeated a number of times . 3 . Around right task : the phrase AROUND RIGHT is held out from the training set ; however , both AROUND and RIGHT are shown in all other contexts ( for example , AROUND LEFT or OPPOSITE RIGHT ) . To succeed at this task , models must learn that both RIGHT and LEFT are directions , and can be combined with AROUND and OPPOSITE . 4 . Length generalization task : the training set contains pairs such that the length of the action sequence in the output-language is shorter than 24 actions . The test set contains all pairs with action sequences of a length greater or equal than 24 actions . The type of compositional ability required to succeed at this task is more difficult to sketch out , as we discuss in Section 6.2 . Lake & Baroni ( 2018 ) use these four tasks to demonstrate that state-of-the-art seq2seq translation models ( Bahdanau et al. , 2015 ) succeed at Simple task , but fail at the other three tasks requiring compositional generalization . Convolutional architectures ( Dessì & Baroni , 2019 ) achieve only slightly better performance , and state-of-the-art methods specially developed to address SCAN tasks fall short from the best achievable performance ( Russin et al. , 2019 ) , or call for substantial amounts of additional supervision ( Lake , 2019 ) . 1Code available at https : //github.com/facebookresearch/Permutation-Equivariant-Seq2Seq In the following , we take a holistic look at the language compositionality problems in SCAN , and highlight their connection to equivariant maps in group theory . 3 SCAN COMPOSITIONALITY AS GROUP EQUIVARIANCE . This section puts forward the hypothesis that : Models achieving the compositional generalization required in certain SCAN tasks are equivariant with respect to permutation group operations2 in the input and output languages . To unfold the meaning of our hypothesis , we must revisit some basic concepts in group theory . A discrete group G is a set of elements { g1 , . . . , g|G| } , equipped with a binary group operation “ · ” satisfying the four group axioms ( closure , associativity , identity , and invertibility ) . The sequel focuses on permutation groups G , whose elements are permutations of a set X , and whose binary group operation composes the permutations contained in G. The set of all permutations of X is a group , but not all subsets of permutations of X satisfy the four group axioms , and therefore they do not form a group . For each element g ∈ G , we define the group operation Tg : X → X as the map applying the permutation g to the element x ∈ X , to obtain Tgx . Armed with these definitions , we are ready to introduce the main object of study in this paper : equivariant maps . Definition 1 ( Equivariant map ) . Let X and Y be two sets . Let G be a group whose group operation on X is denoted by Tg : X → X , and whose group operation on Y is denoted by T ′g : Y → Y . Then , Φ : X → Y is an equivariant map if and only if Φ ( Tgx ) = T ′gΦ ( x ) for all x ∈ X and g ∈ G. The operation groups ( Tg , T ′g ) defined above operate on entire sequences , an enormous space when we consider those sequences to be language sentences . In the following two definitions , we relax group operations and equivariant maps to operate at a word level . Definition 2 ( Local group operations ) . Let X be a set of sequences ( or sentences ) , where each sequence x ∈ X contains elements xi ∈ V from a vocabulary set V , for all xi ∈ x . Let G be a group with associated group operation Tg : X → X . Then , we say that Tg is a local group operation if there exists a group operation Tgw : V × V such that Tgx = ( Tgwx1 , . . . , TgwxLx ) for all x ∈ X . When understanding sequences as language sentences , the group operation Tgw would be a permutation of the words from the language vocabulary . Such operation can be implemented in terms of a permutation matrix , a |V| × |V| matrix with zero/one entries where each row and each column sum to one . Finally , we leverage the definition of local group operations to define locally equivariant maps . Definition 3 ( Locally equivariant map ) . Let X and Y be two sets of sequences . Let G be a group whose group operation on X is local in its vocabulary , denoted by Tg : X → X , and whose group operation on Y is local in its vocabulary and denoted by T ′g : Y × Y . Then , we say that Φ : X → Y is an equivariant map if and only if Φ ( Tgx ) = T ′gΦ ( x ) for all x ∈ X and g ∈ G. x x′ y y′ Φ Tg T ′g Φ ( a ) WALK LEFT AND RUN JUMP LEFT AND RUN LTURN WALK RUN LTURN JUMP RUN ( b ) RUN LEFT AND WALK RUN LEFT AFTER WALK ( 1 ) ︷ ︸︸ ︷ LTURN RUN ( 2 ) ︷ ︸︸ ︷ WALK ( 2 ) ︷ ︸︸ ︷ WALK ( 1 ) ︷ ︸︸ ︷ LTURN RUN ( c ) Figure 1 : ( a ) Commutative diagram for equivariance . ( b ) Local equivariance enables generalization to verb replacement in SCAN . ( c ) Local equivariance does not enable generalization to conjunction replacement in SCAN . 2Standard terminology in group theory denotes the elements of a group “ to act on ” elements of a set . However , as the output language in SCAN is in navigation “ actions ” , we use the “ operate ” terminology for group elements to avoid ambiguity . Now , how do equivariances and local equivariances manifest themselves in the world of SCAN ? To assist our examples , the commutative diagram in Figure 1a summarizes the group theory notations introduced so far . In Figure 1b and Figure 1c , we parallel these notations to two different examples of compositional skills required to solve SCAN : verb and conjunction replacement . In the SCAN domain , X is the set of sentences in the input-language , and Y is the set of sentences in the outputlanguage . Furthermore , let Φ be a locally equivariant SCAN translation model , and let G be a group with associated local group operations that permutes words in the input- and output- languages . On the one hand , we observe in Figure 1b that local equivariance enables compositional generalization in the case of verb replacement . This is because replacing one verb in the input-language can be implemented in terms of a local group operation . In turn , this input-verb replacement corresponds deterministically to a second local group operation that replaces the corresponding verb in the output-language . The same would apply to a SCAN task where we are interested in generalizing to the replacement of LEFT and RIGHT . As such , a translation model Φ with these compositional generalization capabilities must be locally equivariant . On the other hand , we observe in Figure 1c that local equivariance is insufficient to enable compositional generalization in the case of conjunction replacement . This is because no local group operation in the output-language would be able to implement the necessary changes induced by the replacement of AND by AFTER in the input-language . In such cases , we refer to the equivariance as global equivariance . In particular , we can see how blocks of multiple words in the output-language swap their relative location . Local equivariances are also insufficient to enable compositional generalization in the Length generalization SCAN task and we elaborate on this in Section 6.2 . In the following section , we propose a set of tools to implement equivariant seq2seq translation models , and propose a particular architecture with which we conduct our experiments .

Summary: This work focuses on learning equivariant representations and functions over input/output words for the purposes of SCAN task. Basically, the focus is on local equivariances (over vocabulary) such that the effect of replacing and action verb like RUN in the input with the verb JUMP causes a similar change in the output. However, effects requiring global equivariances like learning relationship between "twice" and "thrice", or learning relationships between different kinds of conjunctions are not handled in this work. For learning equivariant functions over vocabulary, group convolutions are used at each step over vocabulary items in both the sequence encoder and decoder.  The results on SCAN task are impressive for verb replacement based experiments and improve over other relevant baselines. Also, improvement is shown on another word replacement task ("around right"), which requires learning corresponding substitutions in output based on the word changes in the input. As expected, for experiments that require global equivariances or no equivariance (simple, length), the difference ion performance is not very pronounced over other baselines.