Symbol-Shift Equivariant Neural Networks

Neural networks have been shown to have poor compositionality abilities : while they can produce sophisticated output given sufficient data , they perform patchy generalization and fail to generalize to new symbols ( e.g . switching a name in a sentence by a less frequent one or one not seen yet ) . In this paper , we define a class of models whose outputs are equivariant to entity permutations ( an analog being convolution networks whose outputs are invariant through translation ) without requiring to specify or detect entities in a pre-processing step . We then show how two question-answering models can be made robust to entity permutation using a novel differentiable hybrid semantic-symbolic representation . The benefits of this approach are demonstrated on a set of synthetic NLP tasks where sample complexity and generalization are significantly improved even allowing models to generalize to words that are never seen in the training set . When using only 1K training examples for bAbi , we obtain a test error of 1.8 % and fail only one task while the best results reported so far obtained an error of 9.9 % and failed 7 tasks . 1 INTRODUCTION . Previous work have shown how neural networks fail to generalize to new symbols ( Lake & Baroni , 2018 ; Sinha et al. , 2019 ; Hupkes et al. , 2019 ) . In particular , Lake & Baroni ( 2018 ) showed that seq2seq models are able to perfectly learn a set of rules given enough data , however they fail to generalize these learned rules to new symbols . 4 ( original ) 100 1000 10000 Number of names 0 20 40 60 80 100 Te st e rro r 5 % error limit MN SMN TPR STPR tensor product RNN ( TPR ) ( Schlag & Schmidhuber , 2018 ) dramatically drops as the number of names increases in contrast to their symbolic counter-part SMN and STPR proposed in this paper . Both models reach low error well below the 5 % limit even when the number of names and vocabulary become considerably larger than the original task . The main contribution of this paper is the proposal of a hybrid semantic/symbolic representation that is equivariant to entity permutation . The main advantage and novelty of our approach is that entities are not required to be identified in advance as we rely solely on differentiation to determine whether a word acts like an entity . We show how to extend two question-answering models to handle this hybrid representation and demonstrate in extensive experiments the benefit of such an approach : the sample-complexity is significantly improved , better compositionality is obtained and symbolic models reach better accuracy on the studied tasks in particular when being trained with less training data . The paper starts by reviewing related works , we then introduce formally what it means to permute entities . We then define layers that are robust to such perturbation and show how two recent questionanswering models can be adapted in this context . Finally , experiments are conducted to assess the benefits of our method . 2 RELATED WORKS . Improving compositionality of neural networks has been an important on-going effort in the past years . The SCAN dataset proposed from Lake & Baroni ( 2018 ) initially showed how standard neural networks baselines can fail to generalize to new symbols when learning a set of artificially constructed rules . Several approaches were proposed to solve this issue . For instance , Lake ( 2019 ) designed meta-learning episodes that led the model to solve the task , ( Nye et al. , 2020 ) showed how one could infer symbolic neural programs with a similar meta-learning procedure . Alternatively , Gordon et al . ( 2020 ) proposed to design an equivariant model ( a model whose latent representations are unchanged when permuting symbols ) . A common limit of those approaches is that they require specifying which words are symbols in advance ( Lake ( 2019 ) ; Nye et al . ( 2020 ) also requires substantial amount of supervision and designing meta-episodes ) . An exception is Russin et al . ( 2019 ) which proposed to decompose syntax and semantic for SCAN . None of those approaches can generalize to arbitrary large amount of entities or entities not seen in the training as the one we propose . The problem of compositionality becomes much easier if symbols ( or entities ) are detected beforehand . For instance , Li et al . ( 2015 ) showed that replacing entities by dedicated token placeholders leads to significant improvement in question answering . The same approach has also been applied in Machine Translation and Data to Text generation Luong et al . ( 2015 ) ; Serban et al . ( 2016 ) ; Lebret et al . ( 2016 ) , to enable sequence to sequence models to generalize to unseen words at inference time . While specifying entities in advance ( or detecting them in a pre-processing step with Named-entity recognition ( Marsh & Perzanowski , 1998 ) ) before applying a model may give compositionality , we would clearly want instead models to be able to infer automatically whether a word should behave as a symbol or not . While positional encoding ( Graves et al. , 2014 ; Vaswani et al. , 2017 ) may give some compositionality - as it allows to reason over positions - this solution is not practical for language as inter-word distances are not fixed . For instance the distance between a noun and its verb varies and positional embedding is not enough to achieve compositionality ( Hupkes et al. , 2019 ) . An interesting line of research have been the study of equivariant models whose representations are invariant ( or equivariant ) to symmetries present in the data ( Zaheer et al. , 2017 ; Ravanbakhsh et al. , 2017 ) . Adding invariance to data symmetries has been theoretically shown to drastically reduces sample complexity ( Sannai & Imaizumi , 2019 ) . For instance , convolution neural networks require significantly less training data and achieve much better performance than a MLP as they are invariant to image translation . Gordon et al . ( 2020 ) proposed the first NLP model provably capable of handling symmetries between symbols albeit requiring the need to specify such symmetries in advance . Tensor product representation ( TPR ) Smolensky ( 1990 ) allows to stores complex relations between value and variables with distributed representations and offer some compositionality . Recently , Schlag & Schmidhuber ( 2018 ) proposed an architecture able to learn TPR parameters by differentiation and obtained state-of-the-art results for bAbi at the time of publishing . However , the compositionality of the proposed approach is limited ( as shown in Fig . 1 ) by the fact that every entity needs to be seen sufficiently many times so that a proper entity vector is found , in addition the model has been shown to learn orthogonal representation for entities which requires as many hidden dimensions as the total number of entities . Finally , the VARS approach ( Vankov & Bowers , 2020 ) consists in outputting a one-hot vector representing a symbolic variable that is randomly assigned to different positions during training to enforce compositionality . While this approach grants some compositionality , the approach is limited as one must draw symbol permutations so that each object is seen in all possible one-hot values . In addition , one must specify in advance which object or word behaves as a symbol and the method only support symbolic output and can not represent symbolic inputs nor perform computation with hybrid representation as we propose ( combining semantic and symbolic representation ) . 3 SYMBOL-SHIFT EQUIVARIANCE . When we learn to answer “ John ” given a specific context we would like to be able to answer “ Sasha ” if both names were permuted in the context . In what follows , we introduce the notion of symbolshift equivariance : e.g . a condition restricting possible permutations as some permutations may perturb the sentence grammar ( permuting “ John ” by “ why ” ) or cause ambiguity ( permuting “ John ” by “ Mary ” if the question involves a gender ) . We assume all words are taken from a vocabulary V which is a discrete set of n words . We are interested in providing an answer a ∈ V given a context consisting in a question q = [ q1 , . . . , qnq ] ∈ V nq and a list of sentences ( or stories ) x = [ x1 , . . . , xT ] with xi = [ xi1 , . . . , xini ] ∈ V ni . We denote X = ( x , q ) and Φ ( X ) = a the function that predicts the answer a ∈ V given the context X = ( x , q ) ∈ X . Let Γ : V → V a word permutation . Given a sequence [ y1 , . . . , yn ] , we define Γ ( y ) = [ Γ ( y1 ) , . . . , Γ ( yn ) ] where the permutation is applied to each word in the sequence , similarly , Γ ( X ) = ( [ Γ ( x1 ) , . . . , Γ ( xT ) ] , Γ ( q ) ) . Assuming each word has an associated vector parameter , we say that a permutation is a symbol-shift if it preserves vectors parameters . For instance in Figure 2 , a map permuting “ John ” with “ Sasha ” is a symbol-shift as both words share the same parameters but a map permuting “ John ” and “ lemon ” is not . Formally , assuming each word i ≤ n has an associated set of parameters vi ∈ RD , we say that a permutation Γ : V → V is a symbol-shift if vi = vΓ ( i ) for all i ≤ n. We are now ready to define symbol-shift equivariance . A critical advantage is that we do not need to specify symmetries in advance between entities , as we instead rely on vector semantics whose embeddings will be learned end-to-end . Definition 1 . Let Φ : X → V a function mapping a context to an answer . We say that Φ is symbolshift equivariant if for any symbol-shift Γ and for any X ∈ X , Φ ( Γ ( X ) ) = Γ ( Φ ( X ) ) 4 SYMBOLIC QUESTION ANSWERING . In this section , we show how to define symbol-shift-equivariant models . The main idea consists of concatenating two representations , a standard semantic representation in Rd as well as a symbolic representation in Rm where m denotes the number of distinct words in the stories and question . The symbolic representation will be made such that for i ≤ m , the i-th component of the symbolic representation corresponds to the i-th word appearing in the context . For instance in Figure 3 , there arem = 5 words in the context and “ apple ” is the fourth word by order of appearance so its symbolic embedding is the fourth one-hot vector . The model symbolic output has larger probability for the fourth word of the context which is dereferenced to “ Apple ” . We now describe formally how the symbolic representations are constructed and how we perform linear transformation and projection back to the original vocabulary . Finally , we derive the symbolic counter-part of Memory-Networks and TPR models that will be proved to be symbol-shift equivariant . Mapping words into and from symbolic representations . In each input example , the set of words present in the stories x and the question q is denoted as CX = { q } ∪ { { xi } , i ≤ T } and we letm = |CX | be the number of distinct words in the context . To project words to its symbolic representation Rm , we represent each unique word by a one-hot vector representing the position of its first appearance , using the bijection ϕX : CX → [ 1 , m ] 1 . To dereference a symbolic representation in Rm back to its vocabulary id , we define the matrixBϕ ∈ Rn×m that maps one-hot vectors of symbolic representations to one-hot vectors in Rn representing the word id in the vocabulary as shown in Fig . 3 , such that Bϕemj = e n ϕ−1 ( j ) for j ≤ m where ekl ∈ Rk denotes the l-th one-hot vector in Rk 2 . Note given a one-hot symbolic representation p̃ ∈ Rm , the i-th coordinates of Bϕp̃ ∈ Rn is given by : [ Bϕp̃ ] i = { p̃ϕ ( i ) if i ∈ C 0 else ( 1 ) hence Bϕ allows to dereference a symbolic representation p̃ ∈ Rm to a word vector Bϕp̃ ∈ Rn . Hybrid semantic-symbolic embeddings . We embed words with the concatenation of a standard semantic word embedding as well as a symbolic embedding respectively parametrized byA ∈ Rd×n and α ∈ [ 0 , 1 ] n. The semantic embedding maps a word x ∈ [ 1 , n ] to Aex ∈ Rd . While the symbolic embedding of x consists in the one-hot vector of the order of appearance of the word x in its context multiplied by a learned parameter . More precisely , it is defined as αxeϕ ( x ) ∈ Rm , where αx is an output of a sigmoid unit on a learnable parameter , i.e . 0 < αx < 1 , that indicates how much each word should behave as a symbol 3 , and eϕ ( x ) is the one-hot vector of the order of appearance of the word x in its context . The final embedding of a word of x then consists in the concatenation of the semantic and symbolic parts : x 7→ Aex ⊕ αxeϕ ( x ) ∈ Rd+m , ( 2 ) where ⊕ denotes the concatenation operator . Note that all parameters are differentiable allowing the model to learn both word semantic and how much each word should behave as a symbol . The symbolic part will be shown to allow the model to be robust to symbol permutation while being able to represent an arbitrary number of symbols and generalize to new ones . For instance in Fig . 3 , one can see that permuting “ John ” by “ Sasha ” would not change embeddings ( as long as both name share the same parameters ) as both words will still appear in the same order in the context . Symbolic projection . Given an internal state h = hsem ⊕ hsym ∈ Rd+m , we interpret it as a distribution on the vocabulary p ∈ Rn with psem = softmax ( Bhsem ) psym = Bϕsoftmax ( hsym ) ( 3 ) p = β psem + ( 1− β ) psym ( 4 ) where B ∈ Rn×d and β ∈ [ 0 , 1 ] are parameters to learn . The final distribution is a mixture of two distributions psem and psym . The first one psem is seen as a semantic output as it increases answer probability of a word if its semantic embedding is closer 1From now on , we drop the subscript notation when there is no ambiguity and denote C , ϕ 2we later omit superscript dimension as they will always be implicitly defined 3Note that if αx = 0 for all word x , the model is reduced to a standard semantic-only model to the semantic state hsem . The second one psym interprets hsym as probabilities of words from the context using Bϕ to dereference positions . Indeed , denote p̃ = softmax ( hsym ) ∈ Rm and using Eq . 1 , we get that the i-th coefficient of psym is given by [ psym ] i = [ Bϕp̃ ] i = { p̃ϕ ( i ) if i ∈ C 0 else ( 5 ) Symbolic transformation . We perform a linear transformation of an internal symbolic representation h = hsem ⊕ hsym ∈ Rd+m with : hsem ⊕ hsym 7→Whsem ⊕ ( λI + γ11T ) hsym ∈ Rd+m , ( 6 ) where W ∈ Rd×d , λ ∈ R , γ ∈ R are parameters to learn , I ∈ Rm×m is the identity matrix and 1 = [ 1 , . . . , 1 ] T ∈ Rm×1 . The linear transformation for the symbolic part is taken from Zaheer et al . ( 2017 ) where it was shown to be the unique form of a linear parametric equivariant function . In our case , the symbolic transformation is invariant to permutation of symbolic coordinates , allowing the model to be independent from the choice of a particular bijection ϕ .

Summary: This paper proposes a new type of models that are equivariant to entity permutations, which is an important criterion to build language models that can easily generalize to new entities. The authors modified a Memory-Network and a Third-order tensor product RNN to make them symbolic-shit invariant. The new models were evaluated and compared on the 20 bAbi tasks. Results show that the symbolic versions of the models yield better performance than the original ones.