Symmetry and Systematicity

1 INTRODUCTION . Convolution ( LeCun & Bengio , 1998 ) has been an incredibly effective element in making Deep Learning successful . Applying the same set of filters across all positions in an image captures an important characteristic of the processes that generate the objects depicted in them , namely the translational symmetry of the underlying laws of nature . Given the impact of these architectures , researchers are increasingly interested in finding approaches that can be used to exploit further symmetries ( Cohen & Welling , 2016 ; Higgins et al. , 2018 ) , such as rotation or scale . Here , we will investigate symmetries relevant to symbolic processing . We show that incorporating symmetries derived from symbolic processes into neural architectures allows them to generalise more robustly on tasks that require handling elements and structures that were not seen at training time . Specifically , we construct convolution-based models that outperform standard approaches on the rule learning task of Marcus et al . ( 1999 ) , a simplified form of the SCAN task ( Lake & Baroni , 2018 ) and a simple context free language learning task . Symbolic architectures form the main alternative to conventional neural networks as models of intelligent behaviour , and have distinct characteristics and abilities . Specifically , they form representations in terms of structured combinations of atomic symbols . Their power comes not from the atomic symbols themselves , which are essentially arbitrary , but from the ability to construct and transform complex structures . This allows symbolic processing to happen without regard to the meaning of the symbols themselves , expressed in the formalist ’ s motto as If you take care of the syntax , the semantics will take care of itself ( Haugeland , 1985 ) . From this point of view , thought is a form of algebra ( James , 1890 ; Boole , 1854 ) in which formal rules operate over symbolic expressions , without regard to the values of the variables they contain ( Marcus , 2001 ) . As a consequence , those values can be processed systematically across all the contexts they occur in . So , for example , we do not need to know who Socrates is or even what mortal means in order to draw a valid conclusion from All men are mortal and Socrates is a man . However , connectionist approaches have been criticised as lacking this systematicity . Fodor & Pylyshyn ( 1988 ) claimed that neural networks lack the inherent ability to model the fact that cognitive capacities always exhibit certain symmetries , so that the ability to entertain a given thought implies the ability to entertain thoughts with semantically related contents . Thus , understanding these symmetries and designing neural architectures around them may enable us to build systems that demonstrate this systematicity . However , the concept of systematicity has itself drawn scrutiny and criticism from a range of researchers interested in real human cognition and behaviour . Pullum & Scholz ( 2007 ) argue that the definition is too vague . Nonetheless , understanding the symmetries of symbolic processes is likely to be fruitful in itself , even where human cognition fails to fully embody that idealisation . We investigate two kinds of symmetry , relating to permutations of symbols and to equivalence between memory slots . The relation between symbols and their referents is , in principle , arbitrary , and any permutation of this correspondence is therefore a symmetry of the system . More simply , the names we give to things do not matter , and we should be able to get equivalent results whether we call it rose or trandafir , as long as we do so consistently . Following on from that , a given symbol should be treated consistently wherever we find it . This can be thought of as a form of symmetry over the various slots within the data structures , such as stacks and queues , where symbols can be stored . We explore these questions using a number of small toy problems and compare the performance of architectures with and without the relevant symmetries . In each case , we use convolution as the means of implementing the symmetry , which , in practical terms , allows us to rely only on standard deep learning components . In addition , this approach opens up novel uses for convolutional architectures , and suggests connections between symbolic processes and spatial representations . 2 RULE LEARNING . The first symmetry to be considered is the one arising from the fact that the correspondence between referring atomic symbols and their referents is entirely arbitrary , which entails that permutations of these symbols are symmetries of the system . In fact , Corcoran & Tarski ( 1986 ) use this permutation invariance to define the difference between the logical and non-logical parts of a language . This symmetry , in which any symbol is as good as any other , is anathema to the sort of problem that neural nets are typically applied to , in which the inputs are not arbitrary names but specific measurement values , e.g . images or medical records . In that case , effective learning requires discovering the appropriate differentiations , and would be sabotaged by randomly permuting the inputs . Nonetheless , the work of Marcus et al . ( 1999 ) suggest that this symmetry may be relevant to human cognition , even for infants as young as 7 months . In these experiments , the infants were habituated to sequences of syllables which obeyed a simple rule , such as ABB ( e.g . la ti ti ) or ABA ( e.g . la ti la ) . Subsequent testing on novel stimuli showed they were able generalise this rule to syllables not present in the training stimuli ( e.g . wo fe fe or wo fe wo ) . In other words the representation of the learned rule allowed it to be abstracted from the particular training stimuli and applied to any syllable . One interpretation is that the infants were treating the stimuli symbolically , in that one syllable was as good as any other , and that as a consequence their behaviour was symmetric under replacement of the syllables . Marcus et al . ( 1999 ) were unable to obtain the same behaviour from a recurrent network architecture , because the statistical regularities it learned were linked to the specific syllables seen at training time and so generalisation to unseen syllables was not achieved . Here we show that this problem can be solved by imposing a symmetry on the architecture , that corresponds to weight sharing between syllables . Practically , this is implemented as a one dimensional convolution of width one , followed by maxpooling across all syllables , and a softmax to produce output probabilities . The input consists of a 12 × 3 array of binary values representing the 12 syllables and 3 time steps , with convolution treating the syllables as positions and the time steps as channels . The output of the convolution has two channels , which after pooling become the logits for the binary outputs . Note that this the opposite of how convolutions are most frequently used in application to language sequences . In that case , invariance to time translations is achieved by weight sharing across time steps , with the representation of each symbol being encoded in the channels . Here , in contrast , weight sharing happens between symbols and temporal information is encoded in the channels . This means our model is not invariant to translations in time , but is instead invariant to permutations of symbols . Figure 1 shows the input sequence wo fe wo being processed by this architecture . The input is first encoded as activation in the first and third channels at the wo position , and activation in the second channel at the fe position . Convolution reduces these three channels down to two , and pooling projects this down to a pair of logits corresponding to the ABB and ABA categories . The training and test inputs are taken from the Marcus et al . ( 1999 ) paper , and we train the model to distinguish ABB sequences from ABA sequences . We also train a multi-layer perceptron and a recurrent net on the same data , with the recurrence happening over the time dimension . The results on the test set in Table 1 show that neither the multi-layer perceptron nor the recurrent network learn a rule that generalises effectively to unseen syllables . However , the weight sharing in the convolutional net requires that the same function is applied to each syllable , giving perfect generalisation . The filter , being applied at every position , can not discriminate between syllables , and instead can only respond to the information about temporal structure in the channels . So , for example , in the case of the sequence wo fe wo , the input channels at the wo position take the values 101 , representing the fact that the same token occurs in the first and third temporal slots . This is very similar to what Marcus et al . ( 1999 ) suggest is being learned by the infants : algebra-like rules that represent relationships between placeholders ( variables ) , such as ‘ the first item X is the same as the third item Y ’ . Thus , by imposing a symmetry on the network , we learn functions that are sensitive to an abstract structure rather than the specific raw syllables in the input . Thus , convolution provides a straightforward solution to this long-standing problem ( Alhama & Zuidema , 2019 ) using the simple expedient of sharing weights between seen and unseen syllables . Moreover , we can make further use of the insight that symmetry allows us to abstract structure away from the particular content it contains . 3 COMPOSITION . Lake & Baroni ( 2018 ) investigated the systematicity of recurrent networks in terms of a task they call SCAN . This requires translating a sequence of instructions - such as turn left and jump twice after walk - into a sequence of actions - such as LTURN WALK JUMP JUMP . They found that while a sequence-to-sequence architecture could achieve near perfect scores when the test instances closely resembled the training data , performance broke down when generalisation outside the training distribution was required . In particular , the network struggled to translate sequences that were longer than those in the training data and those that contained novel combinations of words , e.g . jump twice when both jump and twice had been seen , but their combination had not . A full solution for the SCAN task requires methods for handling a number of tricky problems in a systematic manner , e.g . parsing the instruction sequence , composing items , reversing the order of time constituents linked by after . Instead , we focus on a single problem : that of learning to compose an instruction such as jump with a modifier such as twice in a way that generalises systematically . Specifically , we construct a dataset in which the inputs are two word instructions drawn from a vocabulary of 10 commands ( jump , walk , run , . . . ) and 10 modifiers ( one , two , three , . . . ) . Each instruction is to be translated into the corresponding action repeated the appropriate number of times , e.g . jump four→ JUMP JUMP JUMP JUMP . We randomly sample with replacement 1000 such translation pairs , choose three combinations and remove all instances of them from the training data and then exclusively test on these unseen pairings of command and modifier . For a network to behave systematically , it must learn to associate a modifier , e.g . four , with a structure , e.g . repeat the same action four times , that applies to all actions indiscriminately . Following the lesson learned in the previous experiment , we might expect that the structural information might be best represented in the channels and filter of a convolution , and the actions as positions which are then treated symmetrically . However , rather than hard-code this distinction into the encoder network innately , we allow a recurrent-convolutional architecture to discover this approach for itself . Each symbol is given two input representations , one of which is used as a filter in a width one convolution and the other of which feeds into the data that the convolution is applied to . Our intention is that the former should represent structural information ( how many times to repeat ) and the latter represent the content ( which action ) within that structure . As shown in Figure 2 , information flows through the model from these inputs to the output logits through a series of convolutions , which impose a permutation symmetry on the function learned in training . This invariance to permutations of the output symbols should permit the model to learn representations of structure which abstract away from the particular actions seen during training . Within the recurrent-convolutional architecture , the hidden states , ht , consist of an array of units comprising 5 channels by 11 positions , and one dimensional convolutions of width one are used as the basis for both recurrence between time steps and also to project the output logits from the hidden states . In the encoder , two embeddings x and y are learned for every word , one of which is concatenated with the hidden state , to become the 6th channel , and the other is used as the filter in the convolution that produces the next hidden state . In the decoder , two convolutional filters are learned , one of which projects the 5 hidden channels down to a single channel , to form the output logits , and the other predicts the new hidden state from the old at each time step . The targets are encoded as one-hot values in an 11-dimensional vector ( 10 actions + pad ) , and the loss is the cross entropy with the predictions . We also add an L1 regularizer to the loss . Appendix B describes this architecture in more mathematical detail . The results in Table 2 demonstrates that only the architecture exploiting the convolutional symmetry generalizes systematically . The net learns to associate a command , e.g . jump , with an action , e.g . JUMP , and a modifier , e.g . two , with an abstract structure , e.g . repeat the same thing twice . This is possible because the symmetry across symbols allows a structure to be represented in a way that makes no reference to the particular symbols instantiating that structure .

Summary: The paper investigates the idea of using symmetry and invariance in symbolic reasoning. In particular, it considers models where modeling symbolic symmetries through parameter-sharing help with generalization. The three tasks considered are: 1) rule learning: performs sequence-classification. 2) composition: performs sequence-to-sequence with structured input using encoder-decoder architecture, and 3) context-free language learning using memory. In the first two cases, the convolution (of single width) is used to benefit from the symmetry prior, and in the third task, convolution is applied to stack memory structure. In all cases, the proposed architectures were shown to outperform MLP and RNN.