Symmetry and Systematicity
Authors: Jeff Mitchell
Jeff Bowers
Venue:  (2020)
Keywords: symmetry
systematicity
convolution
symbols
generalisation

Abstract: We argue that symmetry is an important consideration in addressing the problem
of systematicity and investigate two forms of symmetry relevant to symbolic processes. 
We implement this approach in terms of convolution and show that it can
be used to achieve effective generalisation in three toy problems: rule learning,
composition and grammar learning.

Under review as a conference paper at ICLR 2019
SYMMETRY AND SYSTEMATICITY
Anonymous authors Paper under double-blind review
ABSTRACT
We argue that symmetry is an important consideration in addressing the problem systematicity and investigate two forms of symmetry relevant to symbolic pro- cesses. We implement this approach in terms of convolution and show that it can used to achieve effective generalisation in three toy problems: rule learning. composition and grammar learning.
INTRODUCTION
Convolution (LeCun & Bengio 1998) has been incredibly effective element in making Deep Learning successful. Applying the same set filters across all positions in an image captures an important characteristic of the processes that generate the objects depicted in them, namely the translational symmetry of the underlying laws of nature, Given the impact of these architectures, researchers are increasingly interested in finding approaches that can be used to exploit further sym- metries (Cohen & Welling 2016; Higgins et al. (0188), such as rotation or scale. Here, we will investigate symmetries relevant to symbolic processing.
We show that incorporating symmetries derived from symbolic processes into neural architectures allows them to generalise more robustly on tasks that require handling elements and structures that were not seen at training time. Specifically, we construct convolution-based models that outperform standard approaches on the rule learning task Mareus al. (T999), a simplified form of the SCAN task (Lake& & Baroni 2018) and simple context free language learning task.
Symbolic architectures form the main alternative to conventional neural networks as models in- telligent behaviour, and have distinct characteristies and abilities. Specifically, they form represen- tations in terms structured combinations of atomic symbols. Their power comes not from the atomic symbols themselves, which essentially arbitrary, but from the ability to construct and transform complex structures. This allows symbolic processing to happen without regard to the meaning of the symbols themselves, expressed in the formalist's motto If you take care of the syntax, the semantics will take care of itself (Haugeland) 1985).
From this point of view, thought is form of algebra (James 1890 Boole 1854 in which formal rules operate over symbolic expressions, without regard to the values of the variables they contain (Marcus 200T). As consequence, those values contexts they occur in. So, for example, we do not need to know who Socrates is or even what mortal means in order to draw a valid conclusion from All men are mortal and Socrates is a man.
be processed systematically across all the
However, connectionist approaches have been criticised as lacking this systematicity. Fodor & Pylyshyn (1988) claimed that neural networks lack the inherent ability to model the fact that cog- nitive capacities always exhibit certain symmetries, so that the ability to entertain a given thought implies the ability to entertain thoughts with semantically related contents. Thus, understanding these symmetries and designing neural architectures around them may enable us to build systems that demonstrate this systematicity.
However, the concept of systematicity has itself drawn scrutiny and criticism from range of searchers interested in real human cognition and behaviour, Pullum & Scholz (2007) argue that the definition is too vague. Nonetheless, understanding the symmetries of symbolic processes is likely to be fruitful in itself, even where human cognition fails to fully embody that idealisation.
We investigate two kinds of symmetry, relating to permutations of symbols and to equivalence be- tween memory slots. The relation between symbols and their referents is, in principle, arbitrary, and any permutation of this correspondence is therefore symmetry of the system. More simply, the

---

Under review as a conference paper at ICLR 2019
names give to things do not matter, and we should be able to get equivalent results whether call it rose or trandafir, as long as we do so consistently. Following on from that, a given symbol should be treated consistently wherever we find it. This can be thought of as form of symmetry over the various slots within the data structures, such as stacks and queues, where symbols can be stored.
We explore these questions using a number of small toy problems and compare the performance of architectures with and without the relevant symmetries. In each case, we use convolution as the means of implementing the symmetry, which, in practical terms, allows us to rely only on stan- dard deep learning components. In addition, this approach opens up novel uses for convolutional architectures, and suggests connections between symbolic processes and spatial representations.
2
RULE LEARNING
The first symmetry to be considered is the one arising from the fact that the correspondence between referring atomic symbols and their referents is entirely arbitrary, which entails that permutations of these symbols are symmetries of the system. In fact, (orcoran & Tarski (186) use this permutation invariance define the difference between the logical and non-logical parts of a language.
This symmetry, in which any symbol is as good as any other, is anathema to the sort of problem that neural nets are typically applied to, in which the inputs are not arbitrary names but specific measure- ment values, c.g. images or medical records. In that case, effective learning requires discovering the appropriate differentiations, and would be sabotaged by randomly permuting the inputs.
Nonetheless, the work of Marcus et al. 1999) suggest that this symmetry may be relevant to human cognition, even for infants as young as months. these experiments, the infants were habituated to sequences of syllables which obeyed a simple rule, such as ABB (e.g. la ti ti) or ABA (e.g. la ti la). Subsequent testing on novel stimuli showed they were able generalise this rule to syllables not present in the training stimuli (e.g. wo fe fe or wo fe wo).
In other words the representation of the learned rule allowed it to be abstracted from the particular training stimuli and applied any syllable. One interpretation is that the infants were treating the stimuli symbolically, that one syllable was as good as any other, and that as a consequence their behaviour symmetric under replacement of the syllables.
Marcus et al. (999) were unable to obtain the same behaviour from a recurrent network architecture, because the statistical regularities it learned were linked to the specific syllables seen at training time and so generalisation to unseen syllables was not achieved. Here we show that this problem can be solved by imposing symmetry the architecture, that corresponds to weight sharing between syllables.
Practically, this implemented one dimensional convolution of width one, followed by max- pooling across all syllables, and softmax to produce output probabilities. The input consists of 12 array of binary values representing the 12 syllables and 3 time steps, with convolution treating the syllables as positions and the time steps as channels. The output of the convolution has two channels, which after pooling become the logits for the binary outputs.
Note that this the opposite of how convolutions are most frequently used in application to language sequences. In that case, invariance time translations achieved by weight sharing across time steps, with the representation of each symbol being encoded in the channels. Here, in contrast, weight sharing happens between symbols and temporal information is encoded in the channels. This means our model is not invariant to translations in time, but is instead invariant to permutations of symbols.
Figure [ shows the input sequence wo fe wo being processed by this architecture. The input is first encoded as activation in the first and third channels at the wo position, and activation the second channel at the fe position. Convolution reduces these three channels down to two, and pooling projects this down to a pair of logits corresponding to the ABB and ABA categories.
The training and test inputs are taken from the Marcus et al. (9999) paper, and we train the model to distinguish ABB sequences from ABA sequences. We also train a multi-layer perceptron and a recurrent net on the same data, with the recurrence happening the time dimension.
2

---

Under review as a conference paper at ICLR 2019
ABB ABA
Softmax
Max-Pooling
Hidden Units
Convolution
WO
Filters
Input Array
WO
ga
gi
la
fe
ko ni ta de
Figure 1: The architecture applied to the rule learning task of Mareus al. (1999), consisting of convolution followed by max- max-pooling and softmax.
Table Accuracies in identifying sequence structure
Recurrent Net
Multi-layer Perceptron
Convolution
50%
50%
100%
The results the test set in Table show that neither the multi- perceptron nor the recurrent network learn a rule that generalises effectively to unseen syllables. However, the weight sharing in the convolutional generalisation. The filter, being applied at every position, cannot discriminate between syllables, and instead can only respond to the information about temporal structure the channels. So, for example, in the case of the sequence wo fe wo, the input channels at the wo position take the values 101, representing the fact that the same token occurs in the first and third temporal slots.
requires that the same function is applied each syllable, giving perfect
This is very similar to what Marcus et al. 1999) suggest is being learned by the infants: algebra-like rules that represent relationships between placeholders (variables), such 'the first item the same the third item Y'. Thus, by imposing symmetry the network, we learn functions that are sensitive to an abstract structure rather than the specific raw syllables in the input.
Thus, convolution provides straightforward solution to this long-standing problem (Alhama & Zuidemal 2019) using the simple expedient of sharing weights between seen and unseen syllables. Moreover, can make further use of the insight that symmetry allows to abstract structure away from the particular content contains.
3 COMPOSITION
Lake & Baroni (0018) investigated the systematicity recurrent networks in terms of a task they call SCAN. This requires translating sequence of instructions such lurn left and jump twice af- ter walk into a sequence of actions such as L.TURN WALK JUMP JUMP. They found that while sequence-to-sequeei architecture could achieve resembled the training data, performance broke down when generalisation outside the training dis- tribution was required. In particular, the network struggled to translate sequences that were longer than those in the training data and those that contained novel combinations words, e.g. jump twice when both jump and swice had been seen, but their combination had not.
perfect scores when the test instances closely
A full solution for the SCAN task requires methods for handling a number of tricky problems in a systematic manner, e.g. parsing the instruction sequence, composing items, reversing the order of

---

Specifically, construct dataset in which the inputs are two word instructions drawn from Under review as a conference paper at ICLR 2019
time
JUMP
JUMP
Output Units
2
03
Hidden Units
ho
h1
66
h2
h3
Input Embeddings
y/o
1/1
jump
IWO
The mncoder-decoder architecture applied to the composition task. For each input word, two embeddings are learned, one of which concatenated with the current hidden state and the other forms the filter in a convolution applied to that input. In the decoder, actions are predicted by taking a convolution of the hidden state, and recurrence between time steps is also a convolution.
Figure 2:
constituents linked by after. Instead, focus single problem: that of learning to compose instruction such as jump with modifier such as twice in a way that generalises systematically.
vocabulary of 10 commands (jump, walk, rum, ...) and 10 modifiers (one, Iwo, three, ...). Each instruction is to be translated into the corresponding action repeated the appropriate number of times, e.g. jump four + JUMP JUMP JUMP JUMP.
We randomly sample with replacement 1000 such translation pairs, choose three combinations and remove all instances of them from the training data and then exclusively test on these unseen pairings of command and modifier.
For network to behave systematically. it must learn to associate modifier, e.g. ture, e.g. repeat the same action four times, that applies to all actions indiscriminately. Following the lesson learned in the previous experiment, we might expect that the structural information might be best represented in the channels and filter of a convolution, and the actions as positions which are then treated symmetrically.
e.g. four, with struc-
However, rather than hard-code this distinction into the encoder network innately, we allow recurrent-convolutione architecture to discover this approach for itself. Each symbol is given two input representations, one which is used as a filter in width one convolution and the other of which feeds into the data that the convolution is applied Our intention is that the former should represent structural information (how many times to repeat) and the latter represent the content (which action) within that structure.
As shown Figure 2 information flows through the model from these inputs to the output logits through a series of convolutions, which impose permutation symmetry on the function learned in training. This invariance to permutations of the output symbols should permit the model to learn representations of structure which abstract away from the particular actions seen during training.
Within the recurrent-convoltiiona architecture, the hidden states, h4s consist of an array of units comprising 5 channels by 11 positions, and one dimensional convolutions of width one are used as the basis for both recurrence between time steps and also to project the output logits from the hidden states. In the encoder, two embeddings æ and y are learned for every word, one of which is concatenated with the hidden state, to become the 6th channel, and the other is used as the filter the convolution that produces the next hidden state. In the decoder, two convolutional filters are leamned, one of which projects the 5 hidden channels down logits, and the other predicts the new hidden state from the old each time step. The targets are encoded one-hot values in 11 1--dimensional vector (10 actions pad), and the loss is the cross entropy with the predictions. We also add an L, regularizer to the loss. Appendix B describes this architecture in more mathematical detail.
single channel, to form the output
4

---

Under review as a conference paper at ICLR 2019
Table Accuracies in translating instructions into actions.
Recurrent Net
Multi-layer Perceptron
Convolution
0%
0%
100%
The results in generalizes systematically. The net learns to associate a command, c.g. jump, with an action, e.g. JUMP, and a modifier, e-g. two, with an abstract structure, e-g- repeat the same thing twice. This possible because the symmetry across symbols allows a structure to be represented in a way that makes no reference the particular symbols instantiating that structure.
that only the architecture exploiting the convolutional symmetry
CONTEXT FREE LANGUAGE LEARNING
The structures considered in the previous section are extremely simple, having only short-range sequential dependencies. In contrast, the real grammars of natural languages produce long-range dependencies within hierarchical structures. Handling such structures, in which multiple dependen- cies embedded within each other, will typically require some form of memory in order to keep track of the unresolved outer dependencies until the inner dependencies are completed. In this sec- tion we consider the role that symmetry plays in structuring this memory, again using convolution, which allows us to separate the contents of memory from the structure of how it is manipulated. In particular, we consider reverse recall task that captures a key property of how such a memory has to operate.
For example, in the sentence The rucquet is actually very cheap the subject noun, racquet, and main verb, is, display number agreement. In this case, both are singular, but they could also have been plural, i.e. racquets and are. For this simple sentence, noun and verb are adjacent so the span of the dependency is minimal. However, we can, in principle, insert as much material as we like, and this syntactic connection persists.
In particular, we can add a relative clause to the noun: The racquet that the tennis player uses is actually very cheap. Now another subject noun, player, and verb, uses, intervenes between the first pair, but the dependency, and in particular the number agreement, between racquet and is has to be maintained. This remains true even when we insert another relative clause: The racquet that the tennis player we all in of uses is actually very cheap.
In this last case, the subject, we, and verb, are, are both plural, and effective processing of the whole sentence requires that this should not disrupt processing of the outer singular dependencies. In other words, a language user must be able to maintain a trace of multiple open dependencies until the relevant material is encountered. Notably, in the case of centre embedded construction such the sentence above, recall has to happen in fast-in-first-ou manner as processing descends through the hierarchy and then rises back out again. That is, the subjects the sentence above racquet, player, we are matched to verbs in reverse order are, uses, is.
common model for such structures are Context Free Grammars (CFG), which generate sentences in terms of production rules, such as those in Figure 3 These rules describe how the start symbol, S, expanded into sequences terminals symbols, such as adoda or cebabdodbabee. Each rule describes substitution that can be applied to a single non-terminal symbol, i.e. S, A, B, C' or D to yield a sequence of symbols. The context free aspect of such a grammar lies in the fact the substitutions made without regard for the context around the original non-terminal symbol.
In the case of the grammar described in Figure 3 the substitutions applied to the non-terminals A, B, C and D yield inserts an o. As a consequence, the resulting strings are palindromes with single o at their centre.
string with the same terminal at the beginning and the end, and the final rule
An equivalent model is pushdown automata (PDA), in which hierarchical structure is handled by pushing symbols representing the outer structures onto a stack, until the interior structure is completed, and then popping symbols back off the stack to outward in the hierarchy until no
5

---

Under review as a conference paper at ICLR 2019
S A aSa
S A
5 B - bSb
- aSa
5-C C > cSc
a a adSda
5 D
D - dSd
S 0
adoda
(a) Production rules for simple CFG.
(b) Derivation of a palindrome.
Figure A simple CFG, producing palindromic strings.
Table 3: Performance of LSTM Architectures on the palindromic language.
LSTM Architecture
In Domain
Long
NA >> 2
Standard 100%
77%
82%
Convolutional
100%
100%
100%
more symbols remain. Crucially, the stack has cost-in-first-on structure that essentially returns items in the reverse order in which they were pushed onto it.
In principle, such system can handle sentences of unbounded length, containing arbitrarily long dependencies between constituents. In practice, however, language users struggle with nested struc- tures more than two or three levels deep. Moreover, it is generally accepted that an ordinary CFG is not an accurate model of the grammatical structures that natural languages display. Instead, their grammars appear be mildly context sensitive, as shown by evidence from Swiss German and Dutch (Shieber 1985).
However, here we focus on CFGs and the ability of Recurrent Neural Nets learn these structures. In particular, we investigate the ability of a Long Short Term Memory network to learn the simple palindromic language over the terminal symbols a, b, c, and o, defined by the grammar in Figure 3
We train an LSTM containing 100 hidden units, on 100,000 examples of strings of length 15, 17, 19, 21, 23 and 25, and then perform an in-domain test on novel strings of the same length, and an out-of-domain test on longer strings of lengths 29, 33, and 37. We also retrain the LSTT after removing all strings which contain more than 4 tokens of the symbol from the training set, and then test only on examples from the test set containing more than 4 tokens of the symbol a.
The first row of Table gives the results for this evaluation in terms of the proportion of symbols after the central o symbol that were predicted correctly. The in-domain results, in the first column, make it clear that the net has learned the structure of the grammar, and how to make accurate predictions, at least for sequences of lengths seen at training time. The second column, containing the results for longer sequences, shows that generalisation outside the range of the training is not robust. The third column indicates that the model has difficulty generalising to sequences where the nesting of A symbols is deeper than at training time, even though the actual length of sequences is unchanged.
These failures of generalisation can be seen as symptoms of the same underlying problem. In par- ticular, success on each of these out-of-domain tasks simply requires extending the application of the same rules. However, the problem arises because the network lacks the required concept of sameness. The LSTM cells form an unstructured memory resource, without any notion of two cells containing the same information. Nor can there be a meaning to the idea of applying the same rule to that information. The parameters for each cell are learned independently, and so each cell carries out its own isolated task. During training these cells do learn to behave coherent whole, achieving impressive in-domain performance, but there is no way for the model to apply the same rule to the n item as it applied to the previous
To address this shortcoming, we propose to organize the memory cells into an ordered one- dimensional stack structure, and
convolutions to control the flow of information across timesteps, replacing the forget gate. The translational symmetry of this convolutional layer gives meaning the idea of the same symbol being able to be stored in different cells and we use filter

---

Under review as a conference paper at ICLR 2019
Outputs
Memory Cells
C2
Co
C1
C2
Hidden Units
h4
hgg
h1
h22
Input Embeddings
3C1
32:
d
Figure 4: The modified LSTM architecture has a stack of memory cells which a convolutional gate controls the flow of information through, while standard multiplicative output gate and additive input gate read and write to the cells in the bottom of the stack. As in standard LSTM, the hidden units are concatenated with the input embeddings to form the vector that feeds into the units that drive the various gates.
of width three to allow the shifting of stored information between adjacent cells. Along with a re- striction input and outputs only being connected to the bottom of the stack, this turns the LSTM into a differentiable PDA. In particular, the filters [1, O, 0] and [0, 0, 1] shift values back and forth when applied in convolution and this allows the network to learn to shift memory contents up and down the stack in order to perform push and pop operations.
As shown in Figure 4 each token in the input is given an embedding which is then concatenated with the current hidden state. Together these values form the inputs to units that control the flow of information into, out of and between the memory cells, in standard LSTM. In this case, however, the memory cells are a set one channel one dimensional convolutional layers and the forget gate has been replaced with filters are the softmax outputs of units driven by the concatenated input embeddings and hidden units, allowing the network use the input context control the memory cell stack. Information is written to and read out from only the bottom entries in the stack, using standard input and output gates. This architecture described in more mathematical detail Appendix C
of width three filters that shape the recurrent flow of information. These
Performance of this convolutional LSTM on the same evaluations is given in the second row of Table 3 There, all three columns show optimal performance on both the in-domain and out-of- domain tasks, demonstrating the utility of the convolutional layer in helping the model to generalise robustly.
RELATED WORK
Numerous authors have tackled the problem of replicating the rule learning behaviour studied by Marcus et al. (1999) in connectionist system, and Alhama & Zuidemal (2019) give an extensive review, Many of these approaches rely on specific training regime obtain the desired behaviour, rather than our approach of modifying the architecture to embed the appropriate capacities innately. However, our core intention was demonstrate the relevance of symmetry. with convolution being convenient and transparent means to that end. The same end could conceivably be achieved purely through learning.
The SCAN task of Lake & Baroni (2018) has also stimulated a number of responses which attempt to obtain the required systematicity. Lake (2019) uses meta learning approach, which employed training regime that explicitly permuted the correspondence between symbols and their meaning, i.e. between instructions and actions. This could be seen as learning an invariance to these permutations, rather than specifying it innately in the form of convolution. Russin al. (2019), instead, take an approach based on learning separate semantic and syntactic representations for each word, which is comparable approach of learning embeddings for each word.
7

---

Under review as a conference paper at ICLR 2019
The recurrent PDA we describe in Section 4 is very similar to number of other architectures. Sun iet al. (1998) proposed a neural network pushdown automata. Grefenstette et al. (2015) proposed architectures for a number of data structures: queues, dequeues and stacks. Joulin & Mikolov
(2015) proposed a recurrent stack structure, which in practice, is almost equivalent to our proposal. However, none of these works discuss the role of symmetry or the connection convolution.
Symmetries beyond spatial translation have been discussed by a number of authors. [Cohen & Welling (2016) propose a generalisation of convolution for arbitrary discrete symmetries, such as reflections and rotations. The role of invariances in disentangled representations is discussed by Higgins et al. (2018), , and [Bloem-Redee symmetries to neural network architectures. Practical examples of symmetries supporting extrapo- lation and generalisation beyond the training set are discussed by Mitchell et al. (2018).
& Teh (2019) investigate the application of probabilistic
Permutation invariance is relevant to number of representational strategies, such as bag-of-words approaches (e.g. White et al., 2015) or Deep Sets (Zaheer et al.) (017). However, the relevant symmetry in these cases is usually over permutations on the order of inputs, e.g. a ssymmetry between
wo fe fe and fe wo fe. In our case, the permutation is over the identity of the symbols, i.e. a symmetry between wo fe fe and la ti ti.
DISCUSSION
One way to address the criticisms of distributed approaches raised by Fodor & Pylyshyn (1988) has been to focus on methods for binding and combining multiple representations (Smolensky 1990. Hinton 1990 Plate [991 Pollack 1990 Hummel & Holyoak 1997) in order to handle constituent structure more effectively. Here, we instead examined the role of symmetry in the systematicity of how those representations are processed, using a few simple proof-of-coneee problems.
We showed that imposing a symmetry on the architecture was effective in obtaining the desired form of generalisation when learning simple rules, composing representations and learning grammars. In particular, we discussed two forms of symmetry relevant to the processing of symbols, correspond- ing respectively to the fact that all atomic symbols are essentially equivalent and the fact that any given symbol can be represented in multiple places, yet retain the same meaning. The first of these gives rise to a symmetry under permutations of these symbols, which allows generalisation to oceur from one symbol to another. The second gives rise to symmetry across memory locations, which allows generalisation from simple structures to more complex ones.
On all the problems, we implemented the symmetries using convolution. From a practical point of view, this allowed us to build networks using only long-accepted components from the standard neural toolkit. From theoretical point of view, however, this implementation decision draws connection between the cognition of space and the cognition of symbols.
The translational invariance of space is probably the most significant and familiar example of sym- metry we encounter in our natural environment. As such it forms a sensible foundation on which to build understanding of other symmetries. In fact, [orcorra & Tarski (986) use invariances un- der various spatial transformations within geometry as a starting point for their definition of logical notion in terms of invariance under all permutations. Moreover, from an evolutionary perspective. is also plausible that there are common origins behind the mechanisms that support the exploitation of a variety of different symmetries, including potentially spatial and symbolic. In addition, recent research supports the idea that cerebral structures historically associated with the representation of spatial structure, such as the hippocampus and entorhinal cortex, also play a role in representing more general relational structures (Behrens et al. 2018 Duff & Brown-Schmidi 2012).
Thus, our use of convolution is not merely a detail of implementation, but also an illustration of how spatial symmetries might relate to more abstract domains. In particular, the recursive push down automata, discussed in Section 4 utilises push and pop operations that relate fairly transparently to spatial translations. Of course, variety of other symmetries, beyond translations, are likely to be important in human cognition, and an important challenge for future research will be to understand how symmetries are discovered and learned empirically, rather than being innately specified.
A common theme in our exploration of symmetry, was the ability it conferred to separate content from structure. Imposing a symmetry across symbols memory locations, allowed to abstract
8

---

Under review as a conference paper at ICLR 2019
away from the particular content represented represent the structure containing it. So, for example the grammar rule learned by our network on the syllable sequences of Marcus et al. (1999) was able to generalise from
unseen syllables because it represented the abstract structure of ABB and ABA sequences, without reference to the particular syllables involved. We explored how this ability could also be exploited composition and grammar learning tasks, but it is likely that there many other situations where such a mechanism would be useful.
REFERENCES
Raquel G. Alhama and Willem Zuidema. A review of computational models of basic rule learning: The neural-symbolic debate and beyond. Psychonomic Bulletin & Review, 264)))11144411119 Aug 2019. ISSN 1531-5320. doi: 10.37588113423-0990066222 URL [RRnttss://ddi.rg//111
3758/ 00088/11423--- 01011----6220 602-2
TEJ Behrens, TH Muller. JCR Whittington, S Mark. AB Baram, KL Stachenfeld, and Z Kurth- Nelson. What is a cognitive map? organizing knowledge for flexible behavior. Neuron, 100(2): 490-509, 2018. ISSN 08966273. doi: (000l//..eeronn20188..000
Benjamin Bloem Bloem-Reddy and Yee Whye Teh. Probabilistic symmetry and invariant neural networks. CoRR, abs/1901 000//901.06600 .06082, 2019. htt.thttt://aatt org/ baborg/ab/////// absaaas///00 06082)
George Boole. An Investigation of the Laws of Thought: On Which Are Founded the Mathematical Theories of Logic and Probubilities. Walton and Maberley, 1854.
Taco Cohen and Max Welling. Group equivariant convolutional networks. Maria Florina Balcan and Kilian Q Weinberger (eds.), Proceedings of The 33rd International Conference Machine Learning, volume 48 of Proceedings of Machine Learning Reseurch, pp. 2990-2999, New York, New York, USA, 20-22 Jun 2016. PMLR. URL http:://proceeciinnn www.:iinnsmmmm...r.mrmme
V448/chhecc6... html
John Corcoran and Alfred Tarski. What are logical notions? History and Philosophy of Logic, 7(2): 143-154, 1986. doi: 10. 048000044534980883700
Melissa Duff and Sarah Brown-Schmidt. The hippocampus and the flexible and processing of language. Frontiers in Human Neuroscience, 6:69, 2012. ISSN 1662-5161. doi: 10.3389/fnhum. 2012.00069. / http::///www..fottreesss.ooooo//
fnhum.
2012.00063
Jerry A. Fodor and Zenon W. Pylyshyn. Connectionism and cognitive architecture: A critical anal- ysis. Cognition, 28(1-2)3--71, 1988. doi: 10110160011002777899031555
Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. Learning to transduce with unbounded memory. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume NIPS' 15, pp. 1828-1836, Cambridge, MA, USA, 2015. MIT Press. URL http: //dl acm. wwtp://gg///iiioooo htt..oo...cmm..2929222222 2969444]
John Haugeland. Artificial Intelligence: The Very Idea. Massachusetts Institute of Technology, Cambridge, MA, USA, 1985. ISBN 0-262-08155 155-99
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards Definition Disentangled Representations. arXiv e-prints, art. Dec 2018.
Geoffrey E. Hinton. Mapping whrt-whole hierarchies into connectionist networks. Artif. Intell., 46 (1-2):47-75, November 1990. ISSN 0004-3702, doi: 10.001660004370229990000-. URL http::/dxx.o doi. 110 101 1010011/0000331111 (90) 90004-]]
John E. Hummel and Keith J. Holyoak. Distributed representations of structure: A theory of ana-
logical access and mapping. PSYCHOLOGICAL REVIEW. 104(3):427-466, 1997.
W. James. The Principles of Psychology. Number v. 1 in American science series. H. Holt, 1890.
URL | tittps://books. .rrooookssoooooooo googooooooooooooo wtw.kk/ookkkiidddttlaooooo...o

---

Under review as a conference paper at ICLR 2019
Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, pp. 190-198, Cambridge, MA, USA, 2015. MIT Press. URL
http: //tl///l.. acm. www.emowr....tee.ooeeooooooooooe ctt./iitaaai... wtt.iiiooccc.c.mmmmmmmm (ennnndd=292229990 29692611
Brenden M. Lake. Compositional generalization through meta sequence-to-sequense learning. wwggabs/////oooooo..t....t....at
CoRR, Abs/1906.05381. 2019. /ttttp:///axxx org/ abs 19w.19966.00 05381
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the composi- tional skills of sequence-to-sequence recurrent networks. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssaan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Ma- mmlr.
chine Learning Research, pp. (879928888. PMLR, 2018. UtLhttp://ppoceeeee
press/v80 Lakel8a.aaaa........... html
Yann LeCun and Yoshua Bengio. Convolutional networks for images, speech, and time series, In Michael A. Arbib (ed.), The Handbook of Brain Theory and Neural Networks, pp. 255-258. MIT Press. Cambridge, MA, USA, 1998. ISBN 0-262-51102-9, URL http: /dl. acm. org/
eitation.cfm221n 303568 303704
G. F. Marcus, S. Vijayan. S. Bandi Rao, and P. M. Vishton. Rule learning by seven-month-old infants. Science, 283(5398):77-80, 1999. ISSN 0036-8075. doi: 10.1 20.11126/ciiemme223.553...
URL https:
scwencemag htt.://maa.oorrooooooeee////////
Gary F. Marcus. The Algebraic Mind. MIT Press, 2001.
Jeff Mitchell, Pontus Stenetorp, Pasquale Minervini, and Sebastian Riedel. Extrapolation in NLP. In Proceedings of the Workshop on Generalization in the Age of Deep Learning, pp. 28-33, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/
W18-1005. URL BRRLnttps:/// www. ac aww.eb...gw.bb.o.o.oo....oo.... wwt.:orgg/..ntooooyooe.. 111110
Tony Plate. Holographic reduced representations: Convolution algebra for compositional distributed representations. In Proceedings of the 12th International Joint Conference Artificial Intelli- gence Volume 1, IJCAI'91, pp. 30-35, San Francisco, CA, USA, 1991. Morgan Kaufmann Publishers Inc. ISBN 1-55860-160-0.. URL httphhttp///ll..... acm org/ cwtp:iiiiiio.oo..oo.ommoo cfm?id=
1631171.
J. B. Pollack. Recursive distributed representations. Artif. Intell., 46(1-2):77-105, November 1990. ISSN 0004-3702. doi: 10. 10.101660004437729999000066 URL
org/10.
1016/0004-37022 ( (90) 90005-K
Geoffrey K. Pullum and Barbara C. Scholz. Systematicity and natural language syntax. Croatian Journal of Philosophy, 3(21), 2007.
Jake Russin, Jason Jo, Randall C. O' O'Reilly, and Yoshua Bengio. Compositional generalization in deep seq2seq model by separating syntax and semantics. arXiv e-prints, art. arXiv:1904.09777 Apr 2019.
Stuart M. Shieber. Evidence against the context-freeness of natural language. Linguistics and Phi- losophy, 8:333-343, 1985.
P. Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artif. Intell., 46(1-2):159 216, November 1990. ISSN 0004-3702. doi: 10. 0016000044330000900900000-4 URL /tte:://aa..a...or///00000000000 90007-M
Guo-Zheng Sun, C. Lee Giles, and H. H. Chen. The neural network pushdown automaton: Architecture, dynamics and training. In Adaptive Processing of Sequences and Data Struc- fures. International Summer School Neural Networks, "E.R. Caianiello" "iiieelloo-Tuttiii "uutorial Lectures, pp. 296-249633 345, London, UK, UK. 1998. Springer- Verlag. ISBN 3-540-64341-9, URL http: 7/dl acm. www.oo/gtgttttncoccooo 733333 733213
10

---

Under review as a conference paper at ICLR 2019
Lyndon White, Roberto Togneri, Wei Liu, and Mohammed Bennamoun. How well sentence embed- dings capture meaning. In Proceedings of the 20th Australasian Document Computing Sympo- shum, ADCS 15, pp. 9:1-9:8, New York, NY, USA, AC1. doi: 10.1145/283991.28889922 URL http://do cmm org /0/ wet.ee111222888888 2838932
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. Curran Associates, Inc., 2017. URL http: /wpprsppprrs..e/// nips. CC cc/paper/ 6931-deep sets.pdf
APPENDICES
A SYMMETRY
Informally, a symmetry of system is a mapping of the system onto itself which preserves the fundamental properties of the system. In the case of translation symmetry, we have input images, v, and output labels, y, and we want to learn a function, fzz), which predicts these labels, and invariant to spatial translations, T. That is, we want to obey f(Tx) f(x).
Typically, we achieve this by composing two types of function: equivariant convolutions, c, and invariant poolings, p. Equivariant here means that the output from translated input is itself the translation of the original output: c(Tx)) = Tc(x). While invariant means the output unchanged by input translations: pTTx) p(z).
When the width the convolution reduced to one, the function, cQ). becomes equivariant to all permutations, S, not just translations: c(Sx) Sc(w). Permutation equivariance arises, for example, in formal logic, where the rules of deduction depend not on the particular names within an expression, but on its logical structure. So, Socrates mortal follows from Socrates is a man and All men are mortal not because of the meaning of Socrates or mortal, but because the syllogism represents the premises and y represents the conclusions, then the process of deduction d(x) y should be equivariant under any substitution, S, of names. That is, we should be able to reach an equivalent conclusion even if rename Socrates as Bob, and so d(Sx)= Sd(x).
has the right form. Thus,
Symmetries also arise in computational processes. For example, x the state some machine containing addressable memory and running program that references various addresses then we want the behaviour of the machine to be equivalent we utilise a different set of memory addresses. That is, the state transition function, f(). should be equivariant under permutations, S, of states, which apply equivalent permutations the memory content and the addresses referenced in the program: f(Sx) Sfx))
B COMPOSITION
Our approach to the composition task is based on the incoder-decoded architecture shown in Figure This made more precise in the equations below.
In the encoder, each token in the input is given an N-dimensional embedding, x and an M+1 x M dimensional embedding, y. The former is concatenated to the current N x M dimensional hidden
state to become the M+ 1th channel of a vector, g.
9: h
A convolution of width one applied this vector to generate the next hidden state, using the second embedding as filter.
h y: 7 gt

---

Under review as a conference paper at ICLR 2019
The decoder takes the final hidden state from the encoder and projects it down single channel, using a width one convolution. These are then the logits of a softmax output, o, predicting the next action.
0, softmax (fo ht)
The next hidden state is also produced by width one convolution, in this case maintaining the number of channels.
h++1 fr h
(4)
An L1 regularisation is added to the cross entropy between the predictions, and one hot encod- ing, a, of the correct actions.
Loss = Ha, o) + Aa |2/1 + A
(5)
For the composition task, we used N=11 and M=5.
C CONTEXT FREE LANGUAGE LEARNING
Figure gives a visual overview of the architecture we apply to learning the simple palindromic language. This is essentially modified LSTM in which the forget gate has been replaced with convolutional filters. We define this explicitly below.
Each token in the imput is given an M-dimensional embedding, æ, which is concatenated with the current N-dimensional hidden state, h, to give vector, representing the current context.
9. h4 +
This controls N single channel width three filters, f, each of which is the output of a softmax.
fx,t softmax (Wingt + by.n)
As in standard LSTM, values written the cells are the outputs of tanh units gated by sigmoid.
i, (W.gr + b,) tanh (W.g. b.)
(8)
And the output gate is also sigmoid function.
O, = a W.gg + b.)
Recurrence between the cells each of the N memory stacks is based on one-dimenli convo- lution and controlled by the filters f.
Ca.t fr, Cr.t-1
(10)
The 0thh values in each stack are updated using the values
Ca,t.0 = Cn.t.0 + it.m
(11)
And the new hidden state are also read out from the tth values, gated by tanh units.
h441, Ot,a tanh (Cn.t.)
(12)
12

---

Under review as a conference paper at ICLR 2019
The final outputs, to predict hot vector representing the next symbol, apply softmax to these new hidden states.
8; = softmax (W, [W, + b,)
(13)
The loss is the cross entropy over the second half of the sequence, and for this task M and N are both 10, with the memory stack having depth of 20.
13


Reviews:
[{"review_id": "ryx5OlGTYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n======================================== Update after rebuttal =============================================\n\nI have now read the author rebuttal, but my concerns about the paper remain. The training details are not described in anywhere near sufficient detail (optimizer?, batch size?, learning rate?, initialization?, etc). The baseline architectures “recurrent net” or the “multi-layer perceptron” are not described at all, despite my explicit request to that effect. I had also requested to see the source code for the experiments as this would perhaps have illuminated a lot of the details left out in the paper, but the authors have not provided it. I understand that the authors are not required to provide their code, but this should have been a relatively straightforward request in this case given the simplicity of the experiments and as I mentioned in my initial review, it would have been very useful in evaluating the paper. \n\nIn their rebuttal, the authors also claimed that the results in Fig. 1 and Table 1 are training results (that even though the architecture is \"innate\", the weights are learned), but I'm concerned about this claim. I happen to be doing some experiments along these lines at the moment, and it is not trivial at all to get such crisp results as those shown in Fig. 1 & Table 1 in these kinds of experiments (even when the architecture is correctly specified). Again, it would have been very helpful if the authors had either provided their source code or had described their experimental setup in sufficient detail to allow the reproduction of these results. \n\nGiven these concerns, I have decided to keep my score as it is.\n\n========================================================================================================\n\nThis paper addresses an important problem: systematic generalization in neural networks. However, the paper is very confusing and I have some serious concerns about the models and the results presented in sections 3 and 4. Here are the main issues:\n\n1) In section 3, there are only 10x10=100 possible combinations in this composition task. Yet, the paper says “we randomly sample 1000 such translation pairs, choose three combinations and remove all instances of them from the training data and then exclusively test on unseen pairings of command and modifier.” How can you sample 1000 pairs out of a possible 100 combinations? Also being able to generalize to 3 held-out combinations out of 100 is not very impressive. On the contrary, it is almost trivial.\n\n2) No details are given about the “recurrent net” or the “multi-layer perceptron” baselines in section 3. What are these models? The fact that they have exactly zero accuracy is a bit suspicious, especially given the almost trivial nature of the task in section 3. Previous works reported perfect or near perfect accuracy with similar baselines in similar tasks (see e.g., Lake & Baroni, 2018).\n\n3) I'm afraid the proposed model in section 3 also doesn’t make sense to me. It is explicitly acknowledged (Appendix B) that y is an M+1-dimensional vector, g is an Nx(M+1) matrix. Then by all accounts, the convolution of these should be an N-dimensional vector. Yet, somehow, h_t+1 in Equation 2 manages to be an NxM matrix. How? Please clarify this. If possible, making the source code available would be very helpful.\n\n4) What is the semantics of x and y in section 3? What exactly are they supposed to be doing? This is not explained in the paper beyond a vague description.\n\n5) Similar problems arise in section 4. The task is not explicitly described in the text. We only learn from Appendix C that it is actually to predict the next symbol. The task description mentions “strings of length 15, 17, 19, 21, 23 and 24,...” (p. 5). But, the grammar in Fig. 3 can only generate odd length strings, it cannot generate a string of length 24. Is this a typo?\n\n6) Again in section 4, I have no idea how the proposed model is actually supposed to work. The motivation for the model and its description are not clear at all.\n\n7) The model in section 2 is hand-coded. It is not shown that it can actually learn this solution from data. What happens if the sequences are longer or if the rules are different, for example? Then you have to hand-code a completely different architecture.\n\n8) Which brings me to another important issue I have with this paper (and with similar papers): this whole set-up is very misguided in my mind. I think the real problem is not to come up with an architecture that would generalize systematically in a very specific (and usually toy) problem. It is to come up with an architecture that would learn to generalize systematically in a much broader set of problems. The learning aspect in sections 3-4 is a step in the right direction, but there’s no evidence that the models proposed there can learn to generalize in any task other than the very specific tasks they were designed for (if they can actually do that).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJe7pRBAFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates the idea of using symmetry and invariance in symbolic reasoning. In particular, it considers models where modeling symbolic symmetries through parameter-sharing help with generalization. The three tasks considered are: 1) rule learning: performs sequence-classification. 2) composition: performs sequence-to-sequence with structured input using encoder-decoder architecture, and 3) context-free language learning using memory. In the first two cases, the convolution (of single width) is used to benefit from the symmetry prior, and in the third task, convolution is applied to stack memory structure. In all cases, the proposed architectures were shown to outperform MLP and RNN.\n\nThe paper addresses an important area in deep learning, and the paper is accessible and easy to read. However, there are major issues:\n\n-- I found it challenging to identify a novel contribution. For example, the first task is simply using a single convolution layer followed by pooling for sequence prediction. However, using 1D convolution layers are somewhat wide-spread in NLP. \n\n-- The paper is oblivious to a large body of related work in the area of relational learning and invariant/equivariant deep learning. Here are some examples: Permutation invariant model for sets [1,2], and the link between parameter-sharing and invariance [3,4] is theoretically studied in several works. Note that convolution with a filter of width one followed by pooling is exactly invariant to the symmetric group. There are related works that extend these ideas to graphs [5,6], and relational learning [7]. Invariance has also been explored as it relates to memory [8]. Another relevant direction to discussions of the paper is the idea of attention in various architectures, such as transformers.\n\n-- There are vague or misleading claims. In particular, for some tasks, it is not clear why the proposed architecture addresses the targeted symmetry. For example, it is not clear why translation invariance in-memory models the structure in a context-free grammar. \n\n\n[1] Zaheer, Manzil, et al. \"Deep sets.\" Advances in neural information processing systems. 2017. \n[2] Qi, Charles R., et al. \"Pointnet: Deep learning on point sets for 3d classification and segmentation.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. \n[3] Shawe-Taylor, John. \"Building symmetries into feedforward networks.\" 1989 First IEE International Conference on Artificial Neural Networks,(Conf. Publ. No. 313). IET, 1989.\n[4] Ravanbakhsh, Siamak, Jeff Schneider, and Barnabas Poczos. \"Equivariance through parameter-sharing.\" Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017. \n[5] Kondor, Risi, et al. \"Covariant compositional networks for learning graphs.\" arXiv preprint arXiv:1801.02144 (2018). \n[6] Maron, Haggai, et al. \"Invariant and equivariant graph networks.\" arXiv preprint arXiv:1812.09902 (2018). \n[8] Kazemi, Seyed Mehran, and David Poole. \"RelNN: A deep neural model for relational learning.\" Thirty-Second AAAI Conference on Artificial Intelligence. 2018. \n[9] Vinyals, Oriol, Samy Bengio, and Manjunath Kudlur. \"Order matters: Sequence to sequence for sets.\" arXiv preprint arXiv:1511.06391 (2015).\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1ePQyfk9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on modelling invariances or symmetry between various components for solving tasks via convolutions and weight sharing.\nThe proposed tasks are toyish in nature although they do give insights into importance of modeling symmetry for better generalization. The first task is a symbol substitution which considers a permutation in source symbols and maps them to either \"ABA\" or  \"ABB\" categories i.e binary classification. While this task does require generalizability, it is surprising that the mlp and recurrent net baselines are so much inferior (basically random) to the convolution baseline. While this shows efficacy of modeling symmetry, I'd be curious about performance graphs as the training data increases in size.\nThe second task is an artificially created task inspired from the SCAN dataset. The task is to translate a verb-number pair into number repetitions of the verb. The encoder decoder network uses convolution in the recurrences to capture the notion of generalizability. The input and output space is very small (10 verbs and 10 numbers) but shows superiority of convolution and weight sharing over other baselines. Curiously, the recurrent baseline seems to perform better than 0% accuracy (if still poorly) on the original SCAN task which is much harder than the proposed task in this paper. Maybe, the number of examples (1000) is too small recurrent networks but this makes me a little surprised. More details about the architecture and training procedure for baselines would be helpful to ensure that the comparison is fair across baselines.\nThe final task is CFG modeling where convolutions are used to model the forget gate of an LSTM which seems to endow the network with PDA like properties and the convolutions are more effective than baselines at modeling this.\n\nApart from the concerns related to the results mentioned above, my major concern is that the tasks considered are too simple and at least one complicated large-scale task would have strengthened the paper.\nAlso, for tasks 2 and 3, the motivation behind using convolutions is not as clean as in task 1. So more analysis and insights into model performance, the weights learned, ablation studies etc. would have helped in understanding how the convolutions are modeling the symmetry. This should be informative and tractable because of simplicity of the tasks involved.\n\nFinally, as mentioned above, I still cannot intuitively understand why convolutions in the forget architecture would learn about symmetry related to structured repetition produced by a CFG. Hence, more analysis or a better motivation would have helped. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for clarifying several issues raised by the reviewers, which helped us understand the paper.

After all, we decided not to accept this paper due to the weakness of its contribution. I hope the updated comments by the reviewers help you strengthen your paper for potential future submission.
Decision: Reject