Symbol-Shift Equivariant Neural Networks
Authors: David Salinas
Hady Elsahar
Venue:  (2021)
Keywords: compositionality
Symbolic
Equivariance
question answering
Language Processing

Abstract: Neural networks have been shown to have poor compositionality abilities: while they can produce sophisticated output given sufficient data, they perform patchy generalization and fail to generalize to new symbols (e.g. switching a name in a sentence by a less frequent one or one not seen yet). In this paper, we define a class of models whose outputs are equivariant to entity permutations (an analog being convolution networks whose outputs are invariant through translation) without requiring to specify or detect entities in a pre-processing step. We then show how two question-answering models can be made robust to entity permutation using a novel differentiable hybrid semantic-symbolic representation. The benefits of this approach are demonstrated on a set of synthetic NLP tasks where sample complexity and generalization are significantly improved even allowing models to generalize to words that are never seen in the training set. When using only 1K training examples for bAbi, we obtain a test error of 1.8% and fail only one task while the best results reported so far obtained an error of 9.9% and failed 7 tasks.


Under review as a conference paper at ICLR 2021
SYMBOL-SHIFT EQUIVARIANT NEURAL NETWORKS
Anonymous authors
Paper under double-bin suuble-blind review
ABSTRACT
Neural networks have been shown to have poor compositionality abilities: while they can produce sophisticated output given sufficient data, they perform patchy generalization and fail to generalize to symbols (e.g. switching name in a sentence by a less frequent one or one not seen yet). this paper, we define a class of models whose outputs are equivariant to entity permutations (an analog being convolution networks whose outputs are invariant through translation) with- out requiring to specify detect entities in pre-procest proceeseesing step. We then show how two guestion-answering models can be made robust to entity permutation us- ing novel differentiable hybrid semantic-symbolic representation. The benefits of this approach are demonstrated on a set of synthetic NLP tasks where sample complexity and generalization are significantly improved even allowing models generalize to words that are never seen in the training set. When using only 1K training examples for bAbi, we obtain test error of 1.8% and fail only one task while the best results reported so far obtained an error of 9.9% and failed tasks.
INTRODUCTION
Previous work have shown how neural networks fail to generalize to new symbols (Lake & Baroni,
2018 Sinha al. 2019 Hupkes al. 2019). In particular, [ake & Baroni (2018) showed that seq2seq models are able to perfectly learn a set of rules given enough data, however they fail to generalize these learned rules new symbols.
We illustrate the generalization issue of current models in the context of crestion-answeriigg (QA) the first task of bAbi (Weston et al. 2015). This dataset identified set of tasks testing which type of reasoning can be achieved by a question-answering system (e.g. several support- ing facts, compound reference, positional reasoning, etc). Each task consists in set of stories with an associated question such as:
7% emrrr Imst
VM
Test etrer
60
40
20
"John took the apple. John traveled to the hallway. Who has the apple?"
4 (original)
1000
10000
Number of names
Clearly, we would expect a QA system to be able to an- swer the previous example when "John" is replaced by "Sasha", "Bob", or any possible name even if it has not been seen during training. To investigate whether QA models perform abstraction over symbols, we perform an experiment where the training and test sets of the first bAbi task are regenerated with an increasing number of names. Fig. [] shows how the performance of Memory- Networks (MN) (Sukhbaatar et al.] 2015) and Third-order tensor product RNN (TPR) (Shhlag & Schmidhuber, 2018) dramatically drops as the number of names increases in contrast to their symbolic counter-pan part SMN and STPR proposed in this paper. Both models reach low error well below the 5% limit even when the number names and vocabu- lary become considerably larger than the original task.
Figure 1:
Test error on first bAbi task when increasing the number of names. Error of symbolic models are all bellow 1%.

---

Under review as a conference paper at ICLR 2021
The main contribution of this paper is the proposal of a hybrid semantic/symbolic representation that is equivariant to entity permutation. The main advantage and novelty of our approach is that entities are not required to be identified in advance we rely solely differentiation to determine whether a word acts like an entity. We show how to extend two question-answering models to handle this hybrid representation and demonstrate in extensive experiments the benefit of such an approach: the sample-complxxity is significantly improved, better compositionality is obtained and symbolic models reach better accuracy on the studied tasks in particular when being trained with less training data.
The paper starts by reviewing related works, we then introduce formally what means to permute entities. We then define layers that are robust to such perturbation and show how two recent question- answering models can be adapted in this context. Finally, experiments are conducted assess the benefits of our method.
2 RELATED WORKS
Improving compositionality of neural networks has been an important on-going effort in the past years. The SCAN dataset proposed from Lake & Baroni (2018) initially showed how standard neural networks baselines can fail to generalize to new symbols when learning a set of artificially constructed rules. Several approaches were proposed to solve this issue. For instance, [Lkee (2019) designed meta-learning episodes that led the model to solve the task, (Nye et al. 2020) showed how one could infer symbolic neural programs with similar meta-learning procedure. Alternatively, Gordon et al. (020)) proposed to design an equivariant model (a model whose latent representations unchanged when permuting symbols). A common limit of those approaches is that they require specifying which words are symbols in advance (Lake (2019); Nye et al. (2020) also requires sub- stantial amount of supervision and designing meta-episodes). An exception is Russin al. (2019) which proposed to decompose syntax and semantic for SCAN. None of those approaches can gen- eralize to arbitrary large amount of entities or entities not seen in the training as the one we propose.
The problem of compositionality becomes much easier if symbols (or entities) are detected before- hand. For instance, [i al. (2015) showed that replacing entities by dedicated token placeholders leads to sigmificant improvement in question answering. The same approach has also been applied in Machine Translation and Data to Text generation (errratioonLonggge [uong et al. (2015); (Serban et al. (2016); Lebret et al. (2016), to enable sequence to sequence models to generalize to unseen words at inference time. While specifying entities in advance (or detecting them in a pre- vee-prceessing step with Named-entity recognition (Marsh & Perzanowski 1998)) before applying model may give compositionaliiy. we would clearly want instead models to be able to infer automatically whether a word should behave as symbol or not. While positional encoding (Graves et al. 2014 Vaswani et al., (017) may give some compositionality as allows to reason over positions this solution is not practical for language as anter-wort distances are not fixed. For instance the distance between a noun and its verb varies and positional embedding is not enough to achieve compositionality (Hupkes et al.) 2019).
An interesting line of research have been the study of equivariant models whose representations are invariant (or equivariant) to symmetries present in the data (Zaheer et al. [017]] Ravanbakhsh cet al. 2017). Adding invariance data symmetries has been theoretically shown drastically reduces sample complexity (Sannai & Imaizumi 2019). For instance, convolution neural networks require significantly less training data and achieve much better performance than a MLP as they are invariant to image translation. Gordon et al. (2020) proposed the first NLP model provably capable of handling symmetries between symbols albeit requiring the need specify such symmetries in advance.
Tensor product representation (TPR) Smolensky (1990) allows to stores complex relations between value and variables with distributed representations and offer some compositionality. Recently, Schlag & Schmidhuber (2018) proposed an architecture able to learn TPR parameters by differ- entiation and obtained state-of-the-art results for bAbi at the time of publishing. However, the compositionality of the proposed approach is limited (as shown in Fig. [ by the fact that every entity needs to be seen sufficiently many times so that proper entity vector is found, in addition the model has been shown to learn orthogonal representation for entities which requires as many hidden dimensions as the total number of entities.
2

---

Under review as a conference paper at ICLR 2021
lemon orange
John took the apple. Who took the apple?
r
Sasha took the apple. Who took the apple?
John took the appic. Who took the apple?
zhhn took the orange, Who tock the orange?
the
+
+
Sashaa John
Lisa
Word parameters
John
T
Illustration of symbol-shift equivariance. Left: representation of word parameters (identi- cal words are depicted close so that they can be distinguished), Middle and right: two cases where t is symbol-shift equivariant for two different symbol-shtss.............
Figure 2:
Finally, the VARS approach (Vankov & Bowers 2020) consists in outputting a one-hot vector repre- senting symbolic variable that is randomly assigned to different positions during training to enforce compositionality. While this approach grants some compositionality, the approach is limited as one must draw symbol permutations that each object is seen in all possible one-hot values. In addi- tion, one must specify in advance which object or word behaves as symbol and the method only support symbolic output and cannot represent symbolic inputs nor perform computation with hybrid representation as we propose (combining semantic and symbolic representation).
SYMBOL-SHIFT EQUIVARIANCE
When we learn answer "John" given specific context would like be able answer "Sasha" both names were permuted in the context. In what follows, we introduce the notion of symbol- shift equivariance: e.g. condition restricting possible permutations some permutations may perturb the sentence grammar (permuting "John" by "why") or cause ambiguity (permuting "John" by "Mary" if the question involves gender).
We assume all words are taken from a vocabulary V which is a discrete set of n words. We are interested in providing an answer
given a context consisting a question q= [q....., vma and list of sentences (or stories) x [Z1...., Zy]] with = [Ta..... Tini E V", We denote
(x.4) and (X)) the function that predicts the answer a E V given the context X = (z, 9) E X.
Let r : V + a word permutation. Given a sequence [y1...... yul. we define F(y) [T ) ..... [(y))] where the permutation is applied to each word in the sequence, similarly, ((xr)),T(a))
T(X) = ((T(=)).
Assuming each word has an associated vector parameter, we say that permutation is a symbol-shift it preserves vectors parameters. For instance in Figure 2 a map permuting "John" with "Sasha" is symbol-shift as both words share the same parameters but map permuting "John" and "lemon" is not. Formally, assuming each word I n has an associated set of parameters is E R,,, we say that a permutation I' V V is symbol-shift if o Ur(i) for all
We are now ready to define symbol-shift cquivariance. A critical advantage is that we do not need to specify symmetries in advance between entities, we instead rely vector semantics whose embeddings will be learned end-to-end.
Definition 1. Let 2 X + V a function mapping a context to an answer. We say that & is symbol- shift equivariant for any symbol-shift I and for any X E X,
= ((T(())==I((t
SYMBOLIC QUESTION ANSWERING
In this section, we show how to define cumbol-shift-eqqivariin models. The main idea consists of concatenating two representations, a standard semantic representation in Rd as well as a symbolic representation in R. where m denotes the number of distinct words in the stories and question. The symbolic representation will be made such that for i I m, the i-th component of the symbolic representation corresponds to the i-th word appearing in the context. For instance in Figure B there
= 5 words in the context and "apple" is the fourth word by order of appearance its symbolic
3

---

Under review as a conference paper at ICLR 2021
embedding is the fourth one-hot vector. The model symbolic output has larger probability for the fourth word of the context which is dereferenced to "Apple".
We now describe formally how the symbolic representations constructed and how perform linear transformation and projection back to the original vocabulary. Finally, we derive the sym- bolic counter-part of Memory-Netwond and TPR models that will be proved be symbol-shift equivariant.
Mapping words into and from symbolic representations. words present in the stories z and the question is denoted as
In each input example, the set of
Cx {g} ((2)),
and we let |Cxl be the number of distinct words in the context. To project words to its symbolic representation IR.", we represent each unique word by a one-hot vector representing the position of its first appearance, using the bijection yx Cx [1, m
To dereference a symbolic representation in RTTh back to its vocabulary id, we define the matrix Bg E Roxm that maps one-hot vectors of symbolic representations one-hot vectors in R representing the word in the vocabulary as shown in Fig. such that B - (j) for j I m where
ef I denotes the (-th one-hot vector in 1 R the i-th coordinates of B& E R is given
Note given one-hot symbolic representation by:
[B = (o(() if
(I)
else
hence Bi allows to dereference a symbolic representation ş Rn to a word vector B&P E R".
Hybrid semantic-symbolic embeddings. We embed words with the concatenation of standard semantic word embedding as well as symbolic embedding respectively parametrized by A e Rdxx and E [0, 1) The semantic embedding maps word x E [1,n] Ae, E RR.. While the symbolic embedding of x consists in the one-hot vector of the order of appearance of the word æ in its context multiplied by learned parameter. More precisely, is defined as Ogz)((z)) E IR" where og is an output of a sigmoid unit on learnable parameter, i.e. 0 < ar < 1, that indicates how much each word should behave as symbol and (e(a) is the one-hot vector of the order of appearance of the word in its context. The final embedding of word of then consists in the concatenation of the semantic and symbolic parts:
Ae ① CgC)(() IRd+
where @ denotes the concatenation operator. Note that all parameters are differentiable allowing the model to learn both word semantic and how much each word should behave as a symbol. The symbolic part will be shown allow the model to be robust symbol permutation while being able to represent an arbitrary number of symbols and generalize to new ones. For instance in Fig. B can that permuting "John" by "Sasha" would not change embeddings (as long both name share the same parameters) as both words will still appear in the same order in the context.
Symbolic projection. Given an internal state h haem hsym E Id we interpret it as distribution on the vocabulary E R" with
Prem softmax (Bhsem) Psym = B,softmax (Pryy)
(3)
B Prem + (1 (8) Psym
(4)
where B Raxd and B [0, 1] are parameters to learn.
The final distribution is a mixture of two distributions Psem and Psym- The first one Psem is seen as semantic output as it increases answer probability of word if its semantic embedding is closer
FFrm now on, drop the subscript notation when there is no ambiguity and denote C,yp
2 "we later omit superscript dimension as they will always be implicitly defined
Note that if o 0 for all word x, the model is reduced standard semantic-only model
4

---

Under review as a conference paper at ICLR 2021
John took the apple Who 2
apple
John
orunge
apple
John
orange
apple
John
orange
Byp
soltmax (Bhum) R
Plym By sollmax (hem) I
110.0200
Rren R
Aaya
Model
sem, emh Aur E
R
symb, emb, axpip E Rtm
1-1 0
0
1000
0
0000000
1100010
8-300000 Who
32 1000
3 0
0 apple
apple
Figure Illustration of symbolic representation in a case where 2 and
to the semantic state Isse- The second one Psym interprets Asmm probabilities of words from the context using B to dereference positions. Indeed, denote ý softmax (Nsym) E R and using Eq.
get that the i-th coefficient of Psym is given by
SP)(0 0
if eieC
[Paym]: [Bp
(5)
else
Symbolic transformation. We perform linear transformation of internal symbolic represen- tation = heem + /sym E Rd+m with:
hsem + hsym Wheem @ (AI + +7117) Rd+m
(6)
where W E Baxd A R. y R are parameters learn, I E R is the identity matrix and : E mx The linear transformation for the symbolic part is taken from Zaheer et al. shown be the unique form of linear parametric equivariant function. In case, the symbolic transformation is invariant to permutation of symbolic coordinates, allowing the model be independent from the choice of particular bijection (o
(017) where
4.1 SYMBOLIC MODELS
We are now ready show how question-answerte models can be extended that they become symbol-shift equivariant. We recall the definition of Networks (Sukhbaatar et al. 2015) and Third-order tensor product RNN (Schlag Schmidhuber 2018) models before deriving their symbolic counter-part.
Memory-Networks.. Memory-Networks iteratively updates an internal question representation with K self-attention layers (or hops) before projecting the final representation to the answer distribution. The model parameters consists in K + embedding matrices A(o), ...., A(K) Rdx=, The query representation initially set to ne Σn= ( eaz and iteratively updated with:
m, = (k)
Σ
Exgy
C:
A (k++1) Cast
j=l
jml
n (k+1) u (R)
softmax
(mf up(1)
(8)
In words, the internal representation is updated with the vector of output memories weighted by similarity between the current question representation u; (k) and the input memory m,, The final internal representation (K) Rd is mapped to the answer distribution with:
= softmax (A(K) (K(
E T
so that the probability of every word being the answer is a function of the similarity of its embedding (K) and the final question representation u ( In addition the paper proposes temporal and positional encoding allow distinguishing words stories order which discuss in the appendix.
5

---

Under review as a conference paper at ICLR 2021
Symbolic Memory-Networks. We extend Memory-Networke by using K 1 symbolic embed- dings from Eq. and concatenating semantic and symbolic representation. This model has the parameters A (k) Raxd and (k)) E Ih" for S k < K as well 8 E [0, 1]. Instead of mapping internal representations into Ed map them to Id with:
T4
n1
(K) Σ e 2 (o(o))) G
Σ 1=1
A (1++))
(0-1)) Fay @ 2 Cp((ss)
m,
=
The internal representation updated K times with Eq. to obtain a final internal representation (K) e BR To obtain the output predictive distribution, we project (K) RRd as described Eq. 4 with 8" (K) and | E [0, 1].
S
TPR. Third-ordee tensor product RNN proposes to encode stories with a non-standard RNN whose representation a tensor product representation (Schlag & Schmidhuber (018). More precisely, the model state consists in a tensor denoted as F; E x that is initialized with zeros and updated at each story with:
Fe = F1 AF;
The update term A&, depends on a learned parametric representation of entities and roles that are obtained by mapping the story representation 8, with MLPs. The story representation is obtained by summing the k words of a story with 8, = プド Aez.; O Pj where 3 denotes the component- wise product, denotes an embedding matrix and are positional embeddings. Once the story representation is obtained, entities and roles representations are obtained with MLPs:
e = (sc E R r = fr (st 0,c)) E RR
(10)
for 1 < j < 4 and where f is MLP and @ its parameters. The update term AFr given by a close form formula designed to update entity information into F; given the entity and
role embeddings ef") and (() detail it in the appendix for space reasons.
The internal representation Fe is updated after reading each story and to perform inference, the final internal representation Fr is used to decode the entity and role representation of the question. Similarly to the story embedding, entities and role representations of the questions are first obtained by mapping the question embedding sQ = =b Acg, O Pj through MLP:
fa(:;; 0n) E R 1j (sQ: 0z,) E R 15j<4
(11)
Given those representations, the distribution over possible answer is obtained by first obtaining the following internal representations:
7(1) (Fr n) 34 (1) ((2) = (Fr (()) 34 12)) g(a) = (Fr ((0) 34 ((3)
(12)
where *34 denotes tensor inner- product, finally the answer distributions is obtained with the follow- ing projection:
softmax
B
LN(2
E R
(13)
where LN denotes layer-normalization (Ba et al. 2015) and B E RRIXE are projection parameters. For MLPs, [Scllag Schmidhuber (2018) proposes to use two hidden layers with internal dimension d and tanh activation functions, hyperparameters of the method are given in the appendix. The pa- rameters of the models are word and positional embeddings, MLPs parameters as well as projections parameters.
We modify TPR to handle symbolic representations for entity, role as well as intermediate representations. All entities, roles and intermediate representations embedded into pd+mm Stories are embedded symbolically with:
Symbolic TPR.
(Aezy 2 Pp) @ (awrr Ep(zx))) 1R
St

---

Under review as a conference paper at ICLR 2021
1K examples
10K examples
40
MN STPR
35
Test errar
25
20
500 1000 1500 2000 2500 3000 3500 4000
500 1000 1500 2500 3000 3500 4000
Figure 4: Learning curves of average accuracy when training with 1K and 10K examples of bAbi.
Then, we use symbolic MLPs to find entity and role representations:
= (se fgw) E Bdtm, r = fr (st: (,6)) gd+m
(14)
where i denotes a symbolic MLP with linear transformation described in Eq.
The tensor representation updated with the same equations as TPR obtain final representa- tion Fr E R(d+m)" Given the symbolic embedding of the question sQ = -E @ pạ) @ (og, (p(a;)) )) we extract symbolic roles and entities with symbolic MLP:
π = (xQ: 0n) E Rd+m
1, = fi, 01,) E Rd
(15)
We then obtain the hidden representation h an answer distribution as described in Eq. where B E Raxd 33 E [0, 1] are additional projection parameters.
Ck=1 LN( gd+m with Eq. 12 that is projected
3
key result justifying the symbolic qualifier that SMN and STPR are symbol-shift equivariant. Theorem. SMN and STPR are symbol-shifi equivariant.
The proof is in the appendix for space reasons. The main idea consists in remarking that embed- dings are invariant to symbol-shift and consequently, latent representations invariant too since the model is deterministic. The proof then shows that having identical latents gives equivariance given our symbolic projection construction.
We observe that in practice, the learned semantic embeddings of different entities will not be exactly the same in Figure in particular the semantic embeddings of "apple" and "orange" may be close but different for instance. Our experiments will show that having symbol-shift equivariant models is good inductive bias improving accuracy and compositionality of the studied models even if entities vectors are not exactly aligned.
EXPERIMENTS
We perform experiments on bAbi tasks (Weston et al. 2013) with version v1.2 of the dataset. The initial dataset consists in set of 20 synthetic question-answering tasks designed test capabilities of a dialog agent. In each example, an answer must be provided given a question and stories con- sisting in sequence of sentences. We the versions which contains IK and 10K training examples per task respectively. Given our computing budget, we perform all experiments in the single-task setting where models are trained on the 20 tasks independently. Every experiment repeated with 10 different seeds and we report mean/std over those runs.
The performance of MN, SMN, TPR and STPR is reported in Fig. and Tab. Fig. depicts the average error per task over time and shows that symbolic approaches significantly improves the sample efficiency of both methods which achieve much faster convergence rate and also converges to better values. In Tab. [ we report the average error obtained at convergence for all methods. Again, symbolic models SMN and STPR outperforms their non-symbolic counter-part.

---

Under review as a conference paper at ICLR 2021
original test-set
2010 sero-shot room test-set
zcre-abot object testosct
zcro-shot people sest-sct
John took apple kinchen.
John took upple kitchenette.
John took kitchen
Sasha took apple kicchen bedroom..
John bedroom. Where the apple? (hedrocus)
guest Where the apple" (guest-room)
hedroom. Where the key? (bedroom))
Where is the apple? (bedrocus)
Figure 5: Illustration of zero-shot test-datasets
Table 2: Test error when training on original bAbi tasks and evaluating error on different test-dataset. For zero-shot test-datasets, test entities (e.g. rooms, objects, people) are seen during training.
lask
5
original
room
object
people
original
rJom
object
peuple
average
MN
21.8 +0.4
99.5 +0066
38.2 -13
67.1 =288
14.8 +277
14.9 +1.
48.7 +14
24.6 +0.8
41.2
SMN (ours)
3.4 40.6
100.0 =0.0
62.7
2233
62.1 =1.1
6.9 +2.3
7.6
42.8 115
22,9 113
38.5
IPR
1:3.4
130.3
65.4
26.3 17.6
STPR (ours)
0.0 10.11
0.1 1.0.2
22.8 2.3
18.9
124.1
4,3 122
2,3 10.0
14.4 128
7,9 15.0
8,8
Because the embeddings of entities learned by TPR are orthogonal (Schlag & Schmidhuber 2018), we argue that most dimensions are used to represent this basis (requiring least as many dimensions as the number of possible entities) while STPR require less dimen- sions since orthogonal representation are available the model by construction. To see this effect, we run smaller model with = and dropout set to for both STPR and TPR that are called respectively STPR- sm and TPR-sm. Results indicate that STPR-sm per- forms almost well in the 10K setting while reach- ing an average test error of 5.6% when trained with 1K examples opposed to TPR-sm whose performance is severely hurt by the dimension reduction as it cannot represent orthogonal basis for the different entities any- more. To the best of our knowledge, the best reported result with 1K examples 9.9% from QRN model (Seo) et al. 2017) who reported the best run over 10 seeds while we report average result over seeds. Using same procedure, we obtain a test-error 1.8% and pass 19/20 tasks (passing means reaching an enror lower than 5%). This makes significant improvement to the problem of passing all task with only IK examples but the problem still stands (in particular when reporting mean test-crror instead of best error over multiple seeds).
Table 1: Aggregate error bAbi.
# examples
IK
10K
MN
27.3 +
15.6 + 0.3
SMN
25.9 0.4
14.4
+ 1.8
TPR
21.7 2.0
2.2 1 0.9
STPR
20.3 1 1.7
0.7
1 0.6
TPR-sm
17.1 1.0
13.0 1 1.6
STPR-sm
5.6 0.8
3.1 0.4
TPR best
14.77
0.14
TPR-sm best
13.56
7.65
QRN-2r best
9.9
4.6
STPR best
11.65
0.06
STPR-8 sm best
1.79
1.82
Zero-shot entities. In Tab. show the accuracy obtained on two bAbi tasks in zero-shot setting where we leave the training dataset unchanged but perturb the test dataset by introducing unseen entities. Precisely,
replace the 6 rooms present in the task (kitchen, bedroom, office, garden, hallway, bathroom) by kitchenette, guest-room, open-space, entry, terrace, toilet only in the test set, see Fig. 5 We also measure test-accuracy when introducing unseen people and objects the test set with the same procedure (both tasks contain objects, rooms, and people). For all models, the semantic embeddings of words unseen in the training are set to the zero vector. Symbolic models outperform their semantic-onke only counter-par part in this setting as they can perform abstraction over symbols rather than relying only on specific embeddings for each name.
Experiment details. Hyperparameters in bAbi are kept identical across tasks, and we reused hyperparameters of (Sukhbaatar et al.) 2015) for Memory-Networte and |Schlag & Schmidhuber (018)) for TPR (they are given in the appendix). For symbolic variants, we use the same hyper- parameters the semantic version. We used the public implementations (Junki 2015) and (Alexan- der 2019) for MN and TPR based on Pytorch (Paszke et al. 2019) that we adapted to support symbolic computation.
8

---

Under review as a conference paper at ICLR 2021
CONCLUSION
We introduced a novel hybrid semantic and symbolic representation able to handle internal trans- formations and projections of the final representations back to the initial vocabulary in a way that is symbol-shift equivariant. The main advantage of our approach is that we rely on differentiation to detect whether words should behave as symbols and therefore sidestep the need to detect or specify entities in advance.
Our experiments showed that having symbol-shift equivariant models can significantly decrease the amount of training data required. In particular, we were able to solve 19/20 bAbi tasks using only 1K training examples. We also showed that our approach performs well in challenging zero-shot settings or when the number of entities in task becomes very large.
An interesting area for future work consists in extending other architectures such as Transformer Vaswani al. 2017) models with the hope of diminishing the vast amount of data they currently require. Finally, another interesting application will be use this symbolic representation order to ensure fairness in question answering other applications.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Podolskiy Alexander. Tensor product representation recurrent neural network, pytorch implementa- tion. https://gttbb.ommm///oook///t/tr Torro 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer Normalization. 2015. URL
ht.tss
wwwpiiiwwrrgppppo.. wttp://oggppp/////oo////pp// 06450.
Jonathan Gordon, David Lopez-Paz, Marco Baroni, and Diane Bouchacourt. Permutation equivari-
ant models for compositional generalizzinnnnn in language. In International Conference on Learning Representations, 2020. URL https::/.//.........pt httt:p::://peereeeeeeee net/ Stteee//oorumml--y.yWWeeeeee
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing Machines. Arxiv, pp. 1-26, 2014. httt::::/aaaiii org/ abs/ 1110/4110.004
Dicuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: how do neural networks generalise?, 2019.
Ohmura
Junki.
Memory networks, pytorch 2015.
implementation.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequema recurrent networks. 35th International Conference on Machine Learn- ing, ICML 2018, 7:4487-4499, 2018.
Brenden M Lake. Compositional generalization through meta sequence-to-sequenec learning. 906 05381
(NeurIPS):1-122, 2019. URL htRLiitttt://axxiv.ooo
Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 1203-1213. The Association for Computa- tional Linguistics, 2016. doi: 10.18653/v http:////11111l111 128.. URL https: http://oiooooo//////// wtt.or///.............1. v1/a16-1128
Yujia Li, Daniel Tarlow, Mare Brockschmidr, and Richard Zemel. Gated Graph Sequence Neural Networks. (1):1-20, 2015. URL/tttp://axxiv www.aaawwi..oooaaaooaoaaoae/aeee abs/aass/111 05493
Thang Luong, Ilya Sutskever, Quoe Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 11-19, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.31155v 1002. URL http:: //ww:
agl web. labbborg//nttto ogy/P15-1003
Elaine Marsh and Dennis Perzanowski. MUC-7 evaluation of IE technology: Overview of re- sults. In Seventh Message Understanding Conference (MUC-7): Proceedings of a Conference Held in Fairfax, Virginia, April 29 May 1, 1998, 1998. URL https: //wp:////wwwww//////////////// ac web org/
Maxwell I. Nye, Armando Solar-Lezama, Joshua B. Tenenbaum, and Brenden M. Lake. Learning compositional rules via neural program synthesis, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Detito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. http://aapprreaa 8024 8035. Curran Associates, Inc., 2019. URL htttp:://papess..euu... wttp:pppc.cppppp..ppp// 1:15-pytorch an htte--imeettteetetteeee or00- hhgh htg.rpatteeetttteeeeeoeeeeeeerre
haw.aamiinnn-bbbrr.....m Librar.r......
pdd
10

---

Under review as a conference paper at ICLR 2021
Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos. Equivariance through parameter-
sharing. 34th International Conference on Machine Learning. JCML 2017, 144416-4425 2017.
Jake Russin, Jason Jo, Randall C. O'Reilly, and Yoshua Bengio. Compositional generalization in deep seq2seq model by separating syntax and semantics, 2019, URL http://arxiv.org/
abs 1904. 09708
Akiyoshi Sannai and Masaaki Imaizumi. Improved Generalization Bound of Group Invariant / Equivariant Deep Networks via Quotient Feature Space. pp. 1-18, 2019. URLpttpp//aaxiv.
htt//bb//1991 06552
Imanol Schlag and Jürgen Schmidhuber. Learning to reason with third order tensor prod- ucts. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Biances and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 9981- 9993. Curran Associates, Inc., 2018. URL resor--tsssrpproouuccts.
8203 www.naaarnnnnnnnn to reason seeeasonnwith--ae
Minjoon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. Query-reduction networks for question answering. In ICLR, 2017.
Julian Vlad Serban, Alberto García-Durán, Çaglar Gülgehre, Sungjin Ahn, Sarath Chandar, Aaron C. Courville, and Yoshua Bengio. Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus. In Proceedings of the 54th Annual Meeting of the Asso- ciation for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10. 10.18653/v 1056.
URL
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text. 2019. UKl/ttp::/arxxv.org
abs/1908 1108 06177
Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial Intelligence, 46(1-2):159-21 1990. ISSN 00043702. doi: 10.1016/0004-30000009000000
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory net- works. Advances in Neural Information Processing Systems, J44a222442224424444,,,,,,, 2015, ISSN 10495258.
Ivan I. Vankov and Jeffrey Bowers. Training neural networks to encode symbols enables combi- natorial generalization. Philosophical Transactions of the Royal Society B: Biological Sciences, 375(1791):1-16,, 2020. ISSN 14712970. doi: 10.1098/rstb.20199 9.0309.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. (Nips), 2017. ISSN 1469-8714. doi: 10.1017/509525238 10000000000
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy /5... 556.05698, 2015.
tasks. arXiv preprint /502
Manzil Zaheer, Satwik Kottur, Siamak Ravanbhakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J. Smola. Deep sets. Advances in Neural Information Processing Systems, 2017- Decemmii:3392-3344 3402, 2017. ISSN 10495258.
11

---

Under review as a conference paper at ICLR 2021
John
r
3
o r
|
Sasha
Sasha
Figure 6:
Illustration of proof notations.
A PROOF OF SYMBOL-SHIFT EQUIVARIANCE
Theorem. SMN and STPR symbol-shift equivariant.
Proof. Let us take + E (SMN, STPR) and show that for any symbol-shift and context X E X, (T)()) T(T(X)) Since the output of ④ is considered as distribution words, e.g. vector (XX) E R", we first define the action of I on vector R" by defining e' = [(v) as the vector verifying V/(0 U2 or equivalently of = Ur 1()) for all i S We then have by definition [T(((X))]: = [ (r))
Since T is symbol-shift, all word parameters (which comprises A, B and a) equivariant, e.g- for each word < n:
Aro) Ai, Br()) Biy aT() a4
(16)
Denote & Cx + m] and xt Cr(x) [1, ml the two bijections mapping context words in Cx and Cr(x) to their order appearance, see Fig. Using the fact that permuting words does not change entity order of occurrence, we have:
(17)
We first show that the embedding defined in Eq. 2 is invariant to symbol-shift, e.g. that E(x) E(I(=)) for all x E where E(x) = Ae + ag ag (z) Using Eq. 16 we get that Ae, = Aer(z)- get that (T()))C(((((
Then using Eq. 17 and 16
(p()))(pp)) agCC(())
Consequently, since embeddings are invariant by symbol-shift and since the model is deterministic, the same latent representation is obtained for both X and I(XX. If we denote the latent function as ((X)= = Isem (X) @ Msym()) we then have:
h(X) = -({(X))
(18)
Given that the output is defined as
((X) fssoftmax (Bhaem (X)) +1 B3 x (heym (X))
sufficient to show that the two functions fi(X) = Blsen (X) and f2(X)
Be x softmax (/sym (X)) are equivariant as softmax and (constant) linear transformations are trivially
cquivariant. The following sequence of equalities show that fi (X) is equivariant, it uses B, = Br and the fact that the latent function is invariant.
[(f(x))): = [BBh (X))ss
(19)
Blisem (X) (X)lr-1())
(20)
Br 169) // (X)]a
(21)
Bik (nsem (X)] 1=1
Σ
(22)
[Bhsem (x)]
(23)
Bhsem (T(X))
(24)
(T)))
(25)
12

---

Under review as a conference paper at ICLR 2021
It remains to show that f2(X) is equivariant. Denoting the probabilities symbols p softmax (Nsym (X)) = softmax ( e R and using Eq. we obtain
ix else
(D
if E Cr(x)
[f2(X)];
[f(((()));
00
10
else
Then, for any S n,
[x)(( (2(X))r-10)
(26)
PPPT ((0)
ifT- E Cx
(27)
10 else
(Ppr if Cr()) 1o else
(28)
[2(T(X)))):
(29)
B ARTIFICIAL DATASET EXPERIMENT
To investigate how models generalize with different number entities, we generate artificial datasets where questions have the form of:
U1
IN UN. =?
where v, is the last value assigned to x; and ø, Vy, V/ for all j. In Fig. 7 we study how well models perform when the number of variables and values increase. Three datasets generated with [Vi = [V2| k for k {10, 100, 1000), each dataset has 10K training examples and IK non-overlapping examples in the test set and each story has N 10 assignments. While the task elementary, both MN and TPR struggle generalize to larger number of entities in contrast to their symbolic counterpart SMN and STPR that can solve the task even when with thousands of entities.
where expect V
|V]|
100
|
1000
100
100
80
80
80
Test crror
Test CFTOr
60
60
60
TFA
40
symbol commrryymmmryyyooom rebrok
50
100
Iterration
Iteration
Iteration
Figure 7: Artificial dataset error convergence when increasing the number of values and variables.
C EMBEDDINGS LEARNED BY MEMORY-NETWORK.
Fig. shows the word embeddings of the first layer of a MN after learning the first bAbi task generated with 10k names. The words on the upper-left are verbs, stop-words and locations while the words on the bottom-right are names (the figure is zoomable). The model tries to diagonalize entities embeddings given the available dimensions (e.g. finding orthogonal embeddings for names). This means that attention models may also require number of dimensions at least as large as the number entities and hence struggle to generalize to large number of entities or entities not seen at training.
13

---

Under review as a conference paper at ICLR 2021
400
k/a
Mcc
Figure 8: Correlation matrix of word embeddings of the first layer of MN when trained first bAbi task generated with IK names. The learned embeddings of names are approximately orthogonal.
Table 3: Per task error on bAbi with training examples.
model task
MN
SMN
TPR
TPR-sm
STPR
STPR-sm
0.4 0.3
0.0 0.1
0.0 0.0
1.0 + 2.2
0.0 0.0
0.0 + 0.0
71.0 3.1
69.4
27.4 33.7
6.9 18.3
2.1 + 1.3
76.5 + 1.2
7.88
73.6 2.6
68.4
6.51
8.8
21.6.0.7
2..5
0.1
2.2
0.1
0.0
5
22.3 0.9
21.6
2.1
2.6
4.6
2.2
23.8 10.8
16.7
19.9
12.6119
24.0
4.1
23.4
1.00
16.5
102228
17.9
1.8
17.6
1.
104427
4.4
1.6
0.9
9
19.6
1..
31.7
21.6
3.3
4.9
10
23.7
19.1
39.1
21.2
30.3
1..2
3.3
10.8
2.
1.0
7.6
1.2 1.2
0.2
0.0
0.2
0.1
3.2+3.1
2.1 0.8
6.1
6,8133
8.1
4.9006
7.5
1.6
14
3.6
10.9877
14.4
213138
10.3
18
15
0.1
0.0
0.0
0.3
0.0
0.0
1.4
1..5
2.1
1.0
39.1
0.3
47.0
46.1
4.
32.6+22
4..1
24.4 13.0
18
489
48.1
3.8
1.9
5.5
0.9 1.2
19
1.11111
74.3288
88.6
75.8 6.88
86.7
4 14.
20
0.0 0.1
0.0
0.0 0.0
0.4
0.0
0.0 0.0
14

---

Under review as a conference paper at ICLR 2021
Table 4: Per task emrrr bAbi with 10K training examples.
model task
MN
SMN
TPR
TPR-sm
STPR
STPR-sm
0.0 = 0.0
0.0 0.0
0.0 3 0,0
0.000.11
0.0 0.0
0,0 + 0.0
2
23.1 = 1.7
25.1 + 27.0
0.1 0.1
0.3 0.3
0.3 0.2
0.6 0.5
29.4 = 2.0
23.4 1.6
0.3
22.1
0.5
6.1
22.0
21.7
0.0
6.8
4.5
0.0
13.9
6.6
0.8
0.7
0.6
1.
0.8
7.6
0.
1..1
0.4
1.2
0.5
6.1
0.5
6.0
0.3
0.6
3.4
2.6
0.8
3.7
0.9
0.3
0.3
2.
0.
2.,5
0.6
1.0
10
1.0
3.1
0.4
1922
0.6
3.9
11
0.2
1.7
0.4
0.3
0.1
0.2
12
0.1
0.0
0.2
6.9
0.3
1.7
1.
5.4
0.3
4.0
0.5
0.2
14
0.3
0.1
0.1
37.1
0.2
1.3 2.
15
0.0
0.0
0.0
0.1
0.0
0,0 0.0
16
5.
31.6
0.3
5.2
0.3
0.0
4.8
2.
1.93.0
6.96
0.3
1.2
28.0
1.4
0.3
0.5
0.5
0.2
19
25
6..2
35.8
11.0
1.3
42.5
20
0.0
0.0
0.0
0.3
0.0
0.0
D EXPERIMENTS DETAILS
In all experiments, used batch-size of 32 and ADAM with learning-rate of 0.001. In the case TPR models, we follow [Schlag & Schmidhuber (2018) and set 31 0.6, Bg = 0.4. Gradients norm clipped to 5.0. All models trained with early-stopping using 10% of the training set as validation, and we perform 80000 gradient updates 2 times decaying the learning rate by 2 each time (which is setup close both Sukhbaatar al. (2015) that decays the learning rate 4 times and Alexander 2019) that trains for significantly longer).
The main difference in the implementation we used for TPR are that RADAM optimizer was used in Schlag & Schmidhuber (2018), we used ADAM instead to have the same setting with Memory- Networks and also because this optimizer is not available in Pytorch, For Memory-Networks, we reuse Junki (2015) implementation where temporal encoding regularization is not present.
Memory-Networks temporal encoding To allow models to preserve temporal information, Sukhbaatar et al. (2015) proposes to have additional parameters Ta R and Tc E RTxd where T denotes the maximum number of stories and add them to m,, G follows:
m;
(1)) Cxy + +Tl(t)
(30)
j=1
0 =
(k+1)
Exi To(()
(31)
j=1
In the case Symbolic Memory-Networks, we proceed similarly with an additional symbolic tem- poral encoding parametrized with a
RT and oc RT and update m,, G follows:
15

---

Under review as a conference paper at ICLR 2021
(A Ezi +Ta(i)) @ (2 e
(32)
j=1
4
(Alk+1) Ezy + Tc(()) @ (ac(1) (4+ e ep((a))
(33)
In addition both Memory-Networks and Symbolic Memory-Networks use the positional embedding used in TPR.
E TPR UPDATE EQUATIONS
After reading each story St. Fr is updated by adding the sum of three tensors AF; = W, + M2 + B. whose expression is given by:
wi = (Fi 31 Ei
(1)
23 Ti
(1)
(34)
Wr (e, 8r wz + (c ( ort (2)
(35)
m = (Fi 33 23
(36)
Mc = (e (1)
m
+ (et
(1)
@ )))
(37)
b. = (Fi 34
(38)
B, (e. (2)
0
(39)
Where @ denotes the tensor outer-product and `i denotes the tensor inner-product.
The term We is a write term that allows to encode the association (e (1) (2)) while retrieving the previously assigned entity and removing its previous association. The term M, allows asso- ciate this removed entity to a different relation r: (also removing the previously assigned to r (2)) Finally the term B, called backlink it allows associative association, refer the original paper Schlag & Schmidhuber (2018) for detailed intuitions on the three update terms.
16


Reviews:
[{"review_id": "vR4_t_lmP5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes to improve the generalizability of bAbi models through [entity permutations].\nMore specifically the approach assumes domain knowledge of word/entity type equivalences, which helps restricting possible permutations between word POS (e.g., “John” vs “why”) or gender (e.g., “John” vs “Mary”). Each word type has its own embedding param and is concatenated with normal word embeddings to form the final word representation. Experiment with memory networks and Third-order Tensor Product RNN shows that the proposed approach indeed enables the models (especially TPR) to handle artificial data sets with large number of entity names. \n\nOverall I find the proposed research not very well motivated. Leveraging word type knowledge to improve the generalizability of NLP models has been a popular and effective approach. Commonly used strategy is to replace named entities in sentences with their word type tokens . e.g., from [how old is Obama] to [how old is PERSON]\nhttps://arxiv.org/abs/1601.01280\nhttps://arxiv.org/abs/1611.00020\nThe proposed approach seems to achieve a similar effect, but is a lot more complex.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nJ1dB0KSpcK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a network that is equivariant to entity permutations without requiring the pre-specification of the set of entities. To this end, the authors propose a hybrid semantic-symbolic embedding which they integrate into two QA models. Finally, the authors show significant gains on the bAbi tasks, with especially impressive gains in the 1K setting. \n\nThe problem is quite interesting and challenging in the setting where entities are not prespecified.\nHowever, given the model description it is not clear at all how the model is able to learn a symbol-shift equivariant embedding. \nI don't understand how the model is able to determine that \"apple\" and \"orange\" have the same embedding while \"apple\" and \"John\" have different embeddings. \nWhat is the loss/model architecture/data augmentation guiding this? How is the model able to figure out that \"John\" and \"Sasha\" share embedding?\n\nApart from the high level details, I don't understand the following notations and operations:\n* In Section 4, what is $n$? Is it total number of words in the sequence?\n* If $B_\\varphi \\in R^{m \\times n}$  and $e^m_j \\in R^m$, the multiplication $B_\\varphi e^m_j$ doesn't make sense.\n* How exactly is $\\alpha_x e_{\\varphi(x)} \\in R^m$?  What exactly is $\\alpha_x$ and what is it's shape? \n\nThe notation and the working of the model is not clear to me, hence, I am giving a low rating for now.\nApart from this I also doubt the proposed method's generalizability beyond toy settings.\n\n\n\n\n \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5Y8sB-Z26HA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**.\nThis paper proposes a new type of models that are equivariant to entity permutations, which is an important criterion to build language models that can easily generalize to new entities. The authors modified a Memory-Network and a Third-order tensor product RNN to make them symbolic-shit invariant. The new models were evaluated and compared on the 20 bAbi tasks. Results show that the symbolic versions of the models yield better performance than the original ones.\n\n**Positives**.\nThe topic is of great interest and it is indeed crutial that neural language models become symbol-shift invariant to allow them to better generalize. This work is clearly motivated.\n\n**Confusions**.\nThe beginning of Section4 mentions that the main idea of this work is to concatenate a regular \"semantic\" word vector with a \"symbolic\" representation essentially corresponding to a one-hot vector of the token order of appearance.\nIn the following paragraphs, the work presented lacks clarity and seems to over-complicate concepts with hard-to-follow math notations. For instance, the “*Mapping words into and from symbolic representations*” paragraph introduces tedious math notations to describes something simple that was clear before, namely, the mapping from tokens to their respective symbolic vector, which is simply defined as the one-hot vector position appearance of this token in the context.\nSimilarly, the \"*Hybrid semantic-symbolic embeddings*\" paragraph uses again tedious math notations to describe how semantic and symbolic embedding are concatenated.\n\nGiven the confusion presented in Section4, it is currently not clear how adding a one-hot vector to the input embedding can make a neural model symbol-shift equivariant.\nIn particular, below are the two things I could not understand:\n1) The paper mentions that \"*all parameters are differentiable*\". It is not clear if that also includes the symbolic representation or not? If so, then the initial one-hot vector may not be a one-hot vector after the gradient updates performed during training, which would result in a non-symbolic representation? if it is kept fix during training, then it is not clear how it is used by the network.\n2) In addition, assuming that the symbolic representation of all tokens stays the same during training, I don't see how \"_permuted symbols share the same latent representations_\" if the latent representations are made of both on-hot vectors **and** regular word vectors. I understand that the symbolic representation does not change for a permuted word since it will appear at the same place as the original word. But the semantic representation will be different. For instance, the semantic word vector of “banana” is similar but still different than the word vector of “apple”.\n\n**Conclusion**.\nI would suggest the authors to simplify their mathematical notation and make their paper easier to read. As of now, I could not fully understand the paper and unfortunately for that reason could only put a score of 4 with a low confidence of 2.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a  new type of models that are invariant to entities by exploring the symbolic property of entities. This problem is important in language modeling since it gives intrinsically more proper representation of sentences, which can better generalize to new entities.  However I still suggest to reject this paper for the following reasons 
1. The description of model is not clear enough which can certainly use a serious round of revision.
2. The experiments on bAbi is not convincing enough since it is an overly simple and toyish data-set with many ways to hack
3. Similar entity-invariant idea has been explored long time ago by  (https://arxiv.org/pdf/1508.05508.pdf) which attempted to represent entities as “variables”


Decision: Reject