A Simple Recurrent Unit with Reduced Tensor Product Representations
Authors: Shuai Tang
Paul Smolensky
Virginia R. de Sa
Venue:  (2020)
Keywords: RNNs
TPRs

Abstract: Widely used recurrent units, including Long-short Term Memory (LSTM) and Gated Recurrent Unit (GRU), perform well on natural language tasks, but their ability to learn structured representations is still questionable. Exploiting reduced Tensor Product Representations (TPRs) --- distributed representations of symbolic structure in which vector-embedded symbols are bound to vector-embedded structural positions --- we propose the TPRU, a simple recurrent unit that, at each time step, explicitly executes structural-role binding and unbinding operations to incorporate structural information into learning. The gradient analysis of our proposed TPRU is conducted to support our model design, and its performance on multiple datasets shows the effectiveness of it. Furthermore, observations on linguistically grounded study demonstrate the interpretability of our TPRU.

Under review as a conference paper at ICLR 2020
A SIMPLE RECURRENT UNIT WITH REDUCED TENSOR PRODUCT REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Widely used recurrent units, including Long-short Term Memory (LSTM) and the Gated Recurrent Unit (GRU), perform well on natural language tasks, but their ability to learn structured representations still questionable. Exploiting reduced Tensor Product Representations (TPRs)- distributed representations of symbolic structure in which vector-embedded symbols are bound to vector- reboodee struc- tural positions we propose the TPRU, simple recurrent unit that, at each time step, explicitly executes structural-rob binding and unbinding operations to incor- porate structural information into learning. A gradient analysis of our proposed TPRU is conducted to support our model design, and its performance on multiple datasets shows the effectiveness of our design choices. Furthermore, observations on a linguistically grounded study demonstrate the interpretability of our TPRU.
INTRODUCTION
Recurrent units are widely used in sequence modelling tasks, including text, speech, DNA/RNA sequencing data, etc. Due to their ability to produce distributed vector representations for sequences with various lengths, and their property of being a universal function approximators (Schäfer & Zimmermann| 2006), they quickly succeeded in numerous research areas. The training algorithm, backpropagation through time (BPTT, Werbos et al., 1990), has remained the same across the years, with known gradient vanishing and exploding issues. Many theoretically justifiable empirically useful regularisations (Pascanu er al.] [013)] Kanai et al.) (017) have been proposed to ease the learn- ing process and to smooth the gradient flow for long sequences, and informative analysis methods have been conducted to visualise the information encoded in the learnt vector representations (Zeiler & Fergus 2013 Karpathy et al. 2015). The main difficulty in understanding the learning process of particular RNN unit comes from the extreme nonlinear activation functions, which cause the gra- dient flow to be chaotic. Here, we aim to propose recurrent unit that has fewer nonlinear operations by incorporating Tensor Product Representations (Smolensky 1990).
Tensor product representation (TPR) is an instantiation of general neural-symbolic computing in which symbol structures are embedded in vector space via filler-role decomposition: the struc- ture is captured by a set of N roles r. (c.g., left-child-of-root), each of which is bound to a filler f; (a symbol substructure). A TPR embedding of symbol structure S derives from vector embed- dings of the roles (r:) and their fillers [fr} via the outer or tensor product: S = Σ r. 8 fi = EM,r, r.T.T = RFT, where R and F respectively denote matrices having the role or filler vectors of as columns. Each r; @ fa is the embedding of a role-filler binding: a constituent; & is the binding operation. The unbinding operation returns the filler of a particular role in S; it is performed by else 0.
the inner product: f. u, S, where is the dual of ri, satisfying u. ry ōig 1 if i Letting U be the matrix with columns } we have U R=
Our goal is to propose a recurrent unit that uses a reduced version of a tensor product representation, vector rather than 2 tensor; this arises by reducing the dimension of the filler embeddings to I; with role embeddings of dimension d, the representation becomes b = >> ==1 for, E Rd. We let this sum range over all N possible roles, letting fi 0 when role is unfilled. Then at time the hidden state of our RNN - which we call the binding complex - will be b, where f. is the vector of filler values (fr); R at time f. Now binding is performed by a simple matrix-vecte product, and unbinding by simple inner product:
(f()ar
b Rff
f. = U
(I)

---

Under review as a conference paper at ICLR 2020
In this paper:
In the context of related work (Sec. 2 and based on the reduced tensor product representations introduced above, we propose gated recurrent unit, the TPRU (Sec.B), along with set of analyses of the gradient flow to justify our decisions in architecture design (Sec. 4 Even though the number of nonlinear operations is reduced, the success in experiments attests the flexibility of our TPRU in learning and our analysis on derivatives further supports it.
Our proposed TPRU demonstrates strong performance the Logical Entailment task (Evans) ett al. 2018), Natural Language Inference datasets (Williams et al. 2017), and language modelling, while having comparably many parameters the GRU, and significantly fewer than the widely-used LSTM (Sec.5) 5).
The instantiation of reduced TPRs allows our model learn decomposable representations. and the filler ft provides an interface for us to inspect the information encoded in b, (Sec. (6) We conducted a linguistically grounded study, which shows that our TPRU is capable of distinguishing multiple senses of a polysemous word and discovering structured information in sentences.
2 RELATED WORK
Recent efforts on understanding RNNs includes these three classes: (i) theoretical analysis through advancements in dynamic systems on the functions learnt by RNN units (Chang et al.) 2019); (ii) linguistically grounded analysis by developing sets of probing tasks to illustrate the properties of leartt vector representations of RNNs (Conneau et al., 2018 2 Chrupata & Alishahi| 2019); and (iii) introducing inductive biases into the architecture of networks and comparing the behaviour of learnt models against those of widely-used counterparts. Our work falls in this last category: learning structured representations by incorporating the inductive biases inherent in TPRs.
Some prior work has incorporated TPRs into RNN representations. Question-answering SQuAD (Rajpurkar et al. 2016) was addressed (Palangi et al., 2017) with an LSTM with hidden state given by single TPR binding; the present work deploys complexes containing multiple bindings (albeit reduced ones). TPR-style unbinding was applied in caption generation (Huang et al. 2018). but the representations were deep-learnt and not explicitly designed to be TPRs in the present work. A contracted version of TPRs. Holographic Reduced Representations, was utilised to decompose input space and output space with filler-role decomposition (Luo et al.) 2019). Our work differs from these prior papers in that TPR filler-role binding and unbinding operations are explicitly carried out our proposed recurrent unit, and the two operations directly determine the update the hidden states. Closest to the present work is Schlag & Schmidhuber (2018), which performed question- answering by using TPRs to embed simple knowledge graphs as tensors; our model does not deploy graph-based inference, which is less natural for the inference tasks address than it is for the guestion task addressed in |Schlag & Schmidhuber (2018).
Our proposed TPRU inherits the explicit binding and unbinding operations in TPRs in the sense that a set of role vectors are shared across time steps and sequences to serve as a basis for composing distributed vector representations. We provide set of analyses from both theoretical and empirical perspectives: gradient analysis of our model helps us make sense of the architecture design, while empirical results from three different categories of tasks show the effectiveness of the proposed model. Additional study on the linguistic and learning aspects provide evidence that our model indeed inherits the merits of TPRs, which enhance interpretability and the capability of learning structured representations.
3 PROPOSED RECURRENT UNIT: THE TPRU
As shown in both LSTM (Hochreiter & Schmidhuber 1997) and GRU (Chung et al. 2014) design, gating mechanism helps the hidden state at the current time step to directly copy information from the previous time step, and alleviate vanishing and exploding gradient issues. Our proposed recurrent unit deploys single gate, adopting the input gate of the GRU, omitting the reset gate. We expect that further augmenting our TPRU to 2 gates as in the GRU, or gates as in the LSTM, will lead to yet better performance.
2

---

Under review as a conference paper at ICLR 2020
At each time step, the TPRU receives two input vectors: be-1 B the binding complex from the TPRU the previous time-step, and R from the external input; produces b. E Lt An input gate gi E Ied is computed to produce a weighted sum of the information generated at the current time step, b, E RR, and the previous binding complex bi-1.
b. b. (1 gt) b where gr a (W; Wg.:),
a() is the logistic sigmoid function (applied element-wise), is the Hadamard (element-wisel prod- uct, and W, E Ra and W. _axd" are matrices of learnable parameters. As now explain, the calculation of b, is carried out by the unbinding and binding operations of TPRs (Eq. 1b.
3.1 UNBINDING OPERATION
Consider set of hypothesised unbinding vectors U = |/1,, u,, uN] E RdxN At time step i, these can be used to unbind fillers fo, from the previous binding complex b.-1 using Eq. [ We posit two matrices V, Rdxd' and V. Raxd that transform the current input æt TRd and the binding complex b into the binding space g yiclding fillers fa,t and fu,
f = U VibL.
== Va ER"
Normalisation is applied for stable learning by gradient descent, and the specific technique (Yang et al. 2018) we adopted here also enforces sparsity on both fb, and fa.::
(f.). (F.)2. "000/>"=(ff)) (f.)
(4)
where (ft).. = max(0, (fo,t) bb) + max(0, (fz,t)n + bz)
(5)
which by and b, are two scalar parameters for stable learning, and (-), refers to the nith entry of the vector in the parentheses.
3.2 BINDING OPERATION
Given hypothesised set of binding role vectors R= [r1, r2...., rx] R applying the binding operation in Eq. I to the fillers fi at time gives the candidate update b, for the binding complex,
b.
Rf:
(6)
The gating mechanism controls the weighted sum of the candidate vector b, and the previous binding complex b to produce a binding complex b, at the current time step, as given by Eq. 2
3.3 UNBINDING AND BINDING ROLE VECTORS
In the TPRU, there is a matrix of role vectors R used for the binding operation and a matrix of unbinding vectors U used for the unbinding operation. To control the number of parameters in our TPRU, rendering it independent of N, instead of directly learning two sets of vectors, set of N d- dimensional random vectors E is sampled from normal distribution, and two linear projections W., W. E ddd are learnt to transform E to U E.E and R = W.E.
Therefore, in total, our proposed TPRU has six learnable matrices: W.. W., VI, W, E Raxd and V.. W E dxxd", The number of parameter matrices the same as that of GRU and less than that of an LSTM, and, when d' < d, the number of learnable parameters is less than that of a GRU.
GRADIENT ANALYSIS AND COMPLEXITY
This section analyses the derivatives in our proposed TPRU, and aims to justify the architecture design. For simplicity, we omit the derivatives w.r.t. bias terms in our derivations as biases can be absorbed into the weight matrices by adding an extra component to the input.
As known, the gradient of an RNN cell is accumulated time ühg -[- 8b; ing the properties of the derivative of the output the current time step (Eq.. w.r.t. the output
0b therefore, study-
Øb.
3

---

Under review as a conference paper at ICLR 2020
of the previous time step b is important for understanding the behaviour of recurrent unit. For our TPRU, the derivative shown below:
ab.
ab. 0g: ab. ab. a((1 - (()1-g))a b A, + A ab.-1
ab:
As
(7)
ab.1 0g. db:-1 ab. Tb.-1
Öb.-1
where A,, A2 and As diagonal matrices in B Bdxd in which (A1)a (b.): )( (1-(())))))) (.) (A2)i
(gi), and (Aa)a = (1 (g.).). It clear that the gating mechanism smooths the gradient flow as jitters the Jacobian matrix by adding small values to the diagonal terms, and from a dynamic system
perspective, it helps stabilise the learning system (Moya-Cessa & Soto-Eguibar (011). Now we
focus on the following derivative which carries the main information of our proposed TPRU. TPRU.
ab. ob. 0f: Of.. (W, (J.N.S...) (E' W. V.)
(8)
bb,-1 of. af., ūb,-1
where N,, Sb,t E NNN are diagonal matrices, in which (Ne) 2/(f),, and (Se,t) = (Js.)) and (Je)mn = (6 (f),)))). Appendix [ shows that J, is a positive semi-definite matrix, and its eigenvalues are all smaller than max.. (fo,))
There are three crucial design choices, including (1) learnable parameter matrices W, and We to transform the basis matrix E separately, (2) learnable linear transformation V. on the binding com- plex at each time step b,, and (3) the relu-squared normalisation in Eq. [ The following analysis shows that certain modifications could lead to issues during learning, and helps understand the ne- cessity of each component.
Modification I. W, W., V, I, and softmax normalisation in Eq.
The derivative of the softmax normalisation of outputs w.r.t. the inputs is simply Jr as Sa. N: In Given the conditions, the derivative can be simplified to:
/b./0b.-1 = W,EJ:N:S,, W. V. = W,EJ,E' W, 00
(9)
In this case, as the derivative gets multipled several times by BPTT, the gradient w.r.t. each parameter matrix either explodes or vanishes depending on the spectrum of E, Also, a careful initialisation is required otherwise the gradient could overflow or underflow after the first iteration.
Modification II. W. W.. and softmax normalisation.
Here we allow learnable parameter matrices V,, to transform the binding complex from the previous time step b thus, the derivative of b, w.r.t. b and that i-th row of Vi are shown below:
0b./0b.1 = W,EJ,E' W. V. 0b./0(V)). = EW, W. ).:b.-1
(10)
Since W, W. - O, /b./0(V)), always has a non-negative inner product with b.-1, which means that the update of V. always contains positive portion of b. The same situation also applies when updating Vz as it keeps a positive portion of x. as well. It limits the learning system from eliminating accumulated useless information. Although it is still possible that, after backprop- agation through several time steps, the positive portion of b could be suppressed, we set W, and W. be independent matrices so that the above constraint isn't imposed.
Modification III. Softmax normalisation.
The only modification here is to replace the normalisation step in Eq. in our TPRU softmax function that also normalises f. form distribution f.. At each time step, the derivative of b, w.r.t. any parameter matrix before the normalisation step contains 0ft/Oft. and in the softmax normalisation, only the relative difference, (f (f.) Vn (1, ..., N}, matters.
Therefore, there is no explicit regularisation in the derivative of ab. = E,EJ,E' W. V. to penalise extreme values in F., which could lead to gradients with large norms and make the learning process unstable. This may also explain why, in the transformer module (Vaswani al.) 2017). layer normalisation (Ba et al. 2016) is necessary, as it scales the gradient by the variance of the input.
We don't 't rule out the possibility that there exists suitable initialisation which gives stable learning and decent performance, but the normalisation step we adopted from prior work (Yang et al. 2018 avoids heavy tuning of initialisations for stable learning.
Iy is an identity matrix in R N
4

---

Under review as a conference paper at ICLR 2020
Ours. Here, we recall that the derivative of /b://b:-1 is in Eq. The term N; scales the derivative
by the magnitude of the input f: to Eq. 4 which regularises the norm of the gradient to each parameter matrix. Specifically, the following derivative indicates the update of i-th row in W.:
ab, ab. ofc. bb. Öfa. a(W.). 8fc. O(W.). 0f.. a(W.).
W,EJ,NN (V...(Vbb.1). + S..:(Vaar)) E (11)
where N, normalises the contribution from fo.: and fE.: and S and S., select positive contribu- tions only, thus the normalisation step in Eq. helps the gradients w.r.t. parameter matrices to stay stable. Eq. IT also shows that the update of each row W. depends on a sparse linear transfor- mation of W,, which could potentially avoid overfitting as not all dimensions in the representation space are utilised and our TPRU learns to discover the representation space.
Given the derivative above, it is easy to see that the presampled matrix E acts similarly as the preconditioning matrix in solving large linear systems, and with samples from a normal distribution zero-centred uniform distribution, it smooths the spectrum of the Jacobian matrix and stabilizes the learning system.
Time Complexity in Forward Computation
Table 1: Complexity
Table I presents the time complexity of calcu- lating each of our proposed TPRU, LSTM and GRU for single time step. In the calculation, we only kept the terms that involve second or- der of variables. As shown, the time complexity of TPRU is between LSTM and GRU when
Model
Time Complexity
Purams
TPRU
O(2Nd + 6d))
4d + 2dd'
LSTM
O(8d)
+ Add'
GRU
O(6d)
38 + 3dd'
the number of role vectors N is less than the di- mension of the recurrent unit However, the number of parameters in our TPRU is fewer than in an LSTM, and comparable to a GRU.
In addition, as concerns nonlinearities, our TPRU contains max(x,, 0)2 only N times, and sigmoid functions only d times; this is fewer than sigmoid/tanh squashing functions appearing 5d times in an LSTM and 3d times GRU. The nonlinear operations are indeed reduced our TPRU, and the following sections show that our TPRU still provides strong performance on evaluation tasks.
5 TASKS AND TRAINING DETAILS
Three entailment tasks, including an abstract logical entailment task (Evans et al. 2018) and two relatively realistic natural language entailment tasks, including Multi-genre Natural Language Infer- ence (MNLI, Williams et al. (2017)) and Question Natural Language Inference (QNLI, Wang et al. 2018 ). as well as a language modelling task on a large corpus and small one, (WikiText- 103 and WikiText-2 (Merity al.) 2018)), are considered to demonstrate the learning ability of our proposed TPRU. Each of the three entailment tasks provides pairs of samples, and for each pair, the model needs to tell whether the first (the premise) entails the second (the hypothesis). Language modelling is used to demonstrate the ability of our proposed model on sequential modelling.
Experiments are conducted in PyTorch (Paszke et al.) 2017) with the Adam optimiser (Kingma & Ba 2014) and gradient clipping (Pascanu et al.) 2013). Reported results are averaged from the results of three random initialisations. Our proposed TPRU has only input gate and also has similar number of parameters, so the GRU is the most relevant comparison partner. LSTM is also included it is widely-used
well and it generally yields better performance than GRU does. For each model, hyperparameters are searched based on the small selected validation set on each task, including dropout rates, learning rate and batch size. For entailment tasks, we tested both plain and BiDAF architectures (Seo et al. 2017). On both NLI tasks, ELMo (Peters et al.) 2018) encoding applied for embedding words into contextualised vectors. More details about the tasks and the training details can be found in Sections B and in the Appendix.
6 DISCUSSION
On the Logical Entailment task in Table when it comes to larger dimensionality (N 512), our TPRU appears be more stable than GRU during learning as observed that the GRU with the
5

---

Under review as a conference paper at ICLR 2020
Table 2: Accuracy of each model on the propositional Logical Entailment task. Overall, our proposed TPRU outperforms its closest comparison counterpart GRU, and provides comparable performance the LSTM. The value besides "TPRU" indicates the number of role vectors. Results of BiDAF models are parenthesised. 'Mean 2 is the mean number of worlds of proposition- values; here and below, bold indicates the best result among all models with the same architecture.
test
model
valid
easy
hard
big
massive
params
Mean 2 Var
75.7
81.0
184.4
3310.8
848.570.0
5.8
Plain (BiDAF) Architecture dim 64
GRU
84.3 (91.3)
68.2 (76.0)
84.3 (91.))
77.0 (80.0)
57.4 (70.4)
66.8 (74.3)
49. (172.4k)
TPRU- 512
88.6 (90.9)
73.1 (77.18)
88.4 (90.6)
79.0 (82.0)
62.0 (69.9)
71.8 (74.9)
98.3k (213.0k)
LSTM
88.6 (89.1)
76.3 (75.4)
88.4 (89.1)
79.0 (75.0)
64.7 (66.7)
76.4 (74.9)
131.1k (327.7k)
BiDAF architecture failed to converge in two out of three trials. On certain test sets, our (singly- gated) TPRU slightly underperforms the LSTM, possibly due to the three gates (and more parame- ters) of the LSTM: future research could incorporate more sophisticated gating mechanisms into the TPRU.
On the MNLI tasks in Table our proposed TPRU consistently outperforms the LSTM when large number of role vectors deployed. We omit the results from GRUs as they are worse than LSTMs and often unstable in learning. Unexpectedly, all models perform better the mismatched dev set than on the matched one, perhaps because it happens to be casier.
Table 3: Results on MNLI and QNLI Our TPRU outperforms LSTM. The number role vectors in each of our models is indicated by the value beside "TPRU", As shown, our proposed TPRU overall outperforms LSTM on two tasks.
dev
model
matched
mismatehed
QNLI dev
params
Plain (BIDAF) Architecure dim 512
TPRU 256
74,2 (77.7)
74.9 (78.0)
78.2 (81.2)
TPRU 512
74.8 (78.1)
75.6 (77.8)
78.8 (80.1)
74.4 (77.7)
74.8 (78.5)
(80.8)
8.4m (16.8m)
LSTM
(76.5)
73.9
77.2 (79.1)
12.6m (27.3m)
Plain (RiDAF) Architecture dim 1500
TPRU 512
74.5 (77.9)
75.4 (78.8)
79.3 (82.5)
TPRU 1024
74.8 (77.9)
76.1 (78.3)
80.4 (82.2)
TPRU 2048
75.1 (78.7)
75.2 (78.8)
80.0 (82.5)
48.3m (108.4m)
LSTM
74.8 (77.1)
75.7 (77.7)
78.8 (81.2)
60.6m (1 (162.m)
(ALICE ensemble)
On the language modelling task on WikiText- 103 and WikiText-2, our TPRU provides similar per- plexity on the test set to the LSTM, as shown in Table The learning plot in Figure shows that our TPRU learns faster on the training set than the GRU and the LSTM. The learning plot and the performance table both serve as evidential support for the gradient analysis of our model.
6.1 POLYSEMY
In TPR, multiple senses of a single polysemous word can be encoded by binding the same symbol that represents the polysemous word to different role vectors {ri). TPR composing a distributed representation that keeps multiple senses of polysemous word and recovering each sense with no information loss if large number of dimensions is available. In our case, the magnitude of each entry in determines the contribution of each role vector to the update candidate b, (Fqq. 6).

---

Under review as a conference paper at ICLR 2020
Table 4: Perplexity of language modelling on WikiText-2 and WikiText-103. (Lower is better.) (left): Our TPRU with 1024 role vectors gives similar perplexity score as the widely used LSTM does. (right): Negative Log-likelihood on training set of language modelling task on WikiText- 103. (The plotted models are trained without dropout layers.) Our TPRU converges faster on the training set than the GRU and LSTM. As the GRU doesn't generalise very well on the dev test, its results are omitted from the table.
model
WikFToxt-103
WikiText-2
#Params
GRI-1524
LSTM-1024
TPRU 1024 64
Two-layer Architecture - dim 1024
TPRU-1024-
TPRU
41.4
76.8
12.6m
Nega
LSTM
42.3
76.0
16.8m
SOTA
16.4 Krause al. 2019
39.1 (Gong et al.
2018
257m / 35m
15
0.0
25
5.0
3.5
10.2
12.5
Epochs
We train our TPRU with BiDAF architecture on the MNLI dataset, and word vectors are obtained from fastText (Bojanowski et al. 2017) as the input to model so that the vector representation of polysemous word stays the same even when it is presented in different contexts, After learning, model provides interface for us to check whether specific role vector correlates to specific sense of polysemous word in different phrases in the dev set by computing S. arg max,, (fr)n.
For example, common polysemous word bank has two senses when it is used noun. In the dev set of MNLI, the only case when bank is used in the river-context is in the phrase "bank fishing". and the rest belong to the other sense in the context of economy. We observed that /3 is the same when bank occurs in "bank fishing" and we denote it as Biiver. thus, P(3 = fiver |bank in the river-context) = 1. It is also important see if the specific Briver is dedicated to the word bank in the river context. After examining the dev set, we have P(bank in the river-context/
S = P(bank in "the west bank"| (merr = 0.3; clearly, the sense of "bank" relevant the phrase "the west bank" is the one related to rivers, not finances.
6.2 POS TAGGING
We calculated the pointwise mutual information (PMI) between the following two terms: 1) The maximally selected role vector at each time step 3, arg (f.)), and 2) The part-of-speech tag (POS Tag) provided by pretrained tagger[ For given POS tag, the PMIs of role vectors are sorted in a descending order, and Table 5 presents the top two PMI values for those POS tags for which these two values differ by at least 0.5: this includes 7 out of 17 fine-graines POS tags. The idea is that if the top PMI value is sufficiently greater than the next one, and hence all the rest, then it is justified to say that each POS tag in the table shown below has role 'dedicated' to it.
Table 5: PMI between POS Tags and Maximally Selected Role Vectors.
POS Tags
ADV
DET
INTJ
NOUN
PART
PROPN
VERB
Top 2 PMI
7.37/ 5.83
6.06/5.47
/.14
/.88
7.54 6.91
7.27 6,44
7.22/6.28
As the model is only trained on MNLI to predict the entailment relationship between given pair of sentences without access to POS tags, is interesting that some role vectors become associated with specific POS tags. Since POS tags carry structural information of given sentence, and since the roles learnt by the TPRU are informative about the POS tags, we have evidence that the roles that explicitly define the structure of the model's representations capturing some of the distinctions between words that are believed to be fundamental to linguistic structure.
As pointed out in prior work, models trained MNLI focus on words in lexical (non-functional) categories in determining the entailment relationship between a sentence pair; here we see that POS tags of lexical words, including NOUN, PROPN, VERB, etc., have larger gap between the top highest PMI values as they are apparently emphasised by the learning signal. This finding suggests
https://ppacciioapp//oo

---

Under review as a conference paper at ICLR 2020
that our learnt model might help identify the biases in datasets which could be helpful in improving fairness and inclusiveness.
The above two empirical observations point out that our proposed TPRU inherits the interpretability of the original TPRs, and provides a certain level of human- cumaan-eadability.. provides a simple interface for to analyse the learnt model through the explicit binding and unbinding operations.
6.3 EFFECT OF INCREASING THE NUMBER OF ROLE VECTORS
In the original TPRs, roles are typically filled by at most a single symbol, so the total number of distinct roles indicates the number of symbols that be used in the representation of single data item. Since each symbol is capable of representing specific substructure of the input data, increasing the number of role vectors eventually leads to more highly structured representations there is no limit on the dimensionality of role vectors.
Experiments are conducted show the effect of increasing the number of role vectors on the perfor- mance on both Logical Entailment and MNLI. As shown in Table adding more role vectors into our proposed TPRU gradually improves the performance on the two entailment tasks.
Figure 1 presents the learning curves, including training loss and accuracy. of our proposed TPRU with varying number of role vectors on the two entailment tasks. As shown in the graphs, incor- porating more role vectors leads to not only better performance, but also faster convergence during training. The observation is consistent on both the Logical Entailment and MNLI tasks.
120
n48
024
24
(a) Logical Entailment
(b) MNLI
(c) Logical Entailment
(d) MNLI
Training Plots of our TPRU with different number of role vectors (The plotted models are trained without dropout layers.) In general, our proposed recurrent unit converges faster and leads to better results on both tasks when more role vectors are available, but the total number of parameters remains the same. (Best viewed in colour.)
Figure 1:
Interestingly, on the Logical Entailment task, the performance of our proposed TPRU with only eight role vectors matches that of a GRU with the same dimensionality. The number of role vectors required in our TPRU to approximately match the performance of GRU and an LSTM on the MNLI task is much less than the dimension of the hidden states, which suggests that the distributed vector representations learnt in both the LSTM and GRU are highly redundant and can be reduced fewer dimensions; this also implies that the LSTM and GRU the representation space even when the dimensionality of the model is larger than that required for solving the task. Meanwhile, our proposed TPRU explores sparse weighted combinations of role vectors in Eq. [ and it helps the model to specialise individual role vectors for different purposes, which eventually leads to more effective exploitation of the representation space.
able to extensively exploit
CONCLUSION
We have proposed a simple recurrent unit that leverages the explicit binding and unbinding opera- tions in previously proposed TPRs in a reduced fashion, therefore it is named the TPRU. The explicit execution arms the proposed model with the ability learn decomposable representations, and by sharing the role vectors across time steps and sequences, our TPRU provides research an intuitive interface through to study the information encoded in the learnt distributed vector representations.
Since recurrent neural networks are prone to suffer from the gradient vanishing and exploding issues, conduct thorough analysis of the derivatives and the gradient update of our proposed TPRU on
8

---

Under review as a conference paper at ICLR 2020
each time step. The conducted analysis supports our decisions in designing TPRU, and it shows that it encourages relatively more stable gradient flow through BPTT. The time complexity for executing one step of our TPRU is lower than that of LSTM, and it has the same smaller number of parameters than the GRU does, which justifies the simplicity of our recurrent unit.
The empirical evaluation five learning tasks in three different categories demonstrate the effec- tiveness of our TPRU as it performs on a par with or better than the LSTM. The study on increasing the number of role vectors and plots on the loss curves during learning shows that our model con- verges faster than the GRU and the LSTM in general and it is a result of the stable gradient flow through multiple time steps. In addition, our proposed TPRU enables faster convergence rate and better performance by increasing the number of role vectors.
study on correlating the role vectors with linguistic properties of languages shows that our TPRU inherits the strong interpretabiliity of TPRs. In the toy example involving a polysemous word bank, our model can differentiate multiple senses of the word, and can assign a similar sense, even in named entity, to its closely related one. Also, the PMI values between POS tags and maximally se- lected role vectors imply that our TPRU is capable of discovering syntactically-struuuure linguistic information.
Given the current pressing need for interpretable machine learning and learning decomposable rep- resentations, our proposed TPRU can be regarded more liggh- and numan-readabll alter- native to the widely-used LSTM or GRU. Future work will focus on incorporating full TPRs into the design of an RNN unit and into the Transformer module.
REFERENCES
Jimmy Ba, Ryan Kiros, and Geoffrey E, Hinton. Layer normalization. 2016.
CoRR, abs/1607.06450,
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. TACL, 5:135-146, 2017.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. EMNLP, 2015.
Bo Chang. Minmin Chen, Eldad Haber, and Ed H. Chi. AntisymmetricRNN: A dynamical system view recurrent neural networks. In International Conference Learning Representations, 2019. URL
net/ Forummidddooooo....o........o. ryx.podxxxxpppccx. ottpr//www..ttttouooooooeorrrrrr
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open- domain questions. ACL, 2017.
Grzegorz Chrupata and Afra Alishahi. Correlating neural and symbolic representations of language. In ACL, 2019.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 2014.
Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. ACLC 2018.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR, 0bs//810004880 2018.
Richard Evans, David Saxton, David Amos, Pushmeet Kohli, and Edward Grefenstette. Can neural networks understand logical entailment? In ICLR, 2018
Cheng Yue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. Frage: Frequency-agnostic word representation. NeurIPS, 2018.
Sepp Hochreiter and Juergen Schmidhuber. Long short-term memory. Neural Computation, 9: 1735-1780, 1997.

---

Under review as a conference paper at ICLR 2020
Qiuyuan Huang, Paul Smolensky, Xiaodong He, Li Deng, and Dapeng Oliver Wu. Tensor product generation networks for deep NLP modeling. In NAACL-HLT, 2018.
Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. CoRR, abx/1611.01162, 2016.
Sekitoshi Kanai, Yasuhiro Fujiwara, and Sotetsu Iwamura. Preventing gradient explosions in gated recurent units. In NIPS, 2017.
Andrej Karpathy, Justin Johnson, and Fei-Fei. Visualizing and understanding recurrent networks. ArXiv, abs/1506.002778 2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980) 2014.
Ben Krause, Emmanuel Kahembwe, lain Murray, and Steve Renals. Dynamic evaluation of trans- former language models. CoRR, ahss/1904008775 2019.
Jiaming Luo, Yuan Cao, and Yonghui Wu. Towards decomposed linguistic representation with holographic reduced representation, 2019. URL httpps::/oppereeiiiett it rk13 2-13-hA5Y7
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2018.
Héctor Manuel Moya-Cessa and Francisco Soto-Eguibar. Differretiall equations: an operational approach. Rinton Press, Incorporated, 2011.
Hamid Palangi, Paul Smolensky, Xiaodong He, and Li Deng. Deep learning of grammatically- interpretable representations through auestion-answerring CoRR, 701100555084,, 2017.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. the difficulty of training recurrent neural
networks. In ICML, 2013.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Matthew F., Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, 2018.
Ofir Press and Lior Wolf. Using the output embedding to improve language models. In FACL, 2017.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. CoRR, 2018
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP, 2016.
Anton Maximilian Schäfer and Hans Georg Zimmermann. Recurrent neural networks are universal approximators. In International Conference on Artificial Neural Networks, pp. 632-640. Springer, 2006.
Imanol Schlag and Jürgen Schmidhuber. Learning reason with third order tensor products. In Advances in Neural Information Processing Systems, pp. 9981-9993, 2018.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 2016.
Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannanch Hajishirzi. Bidirectional attention flow for machine comprehension. 2017.
Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artif. Intell., 46:159-216, 1990.
10

---

Under review as a conference paper at ICLR 2020
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs/1 00s//804,7741,, 2018.
Paul Werbos et al. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 18010); 1990.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. broad-coverage challenge corpus for sentence understanding through inference. CoRR, abs/1 704.05426, 2017.
Zhilin Yang, Junbo Jake Zhao, Bhuwan Dhingra, Kaiming He, William W. Cohen, Ruslan Salakhut- dinov, and Yann LeCun. Glomo: Unsupervisedly learned relational graphs as transferable repre- sentations. CoRR, abs/1806..o 306.05662. 2018.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. ArXiv, abs/1311.2901, 2013.
APPENDIX
A PROOF
1. Jj is PSD matrix. For any non-zero vector v, we have
N
J.v Σ 2 Σ (f))))n,))
(12)
(fn))) 2 - (fo
(Jensen's Inequality)
(13)
Therefore, Jr is PSD matrix.
B DETAILS ABOUT TASKS
Three entailment tasks, including an abstract logical entailment task and two relatively realistic natural language entailment tasks, as well language modelling task on large corpus and a small one, are considered to demonstrate that the learning ability of our proposed TPRU. Each of the three entailment tasks provides pairs of samples, and for each pair, the model needs to tell whether first (the premise) entails the second (the hypothesis). Language modelling is used demonstrate the ability our proposed model on sequential modelling.
As our goal is to learn structured vector representations, the proposed TPRU serves function, which learns to process a proposition or sentence one token at a time, and produce a vector representation. During learning, two vector representations given a pair of samples, then a simple feature engineering method (e.g. concatenation of the two representations) is applied form input vector for subsequent classification. In general, with simple classifier, e.g. a linear classifier or a multi-layer perceptron with a single hidden layer, the learning process forces the encoding function produce high-quality representations of samples, either propositions or sentences; better representations enable stronger performance.
encoding
produced by the same recurrent unit
B. LOGICAL ENTAILMENT (PROPOSITIONAL LOGIC)
In propositional logic, for a pair of propositions, and B, the value of F= is independent of the identities of the shared variables between A and B, and is dependent only on the structure of the expression and the connectives in each subexpression. Because of a-equivalence, pA q holds no matter how replace variable p or q with any other propositional variables Logical entailment is
For example, p q A

---

Under review as a conference paper at ICLR 2020
thus good test-bed for evaluating model's ability carry out abstract, highly structure reasoning (Evans et al. 2018).
Theoretically, it is possible construct truth table that contains rows/worlds all possible com- binations values of variables in both propositions A and В; the value of A F can be checked by going through every entry in each row. As the logical entailment task emphasises reasoning connectives, it requires the learnt distributed vector representations to encode the structure any given proposition to excel at the task.
The Logical Entailment datase has balanced positive (A F B) and negative (A F B) classes and representative validation set. Five test are provided evaluate the generalisation ability different difficulty levels: some test sets have significantly more variables and operators than the training and validation sets (see Table
B.2 NATURAL LANGUAGE INFERENCE
Natural Language Inference determining whether a premise sentence entails hypothesis sen- tence - combines the structure-dependdnnc of logical entailment with the NLP tasks of inferring word meaning in context as well the hierarchical relations among sentence constituents. Two datasets are considered, which are Multi-genre NLI and QNLI (Williams et al. 2017: Wang et al.) 2018). Compared logical entailment, the inference and reasoning in NLI also relies the iden- tities of words in sentences in addition to their structure. More importantly, the ambiguity and pol- ysemy of language lead the impossibility of creating a truth table that lists all cases. Therefore, NLI is an intrinsically hard task.
The Multi-genre Natural Language Inference (MNLI) dataset (Williams al., 2017) collects sen- tence pairs in ten genres; only five genres are available in the training set, while all ten genres are presented in the development set. QNLI (Wang al. 2018 is collected from Stanford Question An- swering dataset (SQuAD) (Rajpurkar et al. 2016)) on which a model needs to predict whether the context sentence contains the answer to the question sentence or not. The relation between sentences in pair must be classified as Entailment, Neutral or Contradiction.
The performance of a model on the five 'mismatched" genres that exist in the development but not the training set reflects how well the structure encoded in distributed vector representations of sentences learnt from genres seen in training generalises to sentence pairs in unseen genres. As the nature of NLI tasks requires inferring both word meaning and structure of constituents in given sentence, supervised training signals from labelled datasets enforce an encoding function to analyse meaning and structure the same time during learning, which eventually forces distributed vector representations of sentences produced from the learnt encoding function to be structured. To the extent that the inferences required in an NLI task are demanding of proper analysis of the logical structure implicit in sentence, an inductive bias that enhances the ability to learn to encode the structures of sentences should enhance success on the MNLI task.
B.3 LANGUAGE MODELLING
Language modelling is a fundamental task to test generalisation and sequential modelling. Recent research has shown that models trained on language modelling tasks yield representations that can transfer well to other natural language tasks (Peters et al. 2018 Radford et al. 2018 Devlin et al.) 2018). We follow the autoregressive setting of decomposing the joint probability on tokens into product of conditional probabilities on generating the current token given tokens in the left context.
P( 122, WL) II P(we w1-1, u7 21-2.... 101)
(14)
This requires the learnt representation to capture the potentially highly structured information aceu- mulated prior to the current time step in order to make confident prediction of the current token. A proper inductive bias towards constructing structured representations should enhance performance on the task. Two corpora used in our experiment WikiTextt-03 and WikiText-2 (Merity al.) 2018). in which one contains 103 million tokens the training set while the other only has 2 million. The results will tell how well our proposed TPRU generalises across different training scales.
12

---

Under review as a conference paper at ICLR 2020
C DETAILS ABOUT ARCHITECTURE
C.I PLAIN ARCHITECTURE INFERENCE:
For both the Logical Entailment task and the MNLI task, we train two-layer recurrent networks built using our proposed TPRU, networks built using its direct comparison counterpart, the GRU, as well as LSTMs. A global max-pooling over time applied the hidden states produced by each recurrent unit at all time steps to generate the vector representation for given logical proposition or sentence. Given pair of generated representations u and e, a vector constructed represent the difference between two vectors (u; v; /u v|; v). where o is the Hadamard product and Iu - o) is the absolute difference, and the vector is fed into a multi-layer perceptron which has only one hidden layer with ReLU activation function. The feature engineering and the choice of classifier are suggested by prior work (Seo et al. 2017 Wang et al. 2018).
Symbolic Vocabulary Permutation (Evans et al. 2018) is applied data augmentation during learn- ing on the logical entailment task: exploiting a-equivvlenne, this systematically replaces the vari- ables in a proposition pair with randomly sampled alternative variables, only connectives matter presents the results. For the MNLI task, the Stanford Natural Language Infer- ence (SNLI) dataset (Bowman et al., 2015) is included additional training data as recommended [Wang et al. 2018), and ELMo (Peters et al. (0188) is applied for producing word vectors. The best model is chosen according to the averaged classification accuracy on the five matched and five mis- matched dev sets. The results are presented in Table 3 SOTA models come from Alibaba DAMO
on this task. Table 2
NL.P Group, and the results are available on the GLUE leaderboard.
C.2 BIDAF ARCHITECTURE INFERENCE:
Bi-directional Attention Flow (BiDAF) (Seo et al. 2017) has been successfully applied to various NLP tasks (Seo et al. 2017 Chen et al. 2017), and provides strong performance NLI tasks (Wang et al. 2018). The BiDAF architecture contains a layer for encoding two input sequences, and another for encoding the concatenation of the output from the first layer and the context vectors determined by the bi-directional attention mechanism. In our experiments, the dimensions of both layers are set be the same, and the same type of recurrent unit is applied across both layers. The same settings are used for experiments on LSTT and our TPRU models. Specifically, for TPRU, the recurrent units in both layers have the same number of role vectors. Other learning details are as in the plain architecture. Tables 2 and respectively present results on the Logical Entailment and the MNLI task, with BiDAF results in parentheses.
C.3 LANGUAGE MODELLING:
On the language modelling task, the same two-layer architecture (Merity al. 2018) is adopted in our experiments on LSTM and our proposed TPRU. For WikiText- 103, Bytepair encoding (BPE) vocabulary with 100k merges (10k for WikiText-2) are applied to encode the corpus into indices (Sennrich et al.) 2016). The output prediction layer is tied to the input word embedding layer as advised in prior work (Press& & Wolf 2017 Inan et al.) 2016). Each model is trained for 64 epochs on the training set; the best one is selected based on perplexity the validation set, and then evaluated on the test set. Ssows the best performance of each model after the hyperparameter search.
D POLYSEMY
In the dev set of MNLI, the word bank occurs in the following sentences:
1. according to this plan areas that were predominantly arab the gaza strip- the central part of the country the northwest corner, and the west bank were to remain under arab control as palestine while the southern negev desert and the northern coastal strip would form the new state of israel.
13

---

Under review as a conference paper at ICLR 2020
0.40
Traenng tat
(44
Trenno
Epocha
Epochs
(a) Plain Architecture
(b) BiDAF Architecture
(c) Plain Architecture
BiDAF Architecture
Figure 2: Training Plots for comparing the single- and eight-slice dot-product operations of our TPRU, with both plain and BiDAF architectures. (The plotted models are trained without dropout layers.) Overall, the eight-slice dot product, which enhances the fillers from one- to eight- dimensional vectors, leads to faster convergence rate and better performance on the training set when the number of role vectors is limited, and its effect diminishes when the number of role vectors reaches large value. and '4 respectively refer our TPRU with four role vectors and single- or eight-slice dot-productt (Best viewed in colour.)
2. well uh normally like to to go out fishing boat and uh rather than like bank fishing and just like you try and catch anything that 's swimming because 've had such problems of trying to catch any type of fish that uh just really enjoy doing the boat type fishing.
3. the other bank pays the fund interest based upon tiered account levels, more typical of large commercial account.
The above sentences three examples, there other sentences in which the word bank occurs. As we show in the main paper, after learning, our TPRU categories the first two cases, in which one refers the West Bank of the Jordan river, and the other refers the bank in the river context, as the same sense, as there is no clear learning signal that specifies the first one as a named entity, and categorises the third sentence into another sense.
E EFFECT OF INCREASING THE DIMENSIONALITY OF FILLERS/SYMBOLE
Both fillers and roles in TPRs (Smolensky 1990) are tensors with arbitrary dimensions, which em- powers the representation ability TPRs. In our case, as we aim produce a distributed vector representation of the given input sequence rather than tensor representation, the fillers are reduced to filler numbers, which case, the symbol represented by each filler is limited to one-dimensional vector.
In our proposed TPRU, the unbinding operation instantiated by the dot-product finner-produce operation between the binding complex and the unbinding vectors. One plausible way to upgrade filler numbers to filler vectors is to split the binding complex into multiple slices, and split each unbinding vector accordingly: then the dot-product is carried out each pair of slices rather than the entire vector. In this case, the dimension of the resulting filler vectors determined by the number of slices predefined prior to learning, and is limited by the dimension of the binding complex. The normalisation step is operated on the same dimension on multiple filler vectors independently and in parallel, which results in multiple normalised probability distributions, Each binding role vector is split into the same number of slices, and each distribution is applied to bind the same slices from all binding role vectors into a single slice. All slices are concatenated together to form single vector which has the same dimensionality as the multi-head attention mechanism prior work and has been shown to be effective in terms of performance gain without additional computational cost (Vaswani et al. 2017). It was widely adopted in prior work on Transformer networks for various tasks.
the input binding complex. A similar idea was proposed
In our implementation of the Logical Entailment task, the binding complex is split into eight slices, each with the same dimension. Figure 2 shows training logs with eight slices and also with only one slice for three different numbers of role vectors in our proposed TPRU. In general, shifting from filler numbers to filler vectors in our TPRU provides faster convergence and better performance on the training set. However, the number of role vectors increases, the improvement gained decreases,
14

---

Under review as a conference paper at ICLR 2020
which leads to an interesting hypothesis that the effect of increasing number of role vectors might be correlated to that of increasing the dimensionality of filler vectors.
15


Reviews:
[{"review_id": "SJxro8sodH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new recurrent unit with a simplified dynamics during training leading to more stable training algorithms and better performance. The paper is difficult to read because it assumes the reader is an expert on Tensor Product Representations. Many important terms are not clearly defined, which makes difficult to follow. For example, the terms “roles” and “filler” are not defined. I think a quick introduction to the field with a clarifying figure would be greatly appreciated by general readers. However, I think the contribution of the paper is important and presented experimental results, comparing the method against classical LSTM and GRU architectures, seem to be relevant for the field.\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "HklyTuFRFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel model of recurrent unit for RNNs which is inspired from tensor product representation (TPR) introduced by Smolensky et al. in 1990. The authors claim that this allows one to better incorporate structural information into learning and easier interpretability for the learned representations. The proposed approach is motivated by a theoretical analysis showing that using TPR in this context acts as a sort of pre-conditioner and stabilizes learning. Experiments on entailment tasks (given two statement, decide whether the first implies the second) are provided to validate the approach. \n\nI find the paper not easy to follow, with a non-negligible amount of typos in the notations and results. The advantage in terms of accuracy of the proposed approach seems marginal in the experiment, and the analysis of the interpretability of the learned representations could be improved: loosely speaking, particular examples of interpretability are given but sometimes without contexts or baselines to compare to (see the two last comments below). I know \"interpretability\" is a difficult property to assess but I think there may be more principled ways to showcase the approach.\n\nI think this paper is not yet ready for publication: the proposed model is interesting and relevant but its validity could be better assessed and the paper needs some thorough proof-reading. \n\n\n* Questions / Comments *\n\n- page 1: the authors write U^TR = I, but I believe this is only possible if the TPR dimension d is bigger than the number of roles N. Is this always the case? This should be clarified.\n- related to the previous point:  if U^TR = I then shouldn't U^Tb_{t-1} simply be f_{t-1} in Eq. 1?\n- before Eq.1, f should be from R^d\\times R^d' to R^d, not from R^d \\times R^d\n- In Eq. 1, b_{t-1} and x_t are not of the same dimension, so the cannot be multiplied by the same matrix U (this is why the matrices V_x and V_b are introduced later on).\n- there seems to be a problem with Eq. (7): db_t/d_{b_{t-1}} appears on both sides of the equality...\n- Modification 1: what is \\tilde{vb_t}? I don't remember seeing this notation introduced before.\n- Table 1: constants should not be included in big O notation! To compare constants, one should give the exact number of operations needed for inference.\n- POS tagging: Aren't there many other reasons that could lead to this correlation (beside the informal argument that \"TPR captures structured information\")? Maybe the authors should compare with something else, for example the PMI between values of hidden neurons in a learned RNN and POS tags. Out of context, the numbers in Table 5 are not informative.\n- Polysemy: Only a very specific cherry picked example is given here. A more principled or in depth analysis of this phenomenon is needed to make a stronger case.\n\n* Typos *\n\n- \" The number of parameter matrices *is* the same as that of...\"\n- page 4 \"stables\" -> \"stabilizes\" (but rephrasing the sentence altogether would be better).\n- page 5: BiDAF misses the capital letters (\"bidaf\").\n- \"dev set\" -> \"validation set\" or \"development set\".\n- page 8: \"provides research*ers with* an intuitive...\"?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeJA1orFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, a new unit based on the outer product called TPRU is proposed for recurrent neural networks. The performance of TPRU is validated with several NLP tasks such as POS tagging. \n\nWhile my knowledge about RNNs is limited, I feel the paper has room for improvement and I vote for rejection this time. The main reasons are: 1. the paper is not well written, and 2. the way of analysis is not enough.\n\n1. From the viewpoint of an RNN non-expert (i.e., me), the paper somehow fails to introduce the background. For example, tensor product representation (TPR) is introduced in the second paragraph of Introduction. While TPR is an elemental idea of this study, it is introduced with neither motivation (why and when TPR is useful, etc) nor appropriate references, which makes non-experts difficult to catch up with the main body of this study. So the paper is not self-contained enough. \n\n2-1. The paper tries to explain why TPRU is better in terms of gradient vanishing/explosion in Section 4. However, the analysis is mainly performed in a qualitative way, and there is no quantitative analysis of it. For example, I expect something like the evaluation of the magnitude of the gradient, e.g., how much degree the gradient scale is reduced from normal gate to the TPRU gate. Or, at least there should be the numerical experiments for the comparison. Otherwise, it is hard to judge whether the gradient is actually stabilized.\n\n2-2. The paper says one of the advantages of using TPRU is in its interpretability. However, the term \"interpretability\" is very vague and it is not properly defined in this paper. The paper should discuss what is the metric of interpretability here. More specifically, the paper claims TPRU's interpretability by Table 5. It looks, however, improper because there is no baseline and we cannot conclude that TPRU has better interpretability than others.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has been reviewed by three reviewers and received scores such as 3/3/6. The reviewers took into account the rebuttal in their final verdict. The major criticism concerned the somewhat ad-hoc notion of interpretability, the analysis of vanishing/exploding gradients in  TPRU is experimental lacking theory. Finally,  all reviewers noted the paper is difficult to read and contains grammar issues etc. which does not help. On balance, we regret that this paper cannot be accepted to ICLR2020.


Decision: Reject