A Simple Recurrent Unit with Reduced Tensor Product Representations

1 INTRODUCTION . Recurrent units are widely used in sequence modelling tasks , including text , speech , DNA/RNA sequencing data , etc . Due to their ability to produce distributed vector representations for sequences with various lengths , and their property of being a universal function approximators ( Schäfer & Zimmermann , 2006 ) , they quickly succeeded in numerous research areas . The training algorithm , backpropagation through time ( BPTT , Werbos et al. , 1990 ) , has remained the same across the years , with known gradient vanishing and exploding issues . Many theoretically justifiable or empirically useful regularisations ( Pascanu et al. , 2013 ; Kanai et al. , 2017 ) have been proposed to ease the learning process and to smooth the gradient flow for long sequences , and informative analysis methods have been conducted to visualise the information encoded in the learnt vector representations ( Zeiler & Fergus , 2013 ; Karpathy et al. , 2015 ) . The main difficulty in understanding the learning process of a particular RNN unit comes from the extreme nonlinear activation functions , which cause the gradient flow to be chaotic . Here , we aim to propose a recurrent unit that has fewer nonlinear operations by incorporating Tensor Product Representations ( Smolensky , 1990 ) . Tensor product representation ( TPR ) is an instantiation of general neural-symbolic computing in which symbol structures are embedded in a vector space via a filler-role decomposition : the structure is captured by a set of N roles ri ( e.g. , left-child-of-root ) , each of which is bound to a filler fi ( a symbol or substructure ) . A TPR embedding of a symbol structure S derives from vector embeddings of the roles { ri } and their fillers { fi } via the outer or tensor product : S = ∑N i=1 ri ⊗ fi =∑N i=1 rif > i = RF > , where R and F respectively denote matrices having the role or filler vectors of S as columns . Each ri⊗ fi is the embedding of a role-filler binding : a constituent ; ⊗ is the binding operation . The unbinding operation returns the filler of a particular role in S ; it is performed by the inner product : fi = u > i S , where ui is the dual of ri , satisfying u > i rj = δij ≡ 1 if i = j else 0 . Letting U be the matrix with columns { ui } , we have U > R = I . Our goal is to propose a recurrent unit that uses a reduced version of a tensor product representation , a vector rather than an order-2 tensor : this arises by reducing the dimension of the filler embeddings to 1 ; with role embeddings of dimension d , the representation becomes b = ∑N i=1 firi ∈ R d. We let this sum range over all N possible roles , letting fi = 0 when role ri is unfilled . Then at time t , the hidden state of our RNN — which we call the binding complex — will be bt = ∑N i=1 ( ft ) iri where ft is the vector of filler values ( ft ) i ∈ R at time t. Now binding is performed by a simple matrix-vector product , and unbinding by a simple inner product : bt = Rft ft = U > bt ( 1 ) In this paper : • In the context of related work ( Sec . 2 ) and based on the reduced tensor product representations introduced above , we propose a gated recurrent unit , the TPRU ( Sec . 3 ) , along with a set of analyses of the gradient flow to justify our decisions in architecture design ( Sec . 4 ) . Even though the number of nonlinear operations is reduced , the success in experiments attests to the flexibility of our TPRU in learning and our analysis on derivatives further supports it . • Our proposed TPRU demonstrates strong performance on the Logical Entailment task ( Evans et al. , 2018 ) , Natural Language Inference datasets ( Williams et al. , 2017 ) , and language modelling , while having comparably many parameters as the GRU , and significantly fewer than the widely-used LSTM ( Sec . 5 ) . • The instantiation of reduced TPRs allows our model to learn decomposable representations , and the filler ft provides an interface for us to inspect the information encoded in bt ( Sec . 6 ) . We conducted a linguistically grounded study , which shows that our TPRU is capable of distinguishing multiple senses of a polysemous word and discovering structured information in sentences . 2 RELATED WORK . Recent efforts on understanding RNNs includes these three classes : ( i ) theoretical analysis through advancements in dynamic systems on the functions learnt by RNN units ( Chang et al. , 2019 ) ; ( ii ) linguistically grounded analysis by developing sets of probing tasks to illustrate the properties of learnt vector representations of RNNs ( Conneau et al. , 2018 ; Chrupała & Alishahi , 2019 ) ; and ( iii ) introducing inductive biases into the architecture of networks and comparing the behaviour of learnt models against those of widely-used counterparts . Our work falls in this last category : learning structured representations by incorporating the inductive biases inherent in TPRs . Some prior work has incorporated TPRs into RNN representations . Question-answering on SQuAD ( Rajpurkar et al. , 2016 ) was addressed ( Palangi et al. , 2017 ) with an LSTM with hidden state given by a single TPR binding ; the present work deploys complexes containing multiple bindings ( albeit reduced ones ) . TPR-style unbinding was applied in caption generation ( Huang et al. , 2018 ) , but the representations were deep-learnt and not explicitly designed to be TPRs as in the present work . A contracted version of TPRs , Holographic Reduced Representations , was utilised to decompose input space and output space with filler-role decomposition ( Luo et al. , 2019 ) . Our work differs from these prior papers in that TPR filler-role binding and unbinding operations are explicitly carried out in our proposed recurrent unit , and the two operations directly determine the update of the hidden states . Closest to the present work is Schlag & Schmidhuber ( 2018 ) , which performed questionanswering by using TPRs to embed simple knowledge graphs as tensors ; our model does not deploy graph-based inference , which is less natural for the inference tasks we address than it is for the question-answering task addressed in Schlag & Schmidhuber ( 2018 ) . Our proposed TPRU inherits the explicit binding and unbinding operations in TPRs in the sense that a set of role vectors are shared across time steps and sequences to serve as a basis for composing distributed vector representations . We provide a set of analyses from both theoretical and empirical perspectives : gradient analysis of our model helps us make sense of the architecture design , while empirical results from three different categories of tasks show the effectiveness of the proposed model . Additional study on the linguistic and learning aspects provide evidence that our model indeed inherits the merits of TPRs , which enhance interpretability and the capability of learning structured representations . 3 PROPOSED RECURRENT UNIT : THE TPRU . As shown in both LSTM ( Hochreiter & Schmidhuber , 1997 ) and GRU ( Chung et al. , 2014 ) design , a gating mechanism helps the hidden state at the current time step to directly copy information from the previous time step , and alleviate vanishing and exploding gradient issues . Our proposed recurrent unit deploys a single gate , adopting the input gate of the GRU , omitting the reset gate . We expect that further augmenting our TPRU to 2 gates as in the GRU , or 3 gates as in the LSTM , will lead to yet better performance . At each time step , the TPRU receives two input vectors : bt−1 ∈ Rd , the binding complex from the TPRU at the previous time-step , and xt ∈ Rd ′ from the external input ; it produces bt ∈ Rd . An input gate gt ∈ Rd is computed to produce a weighted sum of the information generated at the current time step , b̃t ∈ Rd , and the previous binding complex bt−1 , bt = gt ◦ b̃t + ( 1− gt ) ◦ bt−1 , where gt = σ ( Wbbt−1 +Wxxt ) , ( 2 ) σ ( · ) is the logistic sigmoid function ( applied element-wise ) , ◦ is the Hadamard ( element-wise ) product , and Wb ∈ Rd×d and Wx ∈ Rd×d ′ are matrices of learnable parameters . As we now explain , the calculation of b̃t is carried out by the unbinding and binding operations of TPRs ( Eq . 1 ) . 3.1 UNBINDING OPERATION . Consider a set of hypothesised unbinding vectors U = [ u1 , u2 , ... , uN ] ∈ Rd×N . At time step t , these can be used to unbind fillers fb , t from the previous binding complex bt−1 using Eq . 1 . We posit two matrices Vx ∈ Rd×d ′ and Vb ∈ Rd×d that transform the current input xt ∈ Rd ′ and the binding complex bt−1 into the binding space Rd yielding fillers fx , t and fb , t : fb , t = U > Vbbt−1 ∈ RN , fx , t = U > Vxxt ∈ RN . ( 3 ) Normalisation is applied for stable learning by gradient descent , and the specific technique ( Yang et al. , 2018 ) we adopted here also enforces sparsity on both fb , t and fx , t : ( ft ) n = ( f̃t ) 2 n/ ∑N m=1 ( f̃t ) 2 m ( 4 ) where ( f̃t ) n = max ( 0 , ( fb , t ) n + bb ) + max ( 0 , ( fx , t ) n + bx ) ( 5 ) in which bb and bx are two scalar parameters for stable learning , and ( · ) n refers to the n-th entry of the vector in the parentheses . 3.2 BINDING OPERATION . Given a hypothesised set of binding role vectors R = [ r1 , r2 , ... , rN ] ∈ Rd×N , applying the binding operation in Eq . 1 to the fillers ft at time t gives the candidate update b̃t for the binding complex , b̃t = Rft . ( 6 ) The gating mechanism controls the weighted sum of the candidate vector b̃t and the previous binding complex bt−1 to produce a binding complex bt at the current time step , as given by Eq . 2 . 3.3 UNBINDING AND BINDING ROLE VECTORS . In the TPRU , there is a matrix of role vectors R used for the binding operation and a matrix of unbinding vectors U used for the unbinding operation . To control the number of parameters in our TPRU , rendering it independent of N , instead of directly learning two sets of vectors , a set of N ddimensional random vectors E is sampled from a normal distribution , and two linear projections Wr , Wu ∈ Rd×d are learnt to transform E to U = WuE and R = WrE . Therefore , in total , our proposed TPRU has six learnable matrices : Wu , Wr , Vb , Wb ∈ Rd×d and Vx , Wx ∈ Rd×d ′ . The number of parameter matrices the same as that of a GRU and less than that of an LSTM , and , when d′ < d , the number of learnable parameters is less than that of a GRU .

Summary: This paper proposes a novel model of recurrent unit for RNNs which is inspired from tensor product representation (TPR) introduced by Smolensky et al. in 1990. The authors claim that this allows one to better incorporate structural information into learning and easier interpretability for the learned representations. The proposed approach is motivated by a theoretical analysis showing that using TPR in this context acts as a sort of pre-conditioner and stabilizes learning. Experiments on entailment tasks (given two statement, decide whether the first implies the second) are provided to validate the approach.