Natural- to formal-language generation using Tensor Product Representations

Generating formal-language represented by relational tuples , such as Lisp programs or mathematical operations , from natural-language input is a challenging task because it requires explicitly capturing discrete symbolic structural information implicit in the input . Most state-of-the-art neural sequence models do not explicitly capture such structural information , limiting their performance on these tasks . In this paper we propose a new encoder-decoder model based on Tensor Product Representations ( TPRs ) for Natural- to Formal-language generation , called TP-N2F . The encoder of TP-N2F employs TPR ‘ binding ’ to encode natural-language symbolic structure in vector space and the decoder uses TPR ‘ unbinding ’ to generate , in symbolic space , a sequence of relational tuples , each consisting of a relation ( or operation ) and a number of arguments . On two benchmarks , TP-N2F considerably outperforms LSTM-based seq2seq models , creating new state-of-the-art results : the MathQA dataset for math problem solving , and the AlgoLisp dataset for program synthesis . Ablation studies show that improvements can be attributed to the use of TPRs in both the encoder and decoder to explicitly capture relational structure to support reasoning . 1 INTRODUCTION . When people perform explicit reasoning , they can typically describe the way to the conclusion step by step via relational descriptions . There is ample evidence that relational representations are important for human cognition ( e.g. , ( Goldin-Meadow & Gentner , 2003 ; Forbus et al. , 2017 ; Crouse et al. , 2018 ; Chen & Forbus , 2018 ; Chen et al. , 2019 ) ) . Although a rapidly growing number of researchers use deep learning to solve complex symbolic reasoning and language tasks ( a recent review is ( Gao et al. , 2019 ) ) , most existing deep learning models , including sequence models such as LSTMs , do not explicitly capture human-like relational structure information . In this paper we propose a novel neural architecture , TP-N2F , to solve natural- to formal-language generation tasks ( N2F ) . In the tasks we study , math or programming problems are stated in naturallanguage , and answers are given as programs , sequences of relational representations , to solve the problem . TP-N2F encodes the natural-language symbolic structure of the problem in an input vector space , maps this to a vector in an intermediate space , and uses that vector to produce a sequence of output vectors that are decoded as relational structures . Both input and output structures are modelled as Tensor Product Representations ( TPRs ) ( Smolensky , 1990 ) . During encoding , NL-input symbolic structures are encoded as vector space embeddings using TPR ‘ binding ’ ( following Palangi et al . ( 2018 ) ) ; during decoding , symbolic constituents are extracted from structure-embedding output vectors using TPR ‘ unbinding ’ ( following Huang et al . ( 2018 ; 2019 ) ) . Our contributions in this work are as follows . ( i ) We propose a role-level analysis of N2F tasks . ( ii ) We present a new TP-N2F model which gives a neural-network-level implementation of a model solving the N2F task under the role-level description proposed in ( i ) . To our knowledge , this is the first model to be proposed which combines both the binding and unbinding operations of TPRs to achieve generation tasks through deep learning . ( iii ) State-of-the-art performance on two recently developed N2F tasks shows that the TP-N2F model has significant structure learning ability on tasks requiring symbolic reasoning through program synthesis . 2 BACKGROUND : REVIEW OF TENSOR-PRODUCT REPRESENTATION . The TPR mechanism is a method to create a vector space embedding of complex symbolic structures . The type of a symbol structure is defined by a set of structural positions or roles , such as the leftchild-of-root position in a tree , or the second-argument-of-R position of a given relation R. In a particular instance of a structural type , each of these roles may be occupied by a particular filler , which can be an atomic symbol or a substructure ( e.g. , the entire left sub-tree of a binary tree can serve as the filler of the role left-child-of-root ) . For now , we assume the fillers to be atomic symbols.1 The TPR embedding of a symbol structure is the sum of the embeddings of all its constituents , each constituent comprising a role together with its filler . The embedding of a constituent is constructed from the embedding of a role and the embedding of the filler of that role : these are joined together by the TPR ‘ binding ’ operation , the tensor ( or generalized outer ) product ⊗ . Formally , suppose a symbolic type is defined by the roles { ri } , and suppose that in a particular instance of that type , S , role ri is bound by filler fi . The TPR embedding of S is the order-2 tensor T = ∑ i fi ⊗ ri = ∑ i fir > i ( 1 ) where { fi } are vector embeddings of the fillers and { ri } are vector embeddings of the roles . In Eq . 1 , and below , for notational simplicity we conflate order-2 tensors and matrices . As a simple example , consider the symbolic type string , and choose roles to be r1 = first element , r2 = second element , etc . Then in the specific string S = cba , the first role r1 is filled by c , and r2 and r3 by b and a , respectively . The TPR for S is c ⊗ r1 + b ⊗ r2 + a ⊗ r3 , where a , b , c are the vector embeddings of the symbols a , b , c , and ri is the vector embedding of role ri . A TPR scheme for embedding a set of symbol structures is defined by a decomposition of those structures into roles bound to fillers , an embedding of each role as a role vector , and an embedding of each filler as a filler vector . Let the total number of roles and fillers available be nR , nF , respectively . Define the matrix of all possible role vectors to be R ∈ RdR×nR , with column i , [ R ] : i = ri ∈ RdR , comprising the embedding of ri . Similarly let F ∈ RdF×nF be the matrix of all possible filler vectors . The TPR T ∈ RdF×dR . Below , dR , nR , dF , nF will be hyper-parameters , while R , F will be learned parameter matrices . Using summation in Eq.1 to combine the vectors embedding the constituents of a structure risks non-recoverability of those constituents given the embedding T of the the structure as a whole . The tensor product is chosen as the binding operation in order to enable recovery of the filler of any role in a structure S given its TPR T. This can be done with perfect precision if the embeddings of the roles are linearly independent . In that case the role matrix R has a left inverse U : UR = I . Now define the unbinding ( or dual ) vector for role rj , uj , to be the jth column of U > : U > : j . Then , since [ I ] ji = [ UR ] ji = Uj : R : i = [ U > : j ] > R : i = u > j ri = r > i uj , we have r > i uj = δji . This means that , to recover the filler of rj in the structure with TPR T , we can take its tensor inner product ( or matrix-vector product ) with uj :2 Tuj = [ ∑ i fir > i ] uj = ∑ i fiδij = fj ( 2 ) In the architecture proposed here , we will make use of both TPR binding using the tensor product with role vectors ri and TPR unbinding using the tensor inner product with unbinding vectors uj . Binding will be used to produce the order-2 tensor TS embedding of the NL problem statement . Unbinding will be used to generate output relational tuples from an order-3 tensor H. Because they pertain to different representations ( of different orders in fact ) , the binding and unbinding vectors we will use are not related to one another . 1When fillers are structures themselves , binding can be used recursively , giving tensors of order higher than 2 . In general , binding is done with the tensor product , since conflation with matrix algebra is only possible for order-2 tensors . Our unbinding of relational tuples involves the order-3 TPRs defined in Sec . 3.1.2 . 2When the role vectors are not linearly independent , this operation performs unbinding approximately , taking U to be the left pseudo-inverse of R. Because randomly chosen vectors on the unit sphere in a highdimensional space are approximately orthogonal , the approximation is often excellent ( Anonymous , in prep . ) . 3 TP-N2F MODEL . We propose a general TP-N2F neural network architecture operating over TPRs to solve N2F tasks under a proposed role-level description of those tasks . In this description , natural-language input is represented as a straightforward order-2 role structure , and formal-language relational representations of outputs are represented with a new order-3 recursive role structure proposed here . Figure 1 shows an overview diagram of the TP-N2F model . It depicts the following high-level description . As shown in Figure 1 , while the natural-language input is a sequence of words , the output is a sequence of multi-argument relational tuples such as ( R A1 A2 ) , a 3-tuple consisting of a binary relation ( or operation ) R with its two arguments . The “ TP-N2F encoder ” uses two LSTMs to produce a pair consisting of a filler vector and a role vector , which are bound together with the tensor product . These tensor products , concatenated , comprise the “ context ” over which attention will operate in the decoder . The sum of the word-level TPRs , flattened to a vector , is treated as a representation of the entire problem statement ; it is fed to the “ Reasoning MLP ” , which transforms this encoding of the problem into a vector encoding the solution . This is the initial state of the “ TP-N2F decoder ” attentional LSTM , which outputs at each time step an order-3 tensor representing a relational tuple . To generate a correct tuple from decoder operations , the model must learn to give the order-3 tensor the form of a TPR for a ( R A1 A2 ) tuple ( detailed explanation in Sec . 3.1.2 ) . In the following sections , we first introduce the details of our proposed role-level description for N2F tasks , and then present how our proposed TP-N2F model uses TPR binding and unbinding operations to create a neural network implementation of this description of N2F tasks . 3.1 ROLE-LEVEL DESCRIPTION OF N2F TASKS . In this section , we propose a role-level description of N2F tasks , which specifies the filler/role structures of the input natural-language symbolic expressions and the output relational representations . 3.1.1 ROLE-LEVEL DESCRIPTION FOR NATURAL-LANGUAGE INPUT . Instead of encoding each token of a sentence with a non-compositional embedding vector looked up in a learned dictionary , we use a learned role-filler decomposition to compose a tensor representation for each token . Given a sentence S with n word tokens { w0 , w1 , ... , wn−1 } , each word token wt is assigned a learned role vector rt , soft-selected from the learned dictionary R , and a learned filler vector f t , soft-selected from the learned dictionary F ( Sec . 2 ) . The mechanism closely follows that of Palangi et al . ( 2018 ) , and we hypothesize similar results : the role and filler approximately encode the grammatical role of the token and its lexical semantics , respectively.3 Then each word token wt is represented by the tensor product of the role vector and the filler vector : Tt = f t⊗rt . In addition 3Although the TPR formalism treats fillers and roles symmetrically , in use , hyperparameters are selected so that the number of available fillers is greater than that of roles . Thus , on average , each role is assigned to more words , encouraging it to take on a more general function , such as a grammatical role . to the set of all its token embeddings { T0 , . . . , Tn−1 } , the sentence S as a whole is assigned a TPR equal to the sum of the TPR embeddings of all its word tokens : TS = ∑n−1 t=0 T t. Using TPRs to encode natural language has several advantages . First , natural language TPRs can be interpreted by exploring the distribution of tokens grouped by the role and filler vectors they are assigned by a trained model ( as in Palangi et al . ( 2018 ) ) . Second , TPRs avoid the Bag of Word ( BoW ) confusion ( Huang et al. , 2018 ) : the BoW encoding of Jay saw Kay is the same as the BoW encoding of Kay saw Jay but the encodings are different with TPR embedding , because the role filled by a symbol changes with its context .

Summary: This paper proposes a sequence-to-sequence model for mapping word sequences to relation-argument-tuple sequences. The intermediate representation (output of the encoder) is a fixed-dimensional vector. Both encoder and decoder internally use a tensor product representation. The experimental results suggest that the tensor product representation is helpful for both the encoder and the decoder. The paper is interesting and the experimental results are positive, but in my opinion the exposition could use some substantial work. Fixing the most substantial flaws in the exposition would be sufficient to warrant an accept in my view.