Projected Canonical Decomposition for Knowledge Base Completion

1 INTRODUCTION . The problems of representation learning and link prediction in multi-relational data can be formulated as a binary tensor completion problem , where the tensor is obtained by stacking the adjacency matrices of every relations between entities . This tensor can then be intrepreted as a `` knowledge base '' , and contains triples ( subject , predicate , object ) representing facts about the world . Link prediction in knowledge bases aims at automatically discovering missing facts ( Bordes et al. , 2011 ; Nickel et al. , 2011 ; Bordes et al. , 2013 ; Nickel et al. , 2016a ; Nguyen , 2017 ) . State of the art methods use the canonical polyadic ( CP ) decomposition of tensors ( Hitchcock , 1927 ) or variants of it ( Trouillon et al. , 2016 ; Kazemi & Poole , 2018 ; Lacroix et al. , 2018 ) . While initially motivated by low-rank assumptions on the underlying ground-truth tensor , the best performances are obtained by setting the rank as high as permitted by computational constraints , using tensor norms for regularization ( Lacroix et al. , 2018 ) . However , for large scale data where computational or memory constraints require ranks to be low ( Lerer et al. , 2019 ) , performances drop drastically . Tucker decomposition is another multilinear model which allows richer interactions between entities and predicate vectors . A special case of Tucker decomposition is RESCAL ( Nickel et al. , 2011 ) , in which the relations are represented by matrices and entities factors are shared for subjects and objects . However , an evaluation of this model in Nickel et al . ( 2016b ) shows that RESCAL lags behind other methods on several benchmarks of interest . Recent work have obtained more competitive results with similar models ( Balažević et al. , 2019b ; Wang et al. , 2019 ) , using different regularizers or deep learning heuristics such as dropout and label smoothing . Despite these recent efforts , learning Tucker decompositions remains mostly unresolved . Wang et al . ( 2019 ) does not achieve state of the art results on standard benchmarks , and we show ( see Figure 3 ) that the performances reported by Balažević et al . ( 2019b ) are actually matched by ComplEx ( Trouillon et al. , 2016 ; Lacroix et al. , 2018 ) optimized with Adam , which has less hyperparameters . In this work , we overcome some of the difficulties associated with learning a Tucker model for knowledge base completion . Balažević et al . ( 2019b ) use deep-learning mechanisms such as batch normalization ( Ioffe & Szegedy , 2015 ) , dropout ( Srivastava et al. , 2014 ) or learning-rate annealing to address both regularization and optimization issues . Our approach is different : We factorize the core tensor of the Tucker decomposition with CP to obtain a formulation which is closer to CP and better understand what difficulties appear . This yields a simple approach , which has a single regularization hyperparameter to tune for a fixed model specification . The main novelty of our approach is a more careful application of adaptive gradient techniques . State-of-the-art methods for tensor completion use optimization algorithms with adaptive diagonal rescaling such as Adagrad ( Duchi et al. , 2011 ) or Adam ( Kingma & Ba , 2014 ) . Through control experiments in which our model is equivalent to CP up to a fixed rotation of the embeddings , we show that one of the difficulties in training Tucker-style decompositions can be attributed to the lack of invariance to rotation of the diagonal rescaling . Focusing on Adagrad , we propose a different update rule that is equivalent to implicitely applying Adagrad to a CP model with a projection of the embedding to a lower dimensional subspace . Combining the Tucker formulation and the implicit Adagrad update , we obtain performances that match state-of-the-art methods on the standard benchmarks and achieve significanly better results for small embedding sizes on several datasets . Compared to the best current algorithm for Tucker decomposition of Balažević et al . ( 2019b ) , our approach has less hyperparameters , and we effectively report better performances than the implementation of ComplEx of Lacroix et al . ( 2018 ) in the regime of small embedding dimension . We discuss the related work in the next section . In Section 3 , we present a variant of the Tucker decomposition which allows to interpolate between Tucker and CP . The extreme case of this variant , which is equivalent to CP up to a fixed rotation of the embedding , serves as control model to highlight the deficiency of the diagonal rescaling of Adagrad for Tucker-style decompositions in experiments reported in Section 4 . We present the modified version of Adagrad in Section 5 and present experimental results on standard benchmarks of knowledge base completion in Section 7 . 2 LINK PREDICTION IN KNOWLEDGE BASES . Notation Tensors and matrices are denoted by uppercase letters . For a matrix U , ui is the vector corresponding to the i-th row of U . The tensor product is written ⊗ and the Hadamard product ( i.e. , elementwise product ) is written ⊙ . 2.1 LEARNING SETUP . A knowledge base consists of a set S of triples ( subject , predicate , object ) that represent ( true ) known facts . The goal of link prediction is to recover facts that are true but not in the database . The data is represented as a tensor X̃ ∈ { 0 , 1 } N×L×N for N the number of entities and L the number of predicates . Given a training set of triples , the goal is to provide a ranking of entities for queries of the type ( subject , predicate , ? ) and ( ? , predicate , object ) . Following Lacroix et al . ( 2018 ) , we use the cross-entropy as a surrogate of the ranking loss . As proposed by Lacroix et al . ( 2018 ) and Kazemi & Poole ( 2018 ) , we include reciprocal predicates : for each predicate P in the original dataset , and given an item o , each query of the form ( ? , P , o ) is reformulated as a query ( o , P−1 , ? ) , where o is now the subject of P−1 . This doubles the effective number of predicates but reduces the problem to queries of the type ( subject , predicate , ? ) only . For a given triple ( i , j , k ) ∈ S , the training loss function for a tensor X is then ℓi , j , k ( X ) = −Xi , j , k + log ( ∑ k′ ̸=k exp ( Xi , j , k′ ) ) . ( 1 ) For a tensor decomposition model X ( θ ) parameterized by θ , the parameters θ̂ are found by minimizing the regularized empirical risk with regularizer Λ : θ̂ = argmin θ L ( θ ) = argmin θ 1 |S| ∑ ( i , j , k ) ∈S ℓi , j , k ( X ( θ ) ) + νΛ ( θ ) . ( 2 ) This work studies specific models for X ( θ ) , inspired by CP and Tucker decomposition . We discuss the related work on tensor decompositions and link prediction in knowledge bases below . 2.2 RELATED WORK . 2.2.1 CANONICAL DECOMPOSITION AND ITS VARIANTS . The canonical polyadic ( CP ) decomposition of a tensor X is defined entrywise by ∀i , j , k , Xi , j , k = ⟨ui , vj , wk⟩ : = d∑ r=1 uirvjrwkr . The smallest value of d for which this decomposition exists is the rank of X . Each element Xi , j , k is thus represented as a multi-linear product of the 3 embeddings in Rd associated respectively to the ith subject , the jth predictate and the kth object . CP currently achieves near state-of-the-art performances on standard benchmarks of knowledge base completion ( Kazemi & Poole , 2018 ; Lacroix et al. , 2018 ) . Nonetheless , the best reported results are with the ComplEx model ( Trouillon et al. , 2016 ) , which learns complex-valued embeddings and sets the embeddings of the objects to be the complex conjugate of the embeddings of subjects , i.e. , wk = ūk . Prior to ComplEx , Dismult was proposed ( Yang et al. , 2014 ) as a variant of CP with wk = uk . While this model obtained good performances ( Kadlec et al. , 2017 ) , it can only model symmetric relations and does not perform as well as ComplEx . CP-based models are optimized with vanilla Adam or Adagrad and a single regularization parameter ( Trouillon et al. , 2016 ; Kadlec et al. , 2017 ; Lacroix et al. , 2018 ) and do not require additional heuristics for training . 2.2.2 TUCKER DECOMPOSITION AND ITS VARIANTS . Given a tensor X of size N×L×N , the Tucker decomposition of X is defined entrywise by ∀i , j , k , Xi , j , k = ⟨ui ⊗ vj ⊗ wk , C⟩ : = d1∑ r1=1 d2∑ r2=1 d3∑ r3=1 Cr1 , r2 , r3uir1vjr2wkr3 . The triple ( d1 , d2 , d3 ) are the rank parameters of the decomposition . We also use a multilinear product notation X = [ [ C ; U , V , W ] ] , where U , V , W are the matrices whose rows are respectively uj , vk , wl and C the three dimensional d1 × d2 × d3 core tensor . Note that the CP decomposition is a Tucker decomposition in which d1 = d2 = d3 = d and C is the identity , which we write [ [ U , V , W ] ] . With a non-trivial core tensor , Tucker decomposition is thus more flexible than CP for fixed embedding size . In knowledge base applications , we typically have d ≤ L ≪ N , so the vast majority of the model parameters are in the embedding matrices of the entities U and W . When constraints on the number of model parameters arise ( e.g. , memory constraints ) , Tucker models appear as natural candidates to increase the expressivity of the decomposition compared to CP with limited impact on the total number of parameters . While many variants of the Tucker decomposition have been proposed in the literature on tensor factorization ( see e.g. , Kolda & Bader , 2009 ) , the first approach based on Tucker for link prediction in knowledge bases is RESCAL ( Nickel et al. , 2011 ) . RESCAL uses a special form of Tucker decomposition in which the object and subject embeddings are shared , i.e. , U = W , and it does not compress the relation matrices . In the multilinear product notation above , a RESCAL model is thus written as X = [ [ C ; U , I , U ] ] . Despite some success on a few smaller datasets , RESCAL performances drop on larger datasets ( Nickel et al. , 2016b ) . This decrease in performances has been attributed either to improper regularization ( Nickel et al. , 2011 ) or optimization issues ( Xue et al. , 2018 ) . Balažević et al . ( 2019b ) revisits Tucker decomposition in the context of large-scale knowledge bases and resolves some of the optimization and regularization issues using learning rate annealing , batch-normalization and dropout . It comes at the price of more hyperparameters to tune for each dataset ( label smoothing , three different dropouts and a learning rate decay ) , and as we discuss in our experiments , the results they report are not better than ComplEx for the same number of parameters . Two methods were previously proposed to interpolate between the expressivity of RESCAL and CP . Xue et al . ( 2018 ) expands the HolE model ( Nickel et al. , 2016b ) ( and thus the ComplEx model ( Hayashi & Shimbo , 2017 ) ) based on cross-correlation of embeddings to close the gap in expressivity with the Tucker decomposition for a fixed embedding size . Jenatton et al . ( 2012 ) express the relation matrices in RESCAL as low-rank combination of a family of matrices . We describe the link between these approaches and ours in Appendix 9.4 . None of these approach however studied the effect of their formulation on optimization , and reported results inferior to ours .

Summary: In this paper, a tensor decomposition method is studied for link prediction problems. The model is based on Tucker decomposition but the core tensor is decomposed as CP decomposition so that it can be seen as an interpolation between Tucker and CP. The performance is evaluated with several NLP data sets (e.g., subject-verb-object triplets).