Projected Canonical Decomposition for Knowledge Base Completion

1 INTRODUCTION . The problems of representation learning and link prediction in multi-relational data can be formulated as a binary tensor completion problem , where the tensor is obtained by stacking the adjacency matrices of every relations between entities . This tensor can then be intrepreted as a `` knowledge base '' , and contains triples ( subject , predicate , object ) representing facts about the world . Link prediction in knowledge bases aims at automatically discovering missing facts ( Bordes et al. , 2011 ; Nickel et al. , 2011 ; Bordes et al. , 2013 ; Nickel et al. , 2016a ; Nguyen , 2017 ) . State of the art methods use the canonical polyadic ( CP ) decomposition of tensors ( Hitchcock , 1927 ) or variants of it ( Trouillon et al. , 2016 ; Kazemi & Poole , 2018 ; Lacroix et al. , 2018 ) . While initially motivated by low-rank assumptions on the underlying ground-truth tensor , the best performances are obtained by setting the rank as high as permitted by computational constraints , using tensor norms for regularization ( Lacroix et al. , 2018 ) . However , for large scale data where computational or memory constraints require ranks to be low ( Lerer et al. , 2019 ) , performances drop drastically . Tucker decomposition is another multilinear model which allows richer interactions between entities and predicate vectors . A special case of Tucker decomposition is RESCAL ( Nickel et al. , 2011 ) , in which the relations are represented by matrices and entities factors are shared for subjects and objects . However , an evaluation of this model in Nickel et al . ( 2016b ) shows that RESCAL lags behind other methods on several benchmarks of interest . Recent work have obtained more competitive results with similar models ( Balažević et al. , 2019b ; Wang et al. , 2019 ) , using different regularizers or deep learning heuristics such as dropout and label smoothing . Despite these recent efforts , learning Tucker decompositions remains mostly unresolved . Wang et al . ( 2019 ) does not achieve state of the art results on standard benchmarks , and we show ( see Figure 3 ) that the performances reported by Balažević et al . ( 2019b ) are actually matched by ComplEx ( Trouillon et al. , 2016 ; Lacroix et al. , 2018 ) optimized with Adam , which has less hyperparameters . In this work , we overcome some of the difficulties associated with learning a Tucker model for knowledge base completion . Balažević et al . ( 2019b ) use deep-learning mechanisms such as batch normalization ( Ioffe & Szegedy , 2015 ) , dropout ( Srivastava et al. , 2014 ) or learning-rate annealing to address both regularization and optimization issues . Our approach is different : We factorize the core tensor of the Tucker decomposition with CP to obtain a formulation which is closer to CP and better understand what difficulties appear . This yields a simple approach , which has a single regularization hyperparameter to tune for a fixed model specification . The main novelty of our approach is a more careful application of adaptive gradient techniques . State-of-the-art methods for tensor completion use optimization algorithms with adaptive diagonal rescaling such as Adagrad ( Duchi et al. , 2011 ) or Adam ( Kingma & Ba , 2014 ) . Through control experiments in which our model is equivalent to CP up to a fixed rotation of the embeddings , we show that one of the difficulties in training Tucker-style decompositions can be attributed to the lack of invariance to rotation of the diagonal rescaling . Focusing on Adagrad , we propose a different update rule that is equivalent to implicitely applying Adagrad to a CP model with a projection of the embedding to a lower dimensional subspace . Combining the Tucker formulation and the implicit Adagrad update , we obtain performances that match state-of-the-art methods on the standard benchmarks and achieve significanly better results for small embedding sizes on several datasets . Compared to the best current algorithm for Tucker decomposition of Balažević et al . ( 2019b ) , our approach has less hyperparameters , and we effectively report better performances than the implementation of ComplEx of Lacroix et al . ( 2018 ) in the regime of small embedding dimension . We discuss the related work in the next section . In Section 3 , we present a variant of the Tucker decomposition which allows to interpolate between Tucker and CP . The extreme case of this variant , which is equivalent to CP up to a fixed rotation of the embedding , serves as control model to highlight the deficiency of the diagonal rescaling of Adagrad for Tucker-style decompositions in experiments reported in Section 4 . We present the modified version of Adagrad in Section 5 and present experimental results on standard benchmarks of knowledge base completion in Section 7 . 2 LINK PREDICTION IN KNOWLEDGE BASES . Notation Tensors and matrices are denoted by uppercase letters . For a matrix U , ui is the vector corresponding to the i-th row of U . The tensor product is written ⊗ and the Hadamard product ( i.e. , elementwise product ) is written ⊙ . 2.1 LEARNING SETUP . A knowledge base consists of a set S of triples ( subject , predicate , object ) that represent ( true ) known facts . The goal of link prediction is to recover facts that are true but not in the database . The data is represented as a tensor X̃ ∈ { 0 , 1 } N×L×N for N the number of entities and L the number of predicates . Given a training set of triples , the goal is to provide a ranking of entities for queries of the type ( subject , predicate , ? ) and ( ? , predicate , object ) . Following Lacroix et al . ( 2018 ) , we use the cross-entropy as a surrogate of the ranking loss . As proposed by Lacroix et al . ( 2018 ) and Kazemi & Poole ( 2018 ) , we include reciprocal predicates : for each predicate P in the original dataset , and given an item o , each query of the form ( ? , P , o ) is reformulated as a query ( o , P−1 , ? ) , where o is now the subject of P−1 . This doubles the effective number of predicates but reduces the problem to queries of the type ( subject , predicate , ? ) only . For a given triple ( i , j , k ) ∈ S , the training loss function for a tensor X is then ℓi , j , k ( X ) = −Xi , j , k + log ( ∑ k′ ̸=k exp ( Xi , j , k′ ) ) . ( 1 ) For a tensor decomposition model X ( θ ) parameterized by θ , the parameters θ̂ are found by minimizing the regularized empirical risk with regularizer Λ : θ̂ = argmin θ L ( θ ) = argmin θ 1 |S| ∑ ( i , j , k ) ∈S ℓi , j , k ( X ( θ ) ) + νΛ ( θ ) . ( 2 ) This work studies specific models for X ( θ ) , inspired by CP and Tucker decomposition . We discuss the related work on tensor decompositions and link prediction in knowledge bases below . 2.2 RELATED WORK . 2.2.1 CANONICAL DECOMPOSITION AND ITS VARIANTS . The canonical polyadic ( CP ) decomposition of a tensor X is defined entrywise by ∀i , j , k , Xi , j , k = ⟨ui , vj , wk⟩ : = d∑ r=1 uirvjrwkr . The smallest value of d for which this decomposition exists is the rank of X . Each element Xi , j , k is thus represented as a multi-linear product of the 3 embeddings in Rd associated respectively to the ith subject , the jth predictate and the kth object . CP currently achieves near state-of-the-art performances on standard benchmarks of knowledge base completion ( Kazemi & Poole , 2018 ; Lacroix et al. , 2018 ) . Nonetheless , the best reported results are with the ComplEx model ( Trouillon et al. , 2016 ) , which learns complex-valued embeddings and sets the embeddings of the objects to be the complex conjugate of the embeddings of subjects , i.e. , wk = ūk . Prior to ComplEx , Dismult was proposed ( Yang et al. , 2014 ) as a variant of CP with wk = uk . While this model obtained good performances ( Kadlec et al. , 2017 ) , it can only model symmetric relations and does not perform as well as ComplEx . CP-based models are optimized with vanilla Adam or Adagrad and a single regularization parameter ( Trouillon et al. , 2016 ; Kadlec et al. , 2017 ; Lacroix et al. , 2018 ) and do not require additional heuristics for training . 2.2.2 TUCKER DECOMPOSITION AND ITS VARIANTS . Given a tensor X of size N×L×N , the Tucker decomposition of X is defined entrywise by ∀i , j , k , Xi , j , k = ⟨ui ⊗ vj ⊗ wk , C⟩ : = d1∑ r1=1 d2∑ r2=1 d3∑ r3=1 Cr1 , r2 , r3uir1vjr2wkr3 . The triple ( d1 , d2 , d3 ) are the rank parameters of the decomposition . We also use a multilinear product notation X = [ [ C ; U , V , W ] ] , where U , V , W are the matrices whose rows are respectively uj , vk , wl and C the three dimensional d1 × d2 × d3 core tensor . Note that the CP decomposition is a Tucker decomposition in which d1 = d2 = d3 = d and C is the identity , which we write [ [ U , V , W ] ] . With a non-trivial core tensor , Tucker decomposition is thus more flexible than CP for fixed embedding size . In knowledge base applications , we typically have d ≤ L ≪ N , so the vast majority of the model parameters are in the embedding matrices of the entities U and W . When constraints on the number of model parameters arise ( e.g. , memory constraints ) , Tucker models appear as natural candidates to increase the expressivity of the decomposition compared to CP with limited impact on the total number of parameters . While many variants of the Tucker decomposition have been proposed in the literature on tensor factorization ( see e.g. , Kolda & Bader , 2009 ) , the first approach based on Tucker for link prediction in knowledge bases is RESCAL ( Nickel et al. , 2011 ) . RESCAL uses a special form of Tucker decomposition in which the object and subject embeddings are shared , i.e. , U = W , and it does not compress the relation matrices . In the multilinear product notation above , a RESCAL model is thus written as X = [ [ C ; U , I , U ] ] . Despite some success on a few smaller datasets , RESCAL performances drop on larger datasets ( Nickel et al. , 2016b ) . This decrease in performances has been attributed either to improper regularization ( Nickel et al. , 2011 ) or optimization issues ( Xue et al. , 2018 ) . Balažević et al . ( 2019b ) revisits Tucker decomposition in the context of large-scale knowledge bases and resolves some of the optimization and regularization issues using learning rate annealing , batch-normalization and dropout . It comes at the price of more hyperparameters to tune for each dataset ( label smoothing , three different dropouts and a learning rate decay ) , and as we discuss in our experiments , the results they report are not better than ComplEx for the same number of parameters . Two methods were previously proposed to interpolate between the expressivity of RESCAL and CP . Xue et al . ( 2018 ) expands the HolE model ( Nickel et al. , 2016b ) ( and thus the ComplEx model ( Hayashi & Shimbo , 2017 ) ) based on cross-correlation of embeddings to close the gap in expressivity with the Tucker decomposition for a fixed embedding size . Jenatton et al . ( 2012 ) express the relation matrices in RESCAL as low-rank combination of a family of matrices . We describe the link between these approaches and ours in Appendix 9.4 . None of these approach however studied the effect of their formulation on optimization , and reported results inferior to ours .

Summary: The paper introduces a novel tensor decomposition that is reminiscent of canonical decomposition (CP) with low-rank factors, based on the observation that the core tensor in Tucker decomposition can be decomposed, resulting in a model interpolating between CP and Tucker. The authors argue that a straight application of AdaGrad on this decomposition is inadequate, and propose Ada^{imp} algorithm that enforces rotation invariance of the gradient update. The new decomposition is applied to ComplEx model (called PComplEx) that demonstrates better performance than the baseline.