Modelling Monotonic and Non-Monotonic Attribute Dependencies with Embeddings: A Theoretical Analysis

1 . Introduction . Vector space embeddings are currently the dominant representation framework in Natural Language Processing , Computer Vision and Machine Learning . Essentially , these embeddings represent each entity of interest as a dense vector in some fixed-dimensional space . In addition , the attributes that are used to describe these entities are typically also encoded as vectors . For example , in the case of word embeddings , the entities correspond to words and the attributes correspond to the contexts in which these words occur , e.g . in the form of co-occurring words [ Mikolov et al. , 2013 , Pennington et al. , 2014 ] , syntactic dependencies [ Levy and Goldberg , 2014 , Vashishth et al. , 2019 ] or even full sentences [ Devlin et al. , 2019 ] . In inductive knowledge graph embedding , the entities from our framework correspond to the previously unseen entities for which we want to learn a representation , with the attributes representing links to known entities [ Hamaguchi et al. , 2017 ] . In the case of embeddingbased topic models , the entities correspond to documents and the attributes correspond to the associated topics and words [ Das et al. , 2015 , Li et al. , 2016 , He et al. , 2017 , Xun et al. , 2017 , Dieng et al. , 2020 ] . In zero-shot learning , the entities of interest are the category prototypes and the attributes correspond to semantic attributes of the categories [ Lampert et al. , 2013 ] or associated natural language terms [ Frome et al. , 2013 ] . An important advantage of attribute embeddings is that they can implicitly capture some of the dependencies that hold between the attributes . For instance , the use of embeddings for topic modelling stems from the desire to capture topic correlations [ He et al. , 2017 , Xun et al. , 2017 ] . However , it is currently unclear what kinds of dependencies can be captured in this way . Before we can address this question , we first need to clarify what it means that an embedding captures some dependency . To this end , consider an embedding model in which σ ( a·e ) represents the probability that entity e has attribute a , where σ is the sigmoid function and a and e are the embeddings of a and e. Among many others , the popular skip-gram word embedding model is of this kind [ Mikolov et al. , 2013 ] . Now consider the attributes parent , female and mother , and the associated dependency that mother ≡ parent∧female . What condition would need to be satisfied for the embedding to capture this dependency ? One possibility is to require that σ ( mother ·e ) = σ ( parent ·e ) ·σ ( female ·e ) for each vector e in the embedding space Rn . However , this condition can clearly not be satisfied ; e.g . for e = 0 we obtain the condition 0.5 = 0.25 . Similarly , it is easy to see that the condition σ ( mother · e ) = min ( σ ( parent · e ) , σ ( female · e ) ) only admits trivial solutions . Viewed from this angle , it is clear that popular embedding models are not able to capture even basic logical dependencies . For this reason , several alternative models have been proposed , in which attributes are modelled as linear subspaces [ Garg et al. , 2019 ] , axis-aligned cones [ Vendrov et al. , 2016 , Özçep et al. , 2020 ] , hyperboxes [ Vilnis et al. , 2018 ] or polytopes [ Gutiérrez-Basulto and Schockaert , 2018 ] , among others . In this paper , we follow a different direction , analysing whether embedding models can capture logical dependencies in a less demanding sense . In particular , we consider the common setting where the embedding of an entity e has to be learned from its known attributes , and the aim of the resulting embedding is to infer what other attributes e is likely to satisfy . We then say that the rule a1 ∧ ... ∧ an → b is captured if an entity which is known to have the attributes a1 , ... , an would be represented by a vector from which the attribute b would be predicted . Note that different variants of this setting can be considered , which depend on ( i ) how exactly entity vectors are constructed and ( ii ) how the resulting vectors are used for predicting the attributes of the entity . The aim of this paper is to analyse , for different variants , whether it is always possible to find an embedding which captures the rule a1 ∧ ... ∧ an → b iff that rule is entailed by some propositional knowledge base K. Note that in practice we typically do not have access to such a knowledge base K. However , the question of whether arbitrary propositional knowledge bases can , in principle , be modelled by a given embedding strategy is important , because if this is not the case , then it also means that some ( combinations of ) dependencies can not be learned . In addition to standard propositional entailment , we also look at non-monotonic consequence relations , which is important because attribute dependencies are often defeasible or probabilistic in nature . For instance , in a topic modelling context , we may assume that a document containing the word safari is related to the topic nature , and a document containing the word apple is related to the topic food . However , documents containing both of these words are more likely to be related to technology instead , given that Safari is the name of Apple ’ s internet browser . We may thus want that the rule safari→ topic : nature is captured , while the rule safari ∧ apple→ topic : nature is not . 2 . Problem Setting . We assume that each entity is associated with some attributes from a given set A . In the context of word embeddings , for instance , the set A could be the set of all contexts . Note that we do not consider knowledge graphs , where entities are described in terms of their relationship to other entities , rather than attributes . However , many inductive knowledge graph embedding models can still be cast as special cases of our framework , with the attributes then representing links to entities with known ( or pre-computed ) embeddings . Embedding and Labelling Functions If an entity e is associated with the attributes a1 , ... , an , we assume that its embedding is given by e = Emb ( a1 , ... , an ) for some embedding function Emb : 2A → Rm . The embedding functions that we consider in this paper will rely on pooling attribute vectors . In particular , we consider embedding functions of the following form : Emb ( a1 , ... , an ) = φ ( a1 , ... , an ) for some pooling function φ , where a ∈ Rm represents the embedding of attribute a ∈ A . For instance , we may have Emb ( a1 , ... , an ) = 1 n ( a1 + ... + an ) . Let us furthermore assume that we have a function Lab : R m → 2A that predicts attributes of entities based on their embeddings . Similar as for Emb , we will assume that the labelling function Lab relies on a scoring function that compares the embedding of e with an embedding of the considered attribute . In particular , we consider labelling functions of the following form : Lab ( e ) = { b ∈ A |ψ ( e , b̃ ) ≥ λb } . The embedding b̃ of the attribute b may in general be different from the attribute embedding b that is used for Emb , similar to how word embedding models learn two types of embeddings for each word . The scalar λb represents a threshold , which we allow to be attribute-dependent for generality . For instance , we could have Lab ( e ) = { b ∈ A |σ ( e · b ) ≥ 0.5 } = { b ∈ A | e · b ≥ 0 } . In the following , we will refer to ( φ , ψ ) as an embedding strategy and to ( Emb , Lab ) as an embedding . Note that the embedding ( Emb , Lab ) is determined by the embedding strategy ( φ , ψ ) , together with the embeddings a and ã of the attributes in A . Consequence Relations The function Lab ◦ Emb can be viewed as a logical consequence relation . In particular , we say that an embedding ( Emb , Lab ) captures the rule a1 ∧ ... ∧ an → b iff b ∈ Lab ( Emb ( a1 , ... , an ) ) , i.e . if an entity that is initially associated with the attributes a1 , ... , an would be predicted to have attribute b based on its embedding . Note that under this view , the dependency mother ≡ parent∧ female can be satisfied for the aforementioned choices of Emb and Lab , for instance by choosing parent = ( −1 , 1 ) , female = ( 1 , 1 ) and mother = ( 0 , 1 ) . However , now the issue is that a number of unwanted dependencies are also satisfied , such as female→ mother and parent→ mother . Therefore , rather than treating rules in isolation , the question we are interested in is the following : given a propositional knowledge base K and a given embedding strategy ( φ , ψ ) , does there exist a corresponding embedding ( Emb , Lab ) ( or , equivalently , do there exist attribute embeddings ) such that for all b ∈ A and { a1 , ... , an } ⊆ A we have that b ∈ Lab ( Emb ( a1 , ... , an ) ) iff K |= a1 ∧ ... ∧ an → b holds , where |= is the standard entailment relation from propositional logic . If this is the case , we say that the embedding strategy ( φ , ψ ) can simulate K. In Section 4 , we will similarly look at whether non-monotonic consequence relations can be simulated with embeddings . Throughout this paper , we will assume that the number of dimensions m can be chosen arbitrarily large , as our focus is on identifying limitations that exist regardless of dimensionality . An overview of our results is shown in Table 1 . 3 . Monotonic Reasoning . In Sections 3.1–3.3 , we first discuss embedding strategies which are not capable of capturing certain kinds of propositional knowledge bases . Crucially , these strategies cover many of the most popular embedding models . Section 3.4 then discusses embedding strategies which are capable of modelling arbitrary propositional knowledge bases . 3.1 Averaging Based Embeddings . One of the most natural choices for the embedding function Emb consists in averaging the attribute embeddings , i.e . : Embavg ( a1 , ... , an ) = 1 n ( a1 + ... + an ) ( 1 ) This choice corresponds , among many others , to the common strategy of learning sentence or document vectors by averaging word vectors . It is also closely related to the CBOW model from Mikolov et al . [ 2013 ] . As the labelling function , we first consider the common choice to model the probability that entity e has attribute a as σ ( e · a ) . However , since each condition of the form σ ( e · a ) ≥ δ , with 0 < δ < 1 , is equivalent to the condition e · a ≥ σ−1 ( δ ) , we use a simple dot product in the formulation : Labdot ( e ) = { b ∈ A | e · b̃ ≥ λb } ( 2 ) where λb ∈ R is a threshold that may in general be attribute-specific . We will use φavg and ψdot to denote the pooling functions associated with Embavg and Labdot . For the ease of presentation , throughout the paper , we will similarly use subscripts to link embedding and labelling functions to their corresponding pooling functions , rather than each time introducing these notations explicitly . Note that the embedding strategy ( φavg , ψdot ) also covers models where a linear transformation is applied to the average of the attribute embeddings , as is the case for the à la carte method from Khodak et al . [ 2018 ] . As the following counterexample shows , not all propositional knowledge bases can be simulated with embeddings of the form ( Embavg , Labdot ) . Counterexample 1 . Let K = { a ∧ b → x , c ∧ d → x } . We show that K can not be simulated by ( φavg , ψdot ) . Indeed , if a suitable embedding existed , among others the following inequalities would have to be satisfied : a + b 2 · x̃ ≥ λx c + d 2 · x̃ ≥ λx a + c 2 · x̃ < λx b + d 2 · x̃ < λx This is not possible , since the first two inequalities imply ( a + b + c + d ) · x̃ ≥ 4λ whereas the last two imply ( a + b + c + d ) · x̃ < 4λ . Another natural choice for the labelling function is to rely on Euclidean distance : Labdist ( e ) = { b ∈ A | d ( e , b̃ ) ≤ θb } ( 3 ) where θb ≥ 0 . It is easy to verify that the knowledge base from Counterexample 1 can be simulated by choosing a = ( −1 , 0 ) , b = ( 1 , 0 ) , c = ( 0 , −1 ) , d = ( 0 , 1 ) and x̃ = ( 0 , 0 ) . However , as the following counterexample shows , not all knowledge bases can be simulated using Embavg and Labdist either . Counterexample 2 . Let K = { a ∧ b → x , c ∧ d → x , a ∧ c → y , b ∧ d → y } . We show that K can not be simulated by ( φavg , ψdist ) . Indeed , suppose that a suitable embedding ( Embavg , Labdist ) existed . From the rules with x in the head , it follows that d 2 ( a+b 2 , x̃ ) + d2 ( c+d 2 , x̃ ) < d2 ( a+c 2 , x̃ ) + d2 ( b+d 2 , x̃ ) . This is equivalent with : ‖a‖2 + ‖b‖2 + ‖c‖2 + ‖d‖2 + 8‖x‖2 + 2 ( a · b + c · d ) − 4 ( a + b + c + d ) · x̃ < ‖a‖2 + ‖c‖2 + ‖b‖2 + ‖d‖2 + 8‖x‖2 + 2 ( a · c + b · d ) − 4 ( a + b + c + d ) · x̃ which simplifies to a · b + c · d < a · c + b · d. In the same way , using the rules with y in the head , we find a · b + c · d > a · c + b · d , which is a contradiction . To illustrate the relevance of these results , let the attributes a1 , ... , an correspond to the words that are observed in a document d , and suppose that other attributes , which are not observed , correspond to document categories . We may want the embedding model to capture rules such as tennis ∧ player ∧ won→ cat : sports , meaning that if the words tennis , player and won appear in a document , then it should belong to the category sports . Our results show that , in general , such dependencies can not be captured when Embavg is used in combination with Labdot or Labdist . For instance , this means that there are theoretical limitations to the kinds of categories that may be predicted from document embeddings when using the à la carte method from [ Khodak et al. , 2018 ] together with a linear classifier .

Summary: This paper looks at entity embeddings and theoretically tries to answer which kinds of semantic dependencies can be modeled by them. Mainly focusing on settings where the entity embeddings are obtained by pooling the embeddings of its attributes (although this is a strong assumption), this paper presents various theoretical limitations of the embedding approaches in learning logical/structured attribute representations with known logical or structured dependencies in practice. The paper shows that some of the most popular embedding models are not able to capture basic logical rules.