Context Mover's Distance & Barycenters: Optimal transport of contexts for building representations

1 INTRODUCTION . One of the main driving factors behind the recent successes in machine learning and natural language processing has been the development of better representation methods for data modalities . Examples include continuous vector representations for language ( Mikolov et al. , 2013 ; Pennington et al. , 2014 ) , Convolutional Neural Network ( CNN ) based text representations ( Collobert & Weston , 2008 ; Kim , 2014 ; Kalchbrenner et al. , 2014 ; Severyn & Moschitti , 2015 ; Deriu et al. , 2017 ) , or via other neural architectures such as RNNs , LSTMs ( Hochreiter & Schmidhuber , 1997 ; Kiros et al. , 2015 ) , all sharing one central idea – to map input entities to dense vector embeddings lying in a low-dimensional latent space where the semantics of the inputs are preserved . While these existing methods represent each entity of interest ( e.g. , a word ) directly as a single point in space ( i.e. , its embedding vector ) , we here propose a fundamentally different approach . We focus on co-occurrence information of the entities and their contexts ( e.g . context words ) , and leverage embeddings of the contexts instead of the original entities . So instead of a single point per entity , we represent each entity by the histogram of its contexts , where the contexts themselves are represented as points in a suitable metric space . This allows us to cast the distance between histograms associated with the contexts as an instance of the optimal transport problem ( Monge , 1781 ; Kantorovich , 1942 ; Villani , 2008 ) . Our resulting framework then intuitively seeks to minimize the cost of moving the contexts of a given entity to the contexts of another , which motivates the naming Context Mover ’ s Distance ( CMD ) . Note that the contexts here can be words , phrases , sentences , or any generic entities co-occurring with our entities to be represented , and these entities further could be of various kinds , including e.g. , products such as movies or web-advertisements ( Grbovic et al. , 2015 ) , nodes in a graph ( Grover & Leskovec , 2016 ) , sequence data , or any other entities ( Wu et al. , 2017 ) . Any co-occurrence structure will allow construction of the histogram information , which is the crucial building block of our approach . The main motivation for our proposed approach here comes from the domain of natural language , where the entities ( words , phrases , or sentences ) generally have different semantics depending on the context under which they are present . Hence , it is important that we consider representations that are able to effectively capture such inherent uncertainty and polysemy , and we will argue that histograms ( or probability distributions ) over embeddings allows to capture more of this information compared to point-wise embeddings alone . We will call this histogram over contexts embeddings as the distributional estimate of our entity of interest , while we refer to the individual embeddings of contexts as point estimates . The connection to optimal transport at the level of entities and contexts paves the way to make better use of its vast toolkit ( like Wasserstein distances , barycenters , barycentric coordinates , etc . ) for applications in NLP , which in the past has primarily been restricted to document distances of original words ( Kusner et al. , 2015 ; Huang et al. , 2016 ) , as opposed to contexts . Thanks to the entropic regularization introduced by Cuturi ( 2013 ) , optimal transport computations can be carried out efficiently in a parallel and batched manner on GPUs . Contributions : . • Employing the notion of optimal transport of contexts as a distance measure , we illustrate how our framework can be of benefit for various important tasks , including word and sentence representations , sentence similarity , as well as hypernymy ( entailment ) detection . The method is static and does not require any additional learning , and can be readily used on top of existing embedding methods . • The resulting representations via the transport map give a clear interpretation of the resulting distance ( see also Figure 1 ) , on top of the co-occurrence information . • Our Context Mover ’ s Distance can be used to measure any kind of distance ( even asymmetric ) between words , by defining a suitable underlying cost on the movement of contexts , which we show can lead to a state-of-the-art metric for word entailment . • Defining the transport over contexts has the added benefit that the representations are compositional - they directly extend from entities to groups of entities ( of any size ) , such as from word to sentence representations . To this end , we utilize the notion of Wasserstein barycenters , which to the best of our knowledge has never been considered in the past . The proposed framework is not specific to words or sentences but can allow building unsupervised representations for any entity and composition of entities , where a co-occurrence structure can be devised between entities and their contexts , and we leave this direction for a future work . 2 RELATED WORK . Most of the previous work in building representations for natural language has been focused towards vector space models , in particular , popularized through the groundbreaking work in Word2vec ( Mikolov et al. , 2013 ) and GloVe ( Pennington et al. , 2014 ) . The key idea in these models has been to map words which are similar in meaning to nearby points in a latent space . Based on which , many works ( Levy & Goldberg , 2014a ; Melamud et al. , 2015 ; Bojanowski et al. , 2016 ) have suggested specializing the embeddings to capture some particular information required for the task at hand . One of the problems that still persists is the inability to capture , within just a point embedding , the various semantics and uncertainties associated with the occurrence of a particular word ( Huang et al. , 2012 ; Guo et al. , 2014 ) . A recent line of work has proposed the view to represent words with Gaussian distributions or mixtures of Gaussian distributions ( Vilnis & McCallum , 2014 ; Athiwaratkun & Wilson , 2017 ) , or hyperbolic cones ( Ganea et al. , 2018 ) for this purpose . Also , concurrent works by Muzellec & Cuturi ( 2018 ) and Sun et al . ( 2018 ) have suggested using elliptical and Gaussian distributions endowed with a Wasserstein metric respectively . While these already provide richer information than typical vector embeddings , their form restricts what could be gained by allowing for arbitrary distributions . In addition , hyperbolic embeddings ( Nickel & Kiela , 2017 ; Ganea et al. , 2018 ) are so far restricted to supervised tasks ( and even elliptical embeddings ( Muzellec & Cuturi , 2018 ) to the most extent ) , not allowing unsupervised representation learning as in the focus of the paper here . To this end , we propose to represent each word with a distributional estimate ( i.e. , histogram over context embeddings ) , that inherently relies upon the empirically obtained co-occurrence information of a word and its contexts . Hence , this naturally allows for the use of optimal transport ( or Wasserstein metric ) in the space containing the contexts , and leads to an interpretation ( Figure 1 ) which is not available in the above approaches . Amongst the few explorations of optimal transport in NLP , i.e. , document distances ( Kusner et al. , 2015 ; Huang et al. , 2016 ) , topic modelling ( Rolet et al. , 2016 ; Xu et al. , 2018 ) , document clustering ( Ye et al. , 2017 ) , bilingual lexicon induction ( Zhang et al. , 2017 ) , or orthogonal Procrustes mapping ( Grave et al. , 2018 ) , the focus has been on transporting words directly . For example , the Word Mover ’ s Distance ( Kusner et al. , 2015 ) casts finding the distance between documents as an optimal transport problem between their bag of words representation . Our approach is different as we consider the transport over contexts instead , and use it to propose a representation for words or entities . This provides the added flexibility to establish any kind of distance between entities and extend the representation to composition of entities in a principled manner , as will be illustrated further through the examples of entailment detection and sentence representation respectively . 3 BACKGROUND ON OPTIMAL TRANSPORT . Optimal Transport ( OT ) provides a way to compare two probability distributions defined over a space G ( commonly known as the ground space ) , given an underlying distance or more generally a cost of moving one point to another in the ground space . In other terms , it lifts a distance between points to a distance between distributions . Other methods of comparing distributions , such as Kullback-Liebler ( KL ) , squared Hellinger , etc. , only focus on the probability mass values , thus ignoring the geometry of the ground space : something which we utilize throughout this work via OT . Also , some divergences like KL are not defined when the supports of distributions under comparison don ’ t match . Hence , we give a short yet formal background on OT in the discrete case . Linear Program Formulation . Consider an empirical probability measure of the form µ =∑n i=1 aiδ ( xi ) where X = ( x1 , . . . , xn ) ∈ Gn , δ ( x ) denotes the Dirac ( unit mass ) distribution at point x ∈ G , and ( a1 , . . . , an ) lives in the probability simplex Σn : = { p ∈ Rn+ | ∑n i=1 pi = 1 } . Now given a second empirical measure , ν = ∑m j=1 bjδ ( yj ) , with Y = ( y1 , . . . , ym ) ∈ Gm , and ( b1 , . . . , bm ) ∈ Σm , and if the ground cost of moving from point xi to yj is denoted by Mij , then the Optimal Transport distance between µ and ν is the solution to the following linear program . OT ( µ , ν ; M ) : = min T∈Rn×m+ ∑ ij TijMij such that ∀i , ∑ j Tij = ai , ∀j , ∑ i Tij = bj . ( 1 ) Here , the optimal T ∈ Rn×m is referred to as the transportation matrix : Tij denotes the optimal amount of mass to move from point xi to point yi . Intuitively , OT is concerned with the problem of moving goods from factories to shops in such a way that all the demands are satisfied and the overall transportation cost is minimal . Distance . When G = Rd and the cost is defined with respect to a metric DG over G ( i.e. , Mij = DG ( xi , yj ) p for any i , j ) , OT defines a distance between empirical probability distributions . This is the p-Wasserstein distance , defined as Wp ( µ , ν ) : = OT ( µ , ν ; D p G ) 1/p . In most cases , we are only concerned with the case where p = 1 or 2 . The cost of exactly solving OT problem scales at least inO ( n3 log ( n ) ) ( n being the cardinality of the support of the empirical measure ) when using network simplex or interior point methods . Following Cuturi ( 2013 ) , we consider the entropy regularized Wasserstein distance , Wp , λ ( µ , ν ) , where the search space for the optimal T is instead restricted to a smooth solution close to the extreme points of this linear program . The regularized problem can then be solved efficiently using Sinkhorn iterations ( Sinkhorn , 1964 ) , albeit at the cost of some approximation error . The regularization strength λ ≥ 0 controls the accuracy of approximation and recovers the true OT for λ = 0 . The cost of the Sinkhorn algorithm is only quadratic in n at each iteration . Recently , Altschuler et al . ( 2017 ) have shown that the Sinkhorn algorithm converges in a number of iterations independent of n , thus resulting in an overall complexity of Õ ( n2/ 3 ) for an -accurate solution . Barycenters . Further on in our discussion , we will make use of the notion of averaging in the Wasserstein space . More precisely , the Wasserstein barycenter , introduced by Agueh & Carlier ( 2011 ) , man poaching giraffe ivory en d an ge re d sk in s sp ec ie s fo ss ils El ep h an t Mammal is a probability measure that minimizes the sum of ( p-th power ) Wasserstein distances to the given measures . Formally , given N measures { ν1 , . . . , νN } with corresponding weights η = { η1 , . . . , ηN } ∈ ΣN , the Wasserstein barycenter can be written as Bp ( ν1 , . . . , νN ) = arg minµ ∑N i=1 ηiWp ( µ , νi ) p. We similarly consider the regularized barycenter Bp , λ , using entropy regularized Wasserstein distances Wp , λ in the above minimization problem , following Cuturi & Doucet ( 2014 ) . Employing the method of iterative Bregman projections ( Benamou et al. , 2015 ) , we obtain an approximation of the solution at a reasonable computational cost .

Summary: The submission explores a new form of word representation based on a histogram over context word vectors, allowing them to measure distances between words in terms of optimal transport between these histograms. The authors speculate that this may allow better representations of polysemous words. The approach is mathematically elegant, and requires no additional training on top of existing approaches like Glove. To improve efficiency, they use clustering on context vectors. They present results on various semantic textual similarity and hypernym detection tasks, outperforming some baselines.