Re-evaluating Word Mover's Distance

1 INTRODUCTION The optimal transport ( OT ) distance is an effective tool for comparing probabilistic distributions . Applications of OT include image processing ( Ni et al. , 2009 ; Rabin et al. , 2011 ; De Goes et al. , 2012 ) , natural language processing ( NLP ) ( Kusner et al. , 2015 ; Rolet et al. , 2016 ) , biology ( Schiebinger et al. , 2019 ; Lozupone & Knight , 2005 ; Evans & Matsen , 2012 ) , and generative models ( Arjovsky et al. , 2017 ; Salimans et al. , 2018 ) . A prominent application of OT is the word mover ’ s distance ( WMD ) ( Kusner et al. , 2015 ) for document comparison . WMD regards a document as a probabilistic distribution of words , defines the underlying word geometry using pre-trained word embeddings , and computes the distance using the optimal transport distance between two word distributions ( i.e. , documents ) . WMD is preferable because it takes the underlying geometry into account . For example , bag-of-words ( BOW ) will conclude that two documents are dissimilar if they have no common words , whereas WMD will determine that they are similar if the words are semantically similar ( even if they are not exactly the same ) , as illustrated in Figure 1 . WMD has been widely used in NLP owing to this preferred property . For example , Kusner et al . ( 2015 ) and others ( Huang et al. , 2016 ; Li et al. , 2019 ) used WMD for document classification , Wu et al . ( 2018 ) used WMD for computing document embeddings , Xu et al . ( 2018 ) used WMD for topic modeling , Kilickaya et al . ( 2017 ) and others ( Clark et al. , 2019 ; Zhao et al. , 2019 ; 2020 ; Wang et al. , 2020a ; Gao et al. , 2020 ; Lu et al. , 2019 ; Chen et al. , 2020b ) used WMD for evaluating text generation . Many extensions have been proposed including supervised ( Huang et al. , 2016 ; Takezawa et al. , 2021 ) and fast ( Le et al. , 2019 ; Backurs et al. , 2020 ; Genevay et al. , 2016 ; Dong et al. , 2020 ; Sato et al. , 2020b ) variants . WMD is one of the fundamental tools used in NLP , and understanding the deep mechanism of WMD is crucial for further applications . The most fundamental application of WMD is document classification . The original study on WMD ( Kusner et al. , 2015 ) conducted extensive experiments using kNN classifiers . Figure 2 shows the classification errors reported in ( Kusner et al. , 2015 ) . This figure clearly shows that WMD is superior to classical baselines , BOW and TF-IDF1 Figure 2 is surprising in the following senses . First , WMD outperforms the classical baselines by excessively large margins . BOW and TF-IDF have long been recognized as effective tools for document classification . Although it is reasonable for WMD to outperform them , the improvements are surprisingly large . In particular , the performance is ten times better on the classic dataset and five times better on the bbcsport dataset . Such results are excessively impressive . Second , although TF-IDF is known to be more effective than BOW , it performs worse than BOW on the ohsumed , reuters , and amazon datasets . In fact , the number of misclassification doubles on the reuters datasets . In this paper , we point out the possibility that the evaluations conducted in the original WMD study ( Kusner et al. , 2015 ) are misleading . Specifically , we found that the main improvements of WMD were due to normalization . Using the same normalization , WMD is comparable to BOW and TF-IDF , or WMD achieves improvements of only two to eight percent at the price of heavy computations . We also confirm that TF-IDF is more effective than raw BOW if we employ adequate normalization . To understand the mechanism of WMD , we introduce an analogy between WMD and L1-normalized BOW . We experimentally find that the distribution of the distances between matched words is not Gaussian-like but two-modal in high dimensional spaces . We then find that not only the performance of WMD but also the distance values resemble those of BOW in high dimensional spaces . The contributions of this paper are summarized as follows . • We point out that the performance of WMD is not as high as we previously believed . The performance is comparable to classical baselines in document classification with the same normalization . • We introduce an analogy between WMD and L1-normalized BOW ( Proposition 1 ) and find that WMD resembles BOW in high dimensional spaces ( Figure 5 ) . • We point out several confusing aspects in the evaluations conducted in the original study on WMD . We suspect that many readers and researchers are unaware of these issues . Clarifying them is crucial for a solid evaluation and analysis in this field Reproducibility . We include our code in the supplementary materials . It contains a script to download datasets and pre-computed results , algorithm implementations , and evaluation code . 2 RELATED WORK . Word Mover ’ s Distance ( WMD ) and Optimal Transport ( OT ) in NLP . WMD ( Kusner et al. , 2015 ) is one of the most thriving applications of OT . WMD can take the underlying word geometry into account and inherit many elegant theoretical properties from OT . The success of WMD has facilitated many applications of OT in NLP . EmbDist ( Kobayashi et al. , 2015 ) is a method concurrent 1It should be noted that ( Kusner et al. , 2015 ) used many stronger baselines such as LSI and LDA . We focus on BOW and TF-IDF because ( i ) BOW was used as the base performance ( Figure 4 in ( Kusner et al. , 2015 ) , Figure 4 in ( Yurochkin et al. , 2019 ) ) , and ( ii ) BOW is a special case of WMD ( Propositoin 1 ) . with WMD . It also regards a document as a distribution of word embeddings but uses greedy matching instead of OT . Kumar et al . ( 2017 ) apply WMD to hidden representations of words instead of raw word embeddings . Yurochkin et al . ( 2019 ) consider a document as a distribution of topics and compute the document similarities using OT . They also use the OT distance for computing the ground distance of the topics . Alvarez-Melis et al . ( 2018 ) proposed a structured OT and applied it to a document comparison to take the positional consistency into account . Singh et al . ( 2020 ) consider a word as a probabilistic distribution of surrounding words and compute the similarity of the words using the OT distance of the distributions . Muzellec & Cuturi ( 2018 ) and others ( Deudon , 2018 ; Frogner et al. , 2019 ; Sun et al. , 2018 ) embed words or sentences into distributions instead of vectors and compute the distance between embeddings using OT . Chen et al . ( 2019 ) and others ( Li et al. , 2020 ; Chen et al. , 2020a ) regularize the text generation models based on the OT distance between the generated texts and the ground truth texts to improve the generation . Nested Wasserstein ( Zhang et al. , 2020a ) compares the distributions of sequences and is successfully used in imitation learning for text generation . Lei et al . ( 2019 ) use WMD to generate paraphrase texts for creating adversarial examples . Zhang et al . ( 2020b ) introduced partial OT to drop meaningless words . Michel et al . ( 2017 ) use a Gromov Wasserstein-like distance instead of the standard OT to compare documents . Zhang et al . ( 2016 ) and others ( Zhang et al. , 2017b ; a ; Grave et al. , 2019 ; Dou & Neubig , 2021 ) use OT to align word embeddings of different languages . Trapp et al . ( 2017 ) use WMD to compare compositional documents by weighting each document . Kilickaya et al . ( 2017 ) and others ( Clark et al. , 2019 ; Zhao et al. , 2019 ; 2020 ; Wang et al. , 2020a ; Gao et al. , 2020 ; Lu et al. , 2019 ; Chen et al. , 2020b ) used WMD for evaluating text generation . BERTScore ( Zhang et al. , 2020c ) is a relevant method , but it uses greedy matching instead of OT . To summarize , OT and WMD have been used in many NLP tasks . It is important to understand the underlying mechanism of WMD for further advancements in this field . Re-evaluation of Existing Methods . Back in 2009 , Armstrong et al . ( 2009 ) found that , although many studies have claimed statistically significant improvements against the baselines , most have employed excessively weak baselines , and the performance did not improve from the classical baselines in the information retrieval domain . Dacrema et al . ( 2019 ) recently found that many of the deep learning-based recommender systems are extremely difficult to reproduce , and for the methods whose results authors could reproduce , the performance was not as high as people had believed , and the deep approaches were actually comparable to classical baselines with an appropriate hyperparameter tuning . Their paper has had a large impact on the community and was awarded thr best paper prize at RecSys 2019 . Similar observations have also been made in sentence embeddings ( Arora et al. , 2017 ; Shen et al. , 2018 ) , session-based recommendations ( Ludewig et al. , 2019 ) , and graph neural networks ( Errica et al. , 2020 ) as well . In general , science communication suffers from publication and confirmation biases . The importance of reproducing existing experiments by third-party groups has been widely recognized in science ( Lin , 2018 ; Sculley et al. , 2018 ; Munafò et al. , 2017 ; Collins & Tabak , 2014 ; Goodman et al. , 2016 ) . 3 BACKGROUNDS . 3.1 PROBLEM FORMULATION . In this paper , we consider document classification tasks . Each document is represented by a bag-ofword vector x ∈ Rm , where m is the number of unique words in the dataset . The i-th component of x represents the number of occurrences of the i-th word in the document . We focus on the kNN classification following the original paper . The kNN classification gathers k samples of the smallest distances ( with respect to a certain distance ) to a test sample from the training dataset and classifies the sample to the majority class of the gathered labels . The design of the distance function is crucial for the performance of kNN classification . 3.2 WORD MOVER ’ S DISTANCE ( WMD ) . WMD provides an effective distance function utilizing pre-trained word embeddings . Let zi be the embedding of the i-th word . To utilize OT , WMD first regards a document as a discrete probabilistic distribution of words by normalizing the bag-of-word vector : nL1 ( x ) = x/ ∑ i xi . ( 1 ) WMD defines the cost matrix C ∈ Rm×Rm as the distance of the embeddings , i.e. , Cij = ‖zi−zj‖2 . The distance between the two documents x and x′ is the optimal value of the following problem : minimize P∈Rm×m ∑ ij CijPij ( 2 ) s.t . Pij ≥ 0 , P1 = nL1 ( x ) , P > 1 = nL1 ( x′ ) , where P > denotes the transpose of P , 1 ∈ Rm is the vector of ones . Intuitively , Pij represents the amount of word i that is transported to word j. WMD is defined as the minimum total distance to convert one document to another document . Let OT ( x , x′ , C ) ∈ R be the optimal value of eq . ( 2 ) .

Summary: This paper re-evaluates WMD and identifies issues with the original paper. It shows that the gain from the original paper is not the product of WMD but the normalization. When the normalization is controlled, WMD performs similarly to baseline. Finally, it shows WMD resembles classic BOW when normalization is controlled.