Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper re-evaluates WMD and identifies issues with the original paper. It shows that the gain from the original paper is not the product of WMD but the normalization. When the normalization is controlled, WMD performs similarly to baseline. Finally, it shows WMD resembles classic BOW when normalization is controlled.

main_review: # Strengths

This paper identifies issues with the original paper proposing WMD and during the evaluation, it shows the gain mainly due to normalization instead of WMD. It offers detailed analysis and experiments to support its claim.

# Weaknesses

n/a

summary_of_the_review: This paper revisits the original WMD paper and offers a detailed evaluation showing what contributes to the performance gain: normalization instead of WMD.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper empirically shows that the performance of WMD is not as high as initially reported, and the real performance is comparable to  L1-normalized BOW, which can be formulated as a specific case of WMD. The authors also find that WMD resembles BOW in high-dimensional spaces. 
main_review: This paper targets a fairer evaluation of WMD. This is a critical problem worth exploring as WMD is a fundamental technique in various research fields. The core claims in this paper include: 
1. In the original study of WMD, many duplicate samples exist in the datasets, and applying L2 norm to word embeddings is not explicitly stated.
2. The superiority of WMD over BOW and TF-IDF will weaken enormously by normalizing the BOW and TF-IDF.
3. Both the normalization of word vectors and document-level distance metric (L1 or L2) will impact performance.
4. WMD coincides with L1/L1 BOW empirically and theoretically, consistent with the two modalities characteristic of high-dimensional word embeddings.

By designing extensive experiments, the authors present the above observations and corresponding suggestions. They also provide clean datasets without duplicated samples and related code for further research. Following are some of my questions.

In figure 4, what metric is used for word-level distance?  Experiments show that L1 document-level distance with L1-normalized BOW/TF-IDF will generate the best performance. Regarding WMD, I wonder about the effects of different word-level distances (e.g., L1, L2, or Cosine). For example, WMD uses L2 word-level distance, then what about using other metrics? Besides, what if we use L1 normalization for word embeddings in WMD? 

In my experience, the performance of WMD is extensively affected by the removal of appropriate stop-words. This paper only mentions the stop-word strategy on page 7.  I am wondering if the author removed the stop words in other experiments.

Table 3-5 shows that removing OOV words brings significant performance degradation on bbcsports and ohsumed, which is somewhat against expectations. Is there any explanation for this observation?


summary_of_the_review: This paper re-evaluates the Word Mover's Distance by well-designed experiments. They reveal the true performance of WMD and draw its relationship with L1-normalized BOW. I am more inclined to accept this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a re-evaluation of a well known distance metric for documents with word embeddings. The authors identify some missing information from the original paper and they provide some extra analyses.
main_review: This paper presents a thorough re-evaluation of the word mover's distance paper by Kusner et al. (2015), focusing on a number of points that are considered misleading. The paper has a point about the fact that the normalization of the word vectors used in WMD by Kusner et al. was not mentioned in the paper, even though it was possible to find out in the code released by the authors (misleading point 2). I think that while welcome as a clarification, I think calling this omission misleading is a bit of a stretch, especially given that this was possible to find in the code released by the authors. The paper then criticises Kusner et al. for not using such a nornalization for the document vectors obtained by BOW and TFIDF, which they find improves their results (misleading point 3). However, it should be noted that the original paper included other methods for comparison, which performed rather competitive to WMD, so while I agree with this paper that Kusner et al. didn't do justice to BoW and TFIDF, they did show that other methods were competitive to WMD but not as good. This paper also admits this too. Furthermore, in WMD the normalization is at the word embedding level, while the L1/L2 normalization on BoW/TFIDF is at the document level, thus it is not a direct correspondence. Thus I don't think there is much to see here, unless the paper would like to be more of a case of how to make the most of token-matching distance metrics.

The first misleading point is about the datasets containing duplicates. But not sure this has to do that much with the method of Kusner et al. per se, especially given that the datasets are well-known and used frequently. Perhaps they shouldn't be, and a paper could be written about this. As for the misleading point 4, I have to disagree with this paper. The example from figure 1 in Kusner et al illustrates well how word embeddings could help us do something more useful than 0-1 distances in token matching metrics, which would be the case of their example. The arguments in  section 5 on the modes (not modalities) are not particularly tight. If anything, the BoW representation of a word or document is much more high dimensional than that of a (300 dim) word embedding, assuming that one has thousands of words typically. The point that Kusner et al. made was the word embedding distances are more informative than word matching, and this is confirmed in the plots of figure 4: different word pairs get different distances.

On the whole I believe that much of the criticism of this paper is not justified. The main claims of the Kusner et al. paper still hold, and the experiments in this paper confirm them in my mind. I would suggest that perhaps a better paper would be assessing the evaluation practices in distance metrics more broadly, using multiple state-of-the-art methods. While WMD influenced the way people are thinking, using it for this purpose with old embeddings on text classification which is nowadays dominated by fine-tuning pre-trained language models is not particularly informative.
summary_of_the_review: On the whole, while I appreciate the work to bring to the community's attention some extra information and analyses of the Word Mover's Distance paper, the paper is not very informative. I think a much more interesting paper would be to survey the evaluation of multiple distance metric learning approaches and discuss common pitfalls more broadly, rather than pick on one paper in the context of a task that it wouldn't be considered a standard method for (fine-tuning BERT style pre-trained models is the go-to method for text classification and is much better than word embeddings from 2015 with kNN).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper shows that when standard baseline methods are carefully evaluated, such as using L1 normalization of the tf-idf/bow vector, then the performance difference between the baselines and the WMD distance based methods shrinks considerably.

The authors also claim that the findings in the paper are more general in the sense, that the distance between the embeddings of two words behaves more and more like a delta function, which is bimodal : close to one when the words are distinct, and close to 0 when the words are the same.

main_review: ### Strength
The paper presents experimental results that the WMD distance doesn't improve performance by a factor of 100% but rather by a factor of 10%. This corrects some misconceptions that may have arisen in a particular sub-field of NLP.

### Weakness
The paper will have a low impact, and is too focused on refuting the results of a single paper. I believe that this paper may be well suited to conferences devoted to the particular application area of document classification or NLP in general, but as it is the paper's methodological contribution, or technical contribution is too little.

The paper presents a sub-section on the "re-evaluation of existing methods" in section 2 to answer some of this criticism, but all the papers in that section were much more wide-ranging in their focus-area. E.g. the (Dacrema et al.) paper was not focused on the results of a single paper but on multiple papers, and the paper by (Arora et al. 2017) presented a novel method. The paper by (Shen et al. 2018) was accepted at ACL which is an NLP conference more focused on NLP tasks.

summary_of_the_review: The paper corrects some errors made in an earlier paper by (Kusner et al. 2015) and presents a careful evaluation of baselines for document classification applications. However, I believe that the paper is more suited for an NLP conference than ICLR.
review_title: null

Meta-review: The authors conduct extensive experiments to show that there were some errors in the original claims of the WMD paper and as opposed to what was claimed in the original paper, WMD does not outperform simpler baselines like BOW and TF-IDF. The authors claim that this is significant because WMD is widely used in the literature and hence pointing out errors in the original paper may help the community. 

Out of the 4 reviewers, 1 reviewer wrote a very short review and despite reminders did not elaborate on the reasons for a "Strong Accept". The other reviewer with a "Strong Accept" rating also did not champion the paper in the final discussions. The main objection of the two reviewers who were not in favor of accepting the paper were that (i) it focuses on cirticising a single paper and (ii) some of the criticism is not fair. In response, the authors claim that given the huge amount of derivative work which uses or builds upon the original WMD metric it is crucial to point out these errors. 

Having read the reviews and the responses, it is not clear to me whether such a paper which focuses only on such criticism of a single paper (not matter how popular it is) has enough merit in being accepted. Alternatively, if such criticism was a part of a broader work (maybe a work on new document similarity metrics) then it would have more merit. Further, it should be noted that of the 4 misleading conclusions of the original paper identified by the authors at least 2 are debatable (one being an error in the dataset and the other being a normalisation technique which was not mentioned in the paper but used in the code). The authors have also rephrased one of the original 4 misleading points and from the discussion it seems that they agree it is not misleading. It would have been easier for me to accept the paper if it had a new metric and ablation studies which showed that (i) Hey, normalisation is important and should be done for all baseline algorithms that are being compared (ii) Hey, there are errors in the dataset which affect the results
Decision: Reject