Situating Sentence Embedders with Nearest Neighbor Overlap
Authors: Lucy H. Lin
Noah A. Smith
Venue:  (2020)
Keywords: sentence embeddings
nearest neighbors
semantic similarity

Abstract: As distributed approaches to natural language semantics have developed and diversified, embedders for linguistic units larger than words (e.g., sentences) have come to play an increasingly important role.  To date, such embedders have been evaluated using benchmark tasks (e.g., GLUE) and linguistic probes.  We propose a comparative approach, nearest neighbor overlap (N2O), that quantifies similarity between embedders in a task-agnostic manner.  N2O requires only a collection of examples and is simple to understand: two embedders are more similar if, for the same set of inputs, there is greater overlap between the inputs' nearest neighbors.  We use N2O to compare 21 sentence embedders and show the effects of different design choices and architectures.

Under review as a conference paper at ICLR 2020
SITUATING SENTENCE EMBEDDERS WITH NEAREST NEIGHBOR OVERLAP
Anonymous authors Paper under double-blind review
ABSTRACT
As distributed approaches to natural language semantics have developed and di- versified, embedders for linguistic units larger than words (e.g., sentences) have come to play increasingly important role. To date, such embedders have been evaluated using benchmark tasks (e.g.. GLUE) and linguistic probes. We propose comparative approach, nearest neighbor overlap (N20), that quantifies similar- ity between embedders in task-agnostic manner. N2O requires only collection of examples and is simple to understand: two embedders are more similar if, for the same set of inputs, there is greater overlap between the inputs' nearest neigh- bors. We use N2O compare 21 sentence embedders and show the effects of different design choices and architectures.
INTRODUCTION
Continuous embeddings-of words and of larger linguistic units-are now ubiquitious in NLP. The success of self- self-supervised pretraining methods that deliver embeddings from raw corpora has led to proliferation of embedding methods, with an eye toward "universality" across NLP tasks. Our focus here is sentence embedders, and specifically their evaluation. As with most NLP components, intrinsic ge.g.. Conneau al. 2018) and extrinsic (e.g., GLUE;; Wang et 2019) evaluations have emerged for sentence embedders.
Our approach, nearest neighbor overlap (N20), is different: it compares a pair of embedders in a linguistics- and task-agnostic manner, using only large unannotated corpus. The central idea is that two embedders are more similar if, for a fixed query sentence, they tend to find nearest neighbor sets that overlap to large degree. By drawing random sample of queries from the corpus itself, N2O can computed on in-domain data without additional annotation, and therefore can help inform embedder choices in applications such as text clustering (Cutting et al.) 1992). information retrieval (alton & Buckley 1988), and open-domain question answering (Seo et al. 2018), among others.
After motivating and explaining the N2O method ((2), we apply it to 21 sentence embedders ($3 Our findings (55) reveal relatively high functional similarity among averaged static (noncontextual word type embeddings, strong effect of the use of subword information, and that BERT and GPT are distant outliers. In (6 we demonstrate the robustness of N2O across different query samples and probe sizes. We also illustrate additional analyses made possible by N2O: identifying embedding- space neighbors of query sentence that are stable across embedders, and those that are not (77): and probing the abilities of embedders to find known paraphrase (88). The latter reveals considerable variance across embedders' ability to identify semantically similar sentences from a broader corpus.
CORPUS-BASER EMBEDDING COMPARISON
We first motivate and introduce our nearest neighbor overlap (N20) procedure for comparing em- bedders (maps from objects to vectors). Although experiment with sentence embedders in this paper, we note that this comparison procedure can be applicd to other types of embedders (c.g., phrase-level or document-level)
We also note that nearest neighbor search has heen frequently used on word embeddings (e.g., word anal- ogy tasks).

---

Under review as a conference paper at ICLR 2020
function N2O eg, K,
M gave the dictionary
for each query q E {q:);= do neighbors f nearest C.. 4 k) neighbors n + nearest qj C, oij] f |neighbors A 4 neighbors end for
gave the book to M
M gave the book to
return Σ, oljl/(k x x)) end function
gave the book to M
Figure 2: Computation of N2O for two em- bedders, e^ and eg, using corpus C: the number of nearest neighbors is given by k. n is the number of queries (q.... (q)).. which are sampled uniformly from the corpus with- out replacement. The output is in [0, I], where 0 indicates no overlap between near- est neighbors for all queries, and indicates perfect overlap.
M gave the book
M gave the dictionary to J
Figure 1: A toy example of two sentence em- bedders and how they might affect nearest neighbor sentences.
2.1 DESIDERATA
We would like to quantify the extent to which sentence embedders vary in their treatment of "similar- ity." For example, given the sentence Mary gave the book to John, embedders based on bag-of-wordh will treat John gave the book to Mary as being maximally similar to the first sentence, whereas other embedders may treat Mary gave the dictionary to John more similar; comparison should re- flect this intuition. We would also like to focus on using naturally-occuring text for our comparison. Although there is merit in expert-constructed examples (see linguistic probing tasks referenced in ( we have little understanding of how these models will generalize to text from real documents; many application settings involve computing similarity across texts in corpus. Finally, would like our evaluation to be task-agnostic, since we expect embeddings learned from large unannotated corpora in a self- supervised (and task-agnostic) manner to continue to play an important role in NLP.
As result, we base our comparison on nearest neighbors: first, because similarity is often assumed to correspond to nearness in embedding space (e.g.. Figure I second, because nearest neighbor methods are used directly for retrieval and other applications; and finally, because the nearest neigh- bors of a sentence can be computed for any embedder on any corpus without additional annotation.
2.2 ALGORITHM
Suppose we want to compare two sentence embedders, cx(+) and c.(:), where each embedding method takes as input a natural language sentence s and outputs a d-dimensional vector. For our purposes, we consider variants trained on different data or using different hyperparameters, even with the same parameter estimation procedure, to be different sentence embedders.
Take corpus C, which likely to have some semantic overlap in its sentences, and segment into sentences St....,Sicy Randomly select small subset of the sentences in C. as "queries" (qu...., qn). To see how similar е and eg are, we compute the overlap in nearest neighbor sentences, averaged across multiple queries; the algorithm is in Figure 2 nearest qj C, k) returns the & nearest neighbor sentences in corpus to the query sentence qj. where sentences are embedded with e, There are different ways to define nearness and distance in embedding spaces (e.g., using cosine similarity Euclidean distance); in this paper we cosine similarity.
We can think about this procedure as randomly probing the sentence vector space (through the n query sentences) from the larger space of the embedded corpus, under sentence embedder e;; in some sense, lo controls the depth of the probe. The N2O procedure then compares the sets of sentences recovered by the probes.
"One of these will be the query sentence itself, since we sampled it from the corpus; we assume nearest ignores when computing the k-nearest-neighhor lists.
2

---

Under review as a conference paper at ICLR 2020
3 SENTENCE EMBEDDING METHODS
In the previous section, we noted that consider "sentence embedder" encompass how it was trained, which data it was trained on, and any other hyperparameters involved in its creation. In this section, we first review the broader methods behind these embedders, turning to implementation decisions in
3.1 TF-IDF
We consider tf-idf, which has been clasically used in information retrieval settings. The tf-idf of word token is based off two statisties: term frequency (how often a term appears in document) and inverse document frequency (how rare the term is across all documents). The vector representation of the document the idf-scaled term frequencies of its words; in this work we treat each sentence as "document" and the vocabulary-length tf-idf vector as its embedding.
3.2 WORD EMBEDDINGS
Because sentence embeddings are often built from word embeddings (through initialization when training or other composition functions), we briefly review notable word embedding methods.
Static embeddings. We define "static embeddings" to be fixed representations of every word type in the vocabulary, regardless of its context. We consider three popular methods: word2vee (Mikolov et al. 2013) embeddings optimized be predictive of word given its context (continuous bag of words) or vice versa (skipgram); GloVe (Pennington et al. 2014) embeddings learned based on global cooccurrence counts; and FastTexx (Conneau al. (017), extension of word2vec which includes character n-grams (for computing representations of ofo-ol-vocabulary words).
Contextual embeddings. Contextual word embeddings, where a word token's representation is dependent on its context, have become popular due to improvements over hate-of-the-ar wide variety of tasks. We consider:
ELMo PPters et al., 2018) embeddings are generated from a multi-layer, bidirectional recurrent language model that incorporates character-level information.
GPT (Radford et al.) 2018 embeddings are generated from a unidirectional language model with multi-layer transformer decoder; subword information included via byte-pair encoding (BPE; Sennrich et al. 2016).
BERT Devlin et al. 2019) embeddings are generated from a transformer-based model trained to predict (a) word given both left and right context, and (b) whether a sentence is the "next sentence" given previous sentence. Subword information is incorporated using the WordPiece model (Schuster & Nakajimal 2012).
Composition of word embeddings. The simplest way to obtain sentence's embedding from its sequence of words is average the word embeddings Despite the fact that averaging discards word order, performs surprisingly well on sentence similarity, NLI, and other downstream tasks
(Wieting et al. 2016 Arora et al. 2017
In the case of contextual embeddings, there may be other conventions for obtaining the sentence embedding, such using the embedding for special token position in the sequence. With BERT, the [CLS] token representation (normally used as input for classification) is also sometimes used sentence representation; similarly, the last token's representation may used for GPT.
In the case of GPT and BERT, which yield subword embeddings, treat those would standard word embeddings.
Arora et al. 2017) also suggest including PCA-based projection with word embedding averaging to further improve downstream performance. However, because our focus is behavior of the embeddings themselves, do apply this projection here.
3

---

Under review as a conference paper at ICLR 2020
Embed. method
Composition
Dim.
Model/data description
uf-idf
w/a
[V]
tf-idf statistics obtained on Gigaword corpus (2010 slice)
word2vec
average
300
Google News (3B tokens)
100
Wikipedia 2014 Gigaword 5 (6B tokens, uncased)
Vl
average
300
Wikipedia 2014 Gigaword 5 (6B tokens, uncased)
300
Common Crawl (840B tokens, cased)
300
Wikipedia + UMBC statmt.org (16B tokens)
300
subword information
Text
average
300
Common Crawl (600B tokens)
300
subword information
pretrained small model (1 Billion Word Benchmark)
ELMo
average
1024
pretrained original model (1 Billion Word Benchmark)
1024
pretrained original/S. 5.BB model (Wikipedia/news)
[C1S]
768
pretrained cased/base model (Wikipedia BooksCorpus)
BERT
average
768
[CLS]
768
finetuning MultiNTI (matched subset)
average
768
last
512
GPT
pretrained model (110M paramcters) trained on BooksCorpus
average
512
InferSent
n/a
4096
VI (Glo based) model, trained on SNLI
512
deep averaging network (DAN) encoder; multitask training transformer encoder; multitask training
USE
n/a
512
Table I: Details of the pretrained sentence embedders test in this paper. For methods which produce word embeddings, "composition" denotes how a single embedding was obtained from the sentence's 's word embeddings. ELMo embeddings are averaged across the three bi-LSTM layers; BERT and GPT embeddings come from the final hidden layer. All of the models besides tf-idf and the fine-tuned version of BERT are common pretrained versions; further details are in Appendix
3.3 ENCODERS
A more direct way to obtain sentence embeddings is to learn an encoding function that takes in sequence tokens and outputs a single embedding: often this is trained using a relevant supervised task. We consider two encoder-based methods:
InferSent (Conneau et al. 2017): : supervised training on the Stanford Natural Language Inference (SNLL: Bowman al. 2015) dataset; the sentence encoder provides representa- tions for the premise and hypothesis sentences, which are then fed into clasifier.
Universal Sentence Encoder (USE; Cer et al. 2018); supervised, multi-task training on several semantie tasks (including semantic textual similarity); sentences are encoded either with a deep averaging network or a transformer.
EXPERIMENTAL DETAILS
Our main experiment is broad comparison, using N2O, of the embedders discussed above and listed in Table T Despite the vast differences methods, N2O allows us to situate each in terms of its functional similarity the others.
N20 computation. We describe a N2O sample as, for a given random sample of queries, the computation of N20 CB, C, for every pair of sentence embedders as described ! using cosine similarity to determine nearest neighbors. The results in are with & (the number of sen- tences retrieved) set to 50, averaged across five samples of 100 queries. illustrate the effects of different & and N2O samples in 6
CB,C,
Corpus. For our corpus, we draw from the English Gigaword Parker et al. 2011). which con- tains newswire text from seven sources. For computational feasibility, the articles
4

---

shows N2O Under review as a conference paper at ICLR 2020
from 2010, for a total of approximately 8 million unique sentences R We note preprocessing details
(segmentation, tokenization) in Appendix
Queries. For each N2O sample, we randomly select 100 ledes (opening sentences) from the news articles of corpus, and use the same ones across all embedders. Because the Gigaword corpus contains text from multiple news sources covering events over the same time period, it is likely that the corpus will contain semantically similar sentences for a given lede. The average query length is 30.7 tokens (s.d. 10.2); an example query is: "Sandra Kiriasis and brakewoman Stephanie Schneider of Germany have won the World Cup bobsled race Lake Placid."
Sentence embedders. Table SaaleeJdtaaiie details the sentence embedders we use in our experiments. gen- eral, we use popular pretrained versions of the methods described in $ We also select pretrained variations of the same method (e.g., FastText embeddings trained from different corpora) to permit more controlled comparisons. In couple of cases, we train/finctune models of our own: for tf-idf, we compute frequency statistics using our corpus, with each sentence as its own "document"; for BERT, we use the Hugging Face implementation with default hyperparaamtters @6 and finetune using the matched subset of MultiNLI (Williams et al. 2018) for three epochs (dev. accuracy 84.1%).
We note that additional embedders are easily situated among the ones tested in this paper by first computing nearest neighbors of the same query sentences, and then computing overlap with the nearest neighbors obtained in this paper. To enable this, the code, query sentences, and nearest neighbors per embedder and query will be publicly available.
RESULTS
In this section, present the results from the experiment described in $4 Fig. 3
between each pair of sentence embedders listed in Table [ the values range from 0.04 to 0.62. While even the maximum observed value may not seem large, we reiterate that overlap is computed over two draws of k 50 sentences (nearest neighbors) from approximately 8 million sentences, and even an N2O of 0.04 unlikely from random chance alone.
Averages of static word embeddings. We first observe that there is generally high N2O among this set of embedders in comparison to other categories (Fig. 4 left). Some cases where N20 is high for variations the same embedder: glove- 6b-100d and glove 66-300d, which have differ- ent dimensionality but otherwise trained with the same method and corpus (and to a lesser extent which retains casing and is trained on a different corpus); fast and fasttext-wike which again are trained with the same method, but different corpora.
glove-340b-333
The use of subword information, unique to Fasttext-cr -ext-cc-sbbbbbb and fasttext -usb, has large effect on N2O; there is high (0.52) N20 value for these two and much lower N20 with other embedders, including their analogues without subword information. This effect is also illustrated by measuring, for given embedder, the average token overlap between the query and its neighbors (see Fig. 5 in Appendix B). As we would expect, subword methods find near neighbors with lower token overlap, because they embed afffac--simiarr strings near to each other.
ff-idf. Unsurprisingly, tf-idf has low N2O with other embedders (even those based on static word embeddings). Like the subword case, we can also use token overlap to understand why this is the case: its nearest neighbors have by far the largest token overlap with the query (0.43).
Averages of ELMo embeddings. We test three ELMo pretrained models across different capac- ities (elmo small, elmo-origh but the same training data, and across different training data but the same model capacity (elmo-on orig, e elmoooriiae 5. 5b). These two embedder pairs have high N2O (0.42 and 0.55 respectively): the mismatched pair, with both different training data and capacities, has slightly lower N20 (0.38).
Becauss many news articles show up multiple times in the corpus, 23% of sentences in the English Giga- word are exact duplicates of one another; we remove these duplicates. https:: thub. htt.:ub...oooogggggoooegg
5

---

Under review as a conference paper at ICLR 2020
tfidf
0.6
w2v
glove-6b-100d
glove-6b-300d
glove-840b-3006
0.5
fasttext-cc
fasttext-wiki
fasttext-cc-sub
0..
fasttext-wiki-sub
elmo-small
elmo-orig
elmo-orig-5.5b
0.3
bert-base-cls
bert-base-avg
bert-ft-cls
bert-ft-avg
0.2
gpt-last
gpt-avg
infersent
0.1
use-dan
use-trf
tfitt
w2v
glove-6b-1000
glove-bb- 60- 300d
glove-840b. 300d
fasttext-cc
fasttext-wike
fasttext-cc-suk
fasttext-wiki-sukk
elmo-small
elmo-orig
elmo-orig-5 5.
bert-base-cls
bert-base-ave
bert-ft-cil
bert-ft-avo
gpt-last
gpt-avg
infersent
use-dan
use-trf
Figure 3: Heatmap N2O for every pair of sentence embedders Table for k = 50, averaged across five samples of 100 queries; darker colors indicate higher overlap. A larger version of this plot (annotated with N2O values) is in Appendix
BERT and GPT. We first find that specific-token representations for BERT or GPT bbert-base-lss, gpt-last) are outliers compared other embedders (i.e., low N2O; see Fig. 4.. This itself is not unexpected, els (without finetuning) are not geared towards semantic similarity the way other embedders are. What is surprising is that this effect seems to hold even for the MultiNL.I-inetuned version of BERT (bert-ft-cls) if anything, this decreases N2O with other embedders further Notably, taking averaged BERT and GPT embeddings yields higher N2O with other embedders, especially ELMo- based ones. Fig. 6 (Appendix B) plots the N2O values for each embedder compared all others.
the training objectives for both of the pretrained mod-
Encoder-based embedders. We find that InferSent has highest N2O (~0.2-0.3) with the embed- dings based averaging, despite InferSent being trained using NLI task; that said, this is not wholly surprising as the model was initialized using GloVe vectors (glove 8 Cb-300d) during training. The USE variants (DAN and Transformer) have fairly distinct nearest neighbors compared to other methods, with highest N2O between each other (0.24).
ROBUSTNESS AND RUNTIME CONSIDERATIONS
Varying k. One possible concern how sensitive our procedure is (the number of nearest neighbors from which overlap is computed): we would not want conflicting judgments of how similar two sentence embedders are due to different k. To confirm this, we first compute the ranked lists of N2O output for each & E {5, 10...., .,, 45, 50}, where each list consists of all embedder pairs ordered by N2O for that k. We then compute Spearman's 's rank correlation coefficient (p) between each pair of ranked lists, where I indicates perfect positive correlation. We find that the average
II preliminary experiments, also saw similar results with BERT finetuned on the Microsoft Research Paraphrase Corpus (Dolan al. 2004); that the effect does specific to MultiNL.

---

Under review as a conference paper at ICLR 2020
glove 6b-300d
0.61
elmo-orig cls
0.42
glove Nlove-440 1ee-8400-33 ---84b--3000 300d
0.46
0.50
0.100 0.12
bert-base-de
0.38
bert-base port-base-avy avg
0.24 0.31 0.17
fasttext-ce
fasttext-
wiki
0.35 0.42
0.45
0.60
bert-ft-cl
-.06 0.07 0.07 0.07
fasttext- setteett-c-sub CC-bub
0.25 0.27 0.31
0.34
0.31
gpt-last
0.12 0.14 0.10 0.14 0.06
0.27
0.30
0.34 0.37 0.37 0.52
gpt-avg
0.20 0.22 0.09 0.19 0.06 0.16
fasttext-wiki-ae
dove--b--10d -0od
glove- 1ee--bb-300 6b-300d
glove-840b-38 840b-300d 300d
fasttext-cc
fasttext-cc-sub
elmo-small
elmo-orig
bert-base-dls
bert bert-base Per-base-arn hort-base-avg
bert-ft-cl
gpt-last
fasttext-wii
Figure 4: N2O values for subset of embedders (L: static; R: contextual), k = 50.
Spearman's p is very high (0.996; min. 0.986) - i.e., the rankings of embedder similarity by N2O reasonably stable across different values of k. far k and k 50.
Query sampling. We also examine how the results may vary across different query samples; as noted previously, the presented results are averaged across five samples n 100 queries each. Standard deviations for N2O values across the five samples range from 0.005 to 0.019 (avg. 0.01 I). That is, given the range of N2O values being compared, the differences due to different query sam- ples is small. We compute Spearman's 's across different N2O samples in the same manner above (k: = 50) and find an average of 0.994 (min. 0.991).
Runtime. theoretical concern with N2O is that, naively, its computation linear in the size of the corpus, and to have reasonable semantic overlap within a diverse set of sentences, the corpus should be large. While our implementation of exact nearest neighbor search is sufficiently fast in practice we provide comments use of approximate nearest neighbor methods in Appendix
POPULARITY OF NEIGHBORS
In the previous section, we performed basic comparison between sentence embedders using N2O. Here, we show one kind of analysis enabled by N2O: given a query, which sentences from the cor- pus C' are consistently its neighbors across different embedders? We might expect, for example, that a nearly identical paraphrase of the query will be "popular" neighbor chosen by most em- bedders. Table shows an example query with a sentence that is in the 5-nearest neighborhood for all sentence embedders, as well as sentences that are highly ranked for some embedder but not in the nearest neighbor sets for any other embedder (for larger k = 50). Qualitatively, what we find with this example's outlier sentences is that they are often thematically similar in some way (such as fiscal matters Table 2). but with different participants. We also observe that extremely "popular" neighbors tend have high lexical overlap with the query.
QUERY PARAPHRASING
Attempts derive sentence embeddings that capture semantic similarity are inspired by the phe- nomenon of paraphrase; in this section, we use nearest neighbors to probe how sentence embedders capture paraphrase. More specifically, we carry out a "needle-in-a-baystae" Semantic Textual Similarity Benchmark (STS; Cer et al., 2017). STS contains sentence pairs with human judgments of semantic similarity on a 1-5 continuous scale (least to most similar).
experiment using the
GGiven precomputed sentence embeddings, exact nearest neighbor search across the corpus takes 30s. Imin. (depending dimensionality) for a batch of n 100 queries and à 50, across two I 12-core Intel Xeon CPUs (E5-296002.666)) 60GHz).

---

Under review as a conference paper at ICLR 2020
Query:
Britain's biggest mort lender says that average house prices fell 3.6 percent in September, but analysts believe the market isn't that weak.
Embedder
Rank
Sentence
all embedders
5
Average house prices in Britain fell 3.6 percent in September from a month earlier, the country's ' biggest mortgage lender said Thursday, al- though analysts believe the market isn't 't that weak.
bert-base-cls
6
Some analysts say that the December data indicate that consumer spend- ing remains weak, making it harder for the economy to keep sustained rehound.
bert-it-avg
An industry group says German machinery orders were down percent on the year in January but foreign demand is improving.
gpt-last
8 The economy has since rebound and grew 8.9 percent year-on-year in the second quarter, the central bank said last month, with growth expected exceed percent the full year.
Table 2: Popular and outlier near neighbors for the given query (top). The first sentence in the 5-nearest neighborhood for all embedders; the remaining sentences are highly-ranked by the given embedder and outside the 50-nearest neighborhood for all other embedders. See Table (Appendix B) for additional examples.
We take 75 pairs in the 4-5 range from the STS development and test sets where the sentence pair has word-level overlap ratio < 0.6 1.6-iss i.e., near paraphrases with moderately different surface semantics. We also constrain the sentence pairs to come from the newstext-based parts of the dataset. The first sentence in each sentence pair is the "query," and the second sentence is (temporarily) added to Gigaword corpus. An example sentence pair, scored as 4.6, is: (A) Arkansas Supreme Court strikes down execution law and (B) Arkansas justices strike down death penalty. We then compute the rank of the sentence added to the corpus (i.e., the value of such that the added sentence is part of the query's nearest neighbors). An embedder that "perfectly" correlates semantic similarity and distance should yield a rank of for the sentence added to the corpus, since that sentence would be nearest the query.
Results. Using mean reciprocal rank (MRR), we find that the larger ELMo models and Infersent do particularly well at placing paraphrase pairs near each other. We also find that averaged BERT and GPT embeddings consistently perform better than the [CLS] ] /final token one this is con- sistent with our earlier observation (55) that their training objectives may not yield specifii-token embeddings that directly encode semantic similarity, hence why they outliers by N2O. The full table of results is in Table 4 (Appendix
RELATED WORK
Recent comparisons of sentence embedders have been primarily either (1) linguistic probing tasks or (2) downstream evaluations. Linguistic probing tasks test whether embeddings can distinguish surface level properties, like sentence length; syntactic properties, like tree depth; and semantic properties, like coordination inversion. See Ettinger et al. (2016), Adi et al. (2017), Conneau et al. (2018), and Zhu et al. (2018), among others. Downstream evaluations are often classification tasks for which good sentence representations are helpful (e.g.. NLI). Evaluations like the RepEval 2017 shared task (Nangia et al. 2017), SentEval toolkit (Conneau & Kiela, 2018), and GLUE benchmark (Wang et al. [2019) seek standardize comparisons across sentence embedding methods. N2O is complementary to these, providing a task-agnostic way to compare embedders' functionality.
10 CONCLUSION
In this paper, we introduce nearest neighbor overlap (N20), a comparative approach to quantifying similarity between sentence embedders. Using N2O, we draw comparisons across 21 embedders. We also provide additional analyses made possible with N2O, from which we find high variation in embedders' treatment of semantic similarity.
The BERT results with STS consistent with concurrent work by Riemers & Gurevych 2019
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis
of sentence embeddings using auxiliary prediction tasks. In Proc. of ICLR, 2017.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A simple but tough-to-beal bascline for sentence embeddings. Proc. of ICLR, 2017.
Martin Aumüller, Erik Bernhardsson, and Alexander Faithfull. ANN-benchmarks: A benchmarking tool for approximate nearest neighbor algorithms. Information Systems, in press, 2019.
Chandra Bhagavatula, Sergey Feldman, Russell Power, and Waleed Ammar. Content-based citation recommendation. Proc. of NAACL-HLT, 2018.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. In Proc. of EMNLP, 2015.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Jopez-Gazpio, and Lucia Specia. SemEval-2017 Task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proc. of SemEiva, 2017.
Daniel Cer, Yinfei Yang, Sheng-yi yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Con- stant, Mario Guajardo-Céspedess, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Universal sentence encoder. arXiv:1803.11 111. [cs.CL], 2018
Alexis Conneau and Douwe Kicla. SentEval: An evaluation toolkit for universal sentence represen- tations. In Proc. of LREC, 2018
Alexis Conneau, Douwe Kicla, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In Proc. of ACL, 2017.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. Proc. of ACL, 2018.
Douglass R. Cutting, David R. Karger, Jan O. Pedersen, and John W. Tukey. Scatter/Gather: A cluster-based approach to browsing large document collections. In Proc. of SIGIR, 1992.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL-HLT, 2019.
Bill Dolan, Chris Quirk, and Chris Brockett. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proc. of COLING, 2004.
Allyson Ettinger, Ahmed Elgohary. and Philip Resnik. Probing for semantic evidence of composi- tion by means of simple classification tasks. In Proc. of RepEval, 2016.
Masajiro Iwasaki and Daisuke Miyazaki. Optimization of indexing based on k-nearest neighbor graph for proximity search in high-ding nigh-dimensional data. arXiv: 181.0138 [Bs. 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffery Dean. Distributed representa- tions of words and phrases and their compositionality. In Proc. of NeurlPS, 2013.
Nikita Nangia, Adina Williams, Angeliki Lazaridou, and Samuel R. Bowman. The RepEval 2017 shared task: Multi-genre natural language inference with sentence representations. Proc. of RepEval, 2017.
Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English Gigaword Fifth Edition, LDC201 IT07, 2011. Linguistic Data Consortium.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. GloVe: Global vectors for word representation. In Proc. EMNLP, 2014.

---

Under review as a conference paper at ICLR 2020
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL-HLT, 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and 1lya Sutskever. Improving language
understanding by generative pre-training. 2018. URL. https://oppeaa com/bl
Barquage-unsuperee
Niels Riemers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. In Proc. of EMNLP, 2019.
Gerard Salton and Christopher Buckley. Term-weighting approaches automatic text retrieval. Information Processing & Management, 24(5):513-523, 1988.
Mike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. In Proc. of ICASSP, 2012.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proc. of ACL, 2016.
Minjoon Seo, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi. Phrase- indexed question answering: A new challenge for scalable document comprehension. In Proc. of EMNLP, 2018.
Kohei Sugawara, Hayato Kobayashi, and Masajiro Iwasaki. On approximately searching for similar word embeddings. In Proc. of ACL, 2016.
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proc. of ICLR, 2019.
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. Towards universal paraphrastic sentence embeddings. In Proc. of ICLR, 2016.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. Proc. of NAACL-HLT, 2018.
Xunjie Zhu, Tingfeng Li, and Gerard de Melo. Exploring semantic properties of sentence embed- dings. In Proc. of ACI, 2018
A APPENDIX: IMPLEMENTATION DETAILS
In this section, include additional implementation details for experiments performed in the paper. Generally, we use parameters consistent with the original work when possible.
Sentence segmentation. We use the spacy library (2.0.16) to perform sentence segmentation; for word tokenization, we defer to preferences for the original embedder implementations if speci- fied (see below), or use the spacy tokenizer otherwise.
TY-idf. We use the gensim library (3.7.3) implementation of tf-idf [1 with frequency statistics learned the 2010 section of the Gigaword corpus (i.e., the same corpus used find nearest neighbors). For tokenization, we use the Gensim tokenizer and lowercase all word tokens.
Word2vec. We use pretrained 300D Google News embeddings available from Google 12 We use spacy to perform word tokenization and embedding lookup.
"nttp://spccy.io
https://code..gogll www.aggle..cmmaarccee/////////oa
10

---

Under review as a conference paper at ICLR 2020
We use three sets of standard pretrained GloVe embeddings: 100D and 300D embeddings trained on Wikipedia and Gigaword (6B tokens), and 300D embeddings trained on Common Crawl (840B tokens) M We handle tokenization and embedding lookup identically to word2vec; for the WikipediaaGigawword embeddings, which are uncased, we lower case all tokens as well.
Glo Ve.
FastText. We use four sets of pretrained FastText embeddings: two trained on Wikipedia and other news corpora, and two trained on Common Crawl (each with an original version and one trained on subword information) [4 We use the Python port of the FastText implementation to handle tokenization, embedding lookup, and OOV embedding computation [3
ELMo. We use three pretrained models made available by AlIenNLP: small, original, and original (5.5B) T6 We use spacy perform word tokenization, consistent with the all allennip library; also use allennn allennl 1p (0.7.2) to compute the ELMo embeddings. We average the embeddings over all three bidirectional LSTM layers.
BERT. We use Hugging Face's s pytorch-trans formers (0.6.2) implementation and pre- trained BERT base cased model 17 To tokenize, we use the provided BertTokenizer, which handles WordPiece (subword) tokenization, and in general follow the library's recommendations for feature extraction.
For finetuning BERT on MultiNLI (matched subset), we generally use the default parameters pro- vided in the library's coo.classifiie (batch size 32, learning rate Se-5, etc.). We finetune for three epochs, and obtain 84.1% dev accuracy (reasonably consistent with the original work).
GPT. We the same Hugging Face library and associated pretrained model for GPT; their BPE tokenizer and spacy for subword and word tokenization respectively.
InferSent. We use the authors' implementation of InferSent, as well as their pretrained V1 model based on Glo 18 (Unfortunately, the FastText-based V2 model was not available while perform- ing the experiments in this paper; see issues #108 and #124 in the linked Github.) per their README, use the nltk tokenizer (3.2.5).
Universal Sentence Encoder. We use pretrained models available on TensorFlow Hub for both the DAN and Transformer variants 19
The modules handle text preprocessing their own.
COMPUTATIONAL DETAILS
Experiments for ELMo, BERT, GPT, and the Transformer version of USE were run on NVIDIA Titan XP GPU with CUDA 9.2, All other experiments were performed on CPUs.
B APPENDIX: ADDITIONAL RESULTS & PLOTS
ANNOTATED N2O HEATMAP
The heatmap the next page is larger version of Fig. that includes the N2O values (A: 50, 100, averaged over five runs).
Transformer:

---

0.6
0.5
00.4
0.3
0.2
0.24
0.16
0.15
Ame-uu
0.16
0.14
0.14
0.16
0.11
0.11
0.12
0.08
0.10
0.10
0.08
0.08
0.10
0.06
0.06
0.06
0.05
0.18 0.05
Dert---an
0.07
0.16
0.14
0.19
0.19
0.16
0.17
0.07
0.08
0.10
0.09
0.09
0.07
0.21 0.09
0.12
0.32
0.07
0.13
0.14
0.23
0.22
0.19
-
0.56
0.12
0.31
0.07
0.13
0.14
0.22
0.23
0.19
0.15 0.21
dmmoooig
0.42
0.39
0.10
0.24
0.06
0.11
0.12
0.20
0.19
0.15
0.19
0.20
0.21
0.08
0.17
0.05
0.10
0.10
0.16
0.22
0.12
0.11
0.52
0.17
0.18
0.19
0.07
0.16
0.05
0.09
0.09
0.14
0.20
0.11
0.10
0.31
0.37
0.23
0.27
0.28
0.08
0.23
0.06
0.11
0.13
0.20
0.27
0.17
0.15 0.17 0.16
0.60
0.34
0.37
0.24
0.29
0.30
0.08
0.24
0.06
0.12
0.13
0.21
0.28
0.19
0.53
0.45
0.31
0.34
0.24
0.26
0.27 0.27
0.08
0.22
0.06
0.11
0.12
0.18
0.27
0.16
0.50
0.46
0.42
0.27
0.30
0.24
0.27
0.08
0.22
0.06
0.11
0.12
0.19
0.24
0.16
0.15
00000000000000000000000000000000
0.61
0.46
0.38
0.35
0.25
0.27
0.22
0.23
0.24
0.08
0.20
0.05
0.10
0.11
0.16
0.21
0.14
0.13
0.32
0.41
0.40
0.49
0.43
0.24
0.28
0.25
0.31
0.32
0.09
0.25
0.06
0.12
0.14
0.21
0.26
0.21
0.19
W2v
0.23
0.13
0.17
0.15
0.20
0.18
0.11
0.12
0.12
0.16
0.16
0.06
0.14
0.05
0.08
0.12
0.14
0.14
0.14
0.13
Rldd
tfidf
w2v
glove-6b-100 100d
(love-6b-300g
glove-840b-30GG 3000
fasttext-cc
fasttext-wiki
fasttext-cc-sub
fasttext-wikki-sub
elmo-small
elmo-orig
elmo-orig-5555 5b
bert-base-cls
bert-base-avg
bert-ft-cls
bert-ft-avg
gpt-last
gpl-avg
infersent
use-dan
use-trf
12

---

Under review as a conference paper at ICLR 2020
B.2
TOKEN OVERLAP
In | we noted that the FastText subword variants had much lower N2O compared to other em- bedders (including analogues without subword information). Fig. shows average token overlap between query and its nearest neighbors, averaged over all queries. Unsurprisingly, ffidf has by far the highest overlap, and fasttext-wi wiiki i-sub and Fasttext-cc--0 the lowest.
avg. token overlap
0.4
0.3
tfidf
glove-6b-1000 -bo-
w2v
glove plove-6b-33 6b- 30-300d
glove glove-840b-2 840b-8400-22 840-3300 300d
fasttext-cc
fasttext-cc-sult
fasttext-wikl-suk
fasttext-wit
Figure 5: Average token overlap between a query and nearest neighbors (k 50), averaged over all queries. Error bars represent 95% confidence intervals.
B.3 N2O RANGES PER EMBEDDER
In S we found that the BERT and GPT based embedders had low N2O with all other embedders, and averaging (rather than taking the [CLS] or last embedding) generally raised N2O, Fig. shows boxplots of N2O values between each embedder and all other embedders.
0.6
N20 w/other embedders
0.5
0.4
0.3
0.2
0.1
tridf
glove-6b-1pe -ove-6bb-00d 100d
w2v
glove-6r 300d
glove-844 Gove-8840-233 300d
fasttexi-cc
fasttext-wild wikl
fasttext-cc-sub
fasttext-wiki--se
elmo-small
elmo-orig
elmo-on -olo-orgg--- 5.5) 5b
bert-base bert-base-cls
bert-base-awe
bert-ft-cls
bert-ft-avg
gpt-last
gpt-avg
Infersent
use- dan use-tri
Figure 6: Comparison of N2O distribution between each embedder and all others.
13

---

Under review as a conference paper at ICLR 2020
B.4 POPULAR & OUTLIER NEAREST NEIGHBORS
Table 3 shows additional outlier nearest neighbors from Table 2
Britain's biggest mortgage lender says that average house prices fall 3.6 percent in September, but analysts believe the market isn't 't that weak.
Query:
Embedder all embedders
Rank
Sentence
Average house prices in Britain fell 3.6 percent in September from month earlier, the country's biggest morigage lender said Thursday, al- though analysts believe market isn't that weak.
bert-ft-cls
2
Japanese consumer prices fell for 13th straight month in March, though the GDP data suggests that deflationary pressures are starting to case.
fasttext-co-sun
6 It cautioned however that the economic situation abroad could still slow Sweden's recovery. and said the country's 's gross domestic product (GDP) would grow just percent percent growth.
I, down from its May estimate of
glove-840b-300d
12 Mcanwhile, Australia's central bank left key interest unchanged 3.75 percent on Tuesday, surprising investors and analysts who had pre- dicted the bank would continue raising the rate as the nation's economy rebounds.
Table 3: Additional outlier near neighbors for the given query (top; same Table 2 sentence is in the 5-nearest neighborhood for all embedders; the remaining sentences are highly- ranked by the given embedder and outside the 50-nearest neighborhood for all other embedders.
The first
B.5 QUERY-PARAPIIRAEE EXPERIMENT RESULTS
Table 4 shows results for the query-parre paraphaaee experiment in 1 mean reciprocal rank (MRR), the number of queries for which its paraphrase was its nearest neighbor, and the number of queries for which the paraphrase was in its 5-nearest neighborhood.
Embedder
MRR
top
top-5
olmo-orlg-5.55
0.910
70
elmo-orig
0,829
60
65
infersent-vl
0.799
64
w2v
0.760
52
64
use-trf
0.759
60
fasttext-cc
0.756
52
62
use-dan
0.718
55
bert-base-avg
0.674
47
55
glove-6b-300d
0.673
48
52
tfidf
0.672
45
55
fasttext-wikt
0.662
45
54
elmo-small
0.638
44
51
şlove-840b-3004
0.601
42
49
gpt-avg
0.600
50
fasttext-wiki-eut
0.552
37
47
glove-6b-100d
0.529
43
fasttext-cc-subb
0.515
35
41
bert-ft-avg
0.493
44
bert-base-cls
0.450
27
42
spt-last
0.365
24
30
bert-ft-ols
0.302
19
27
Table 4: Results for the guery-paraphrase experiment ($8) top-5 are the number of queries for which the paraphrase was the nearest neighbor and in the 5-nearest neighborhood (max. 75), respectively.
sorted by decreasing MRR. # top and
14

---

Under review as a conference paper at ICLR 2020
C APPENDIX: APPROXIMATE NEAREST NEIGHBORS
As noted in 6 N2O computation is linear in the size of the corpus, and to have reasonable semantic overlap within diverse of sentences, the corpus should be large. The upfront cost of computing sentence embeddings across the corpus is unavoidable (and, for many applications, necessary any- ways); our implementation of exact search is fast enough that repeated queries given precomputed embeddings is not concern (see footnote 8).
However, we note that approximate nearest neighbor (ANN) methods are also a viable option, where computation of building an index of the corpus is front-loaded to ensure sub-linear search time. We recommend use of a small held-out set of queries tune the ANN method parameters towards higher precision/recall (vs. speed).
All of the results in this paper were obtained using exact (linear) search. However, we also performed preliminary experiments using the NGT (neighborhood graph tree) library, which achieves good recall in high-dimensional settings ([wasaki Miyazaki 2018 Aumüller et al. 2019 We were
able to obtain similar N2O-ranked results (query recall ~.96 relatively quickly: 0.25 (depending on embedding dimension).
s./query
We note that, in related work, ANN is commonly used in retrieval settings; e.g., Sugawara et al. (0166) test multiple ANN methods for similar word embedding search, and Bhagavatull et al. al.1) ANN method to index documents for citation recommendation. We believe that approximate methods can be of use for scalable N2O computation as well.
15


Reviews:
[{"review_id": "r1xgsDq6KB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper provides a method/metric for comparing sentence embedders, based on a nearest neighbor analysis. The method is straightforward: sample a sentence, embed the sentence and lots of sentences from a corpus, find the k nearest neighbors in the corpus to the sampled sentence, do the same for another embedder, calculate the overlap of the two sets of nearest neighbors.\n\nThe paper is commendably clear and easy to read.\n\nThe main problems I have with the paper are twofold: 1) it's not clear this is enough of a contribution for a top-tier ML conference; and 2) it's not clear what I do with the results.\n\nI think the idea of analysing embedders in this way is potentially interesting, but it feels like the paper needs more. This analysis could be a great section in another paper (e.g. one which proposes another embedding method); or perhaps it could be extended in some way, so that the current content only takes up 1/2 the space, and then there's 1/2 the paper showing how useful this analysis is, for e.g. building better embedders for particular tasks.\n\nThe abstract starts by talking about how embedders have been evaluated using various benchmarks, and hints at the idea that this new comparative approach could be an alternative. But the new method can't really be used for evaluation: I don't come away from the paper knowing whether embedding method A is better than method B, only that A is more like B than C.\n\nI think the problem with the paper as it stands is neatly summed up in the conclusion of the paper, which isn't a conclusion at all: it's just a mini-abstract. I'd like to know what readers should take away from the results, so that they can potentially build better embedders.\n\nI've given the paper a 1. rating only because I really don't think it's ready for a full ICLR paper, not because I think the method is uninteresting or useless. On the contrary, with some more work and thought about how the analysis could be used, this could be a potentially useful tool.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SyeEdLChFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes N2O, a tool for probing the similarity among sentence embedders. Given two sentence embedders, N2O measures the amount of overlap of the k-nearest neighbor sets reported by the two embedders, averaged over a sample of probing queries. Cosine similarity is used as the similarity metric. The paper computes all-pair N2O scores for common sentence embedders and analyzes the results.\n\nOverall, while the idea of probing similarity between embedders is interesting, the paper has the following weaknesses:\n\n- The use of neighbor overlap to compare embedders has precedence in the context of word embeddings [https://www.aclweb.org/anthology/C16-1262/ | https://hal.archives-ouvertes.fr/hal-01806468/ | https://www.aclweb.org/anthology/Q18-1008/]. The methods in these works are mostly identical to N2O with some minor variations (e.g., using Jaccard distance).\n\n- Using an existing method is justified if it provides new insights for the new setting. However, the results do not offer a lot of new insights. The fact that static embeddings, ELMo, and BERT give different neighbors is unsurprising. (The paper even admitted this.) Moreover, since sentence embedders are usually used as features when fine-tuned on downstream tasks, the importance of observations based on the pre-fine-tuned models is unclear. Contrast this with the work on word embedding similarity. Since some research areas (e.g., social science) directly use the similarity of word embeddings to conclude findings, the insights of how different word embeddings behave are more directly applicable.\n\n- The paper does address some design choices, such as the number of neighbors and the number of probing queries, showing that different choices have little effects on the scores. However, the use of cosine similarity is problematic. As noted in the paper, some embedders such as ELMo were not trained on similarity objectives. BERT does have the next-sentence task, but the sparse attention-based architecture does not necessarily push similar sentences to have low cosine similarity.\n\nAdditional comments and questions:\n\n- Instead of using the amount of overlap for a fixed k, it would be nice to have a metric that captures the whole distribution. For instance, maybe two embedders disagree on the first 20 neighbors, but end up retrieving the same set when considering 50 neighbors. The current overlap-based metric cannot capture such a phenomenon.\n\n- Page 2: The method is technically not task-agnostic --- the task is sentence similarity with respect to a specific corpus.\n\n- The paper tests variants of the same embedders by training on different corpora, with the conclusion that mismatched corpora give lower N2O scores. What is the N2O between two embedders of the same type trained on the same corpora but with different seeds?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eHKnBw5r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method to estimate the similarity of sentence embedders called N2O with the goal to better inform embedder choice in downstream applications. For two embedders A and B, N2O samples sentences called queries from a corpus, uses A and B to compute embeddings for each sentence, determines the k nearest neighbors (= other sentences from the corpus) for each sentence, and computes the overlap of the resulting sets of neighbors. Nearest neighbors are computed with Cosine similarity.\n\nThe paper should be rejected, mainly because the proposed method is not appropriate to inform embedder choices as claimed by the paper.\n\nI doubt that the results from this paper can be used to make better embedder choices. The paper measures which embedders are similar in a specific way, but this does not tell us much about the performance in downstream tasks. Only based on the N2O measure, no informed decision can be made which embedder will perform well on a given task. Hence, the approach is not well motivated. Evaluation metrics such as the GLUE benchmark or SentEval provide much more information what we can expect from specific embedders. Hence, I think the results in the paper are only interesting to very few readers. Furthermore, the paper uses Cosine similarity to compute the similarity between embedders. However, embeddings are often not used directly, but used as input for a model (e.g. a neural network), which learns to predict something based on the embeddings. Hence, two embedders which are similar according to N2O may perform differently in downstream tasks.\n\nTo improve the paper, I recommend to test whether the N2O similarity correlates well with downstream performance. Concretely, the paper should clearly answer the question whether we can predict the downstream performance of an embedder A given its N2O similarity to an embedder B and the downstream performance of B. Furthermore, I recommend to also use other metrics to compute the distances between embeddings, and not only Cosine similarity. Th robustness and relevance of the findings is much higher if the findings are consistent across many different metrics. Instead of computing the overlap of the neighbors for different values of k, the paper could also use ranking measures to estimate the similarity of embedders. Furthermore, I think this paper fits much better to a conference focused on natural language processing. \n\nQuestions:\n1. The paper claims that N2O can be used to inform embedder choices. Based on the main results in Figure 3, how can this help to decide which embedder to use for my task at hand?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to analyze the space of known sentence-to-vector functions by comparing the ways in which they induce nearest neighbor lists in a text corpus.

The primary results of the study are somewhat unclear, and the reviewers do not find the method to be novel enough—or sufficiently well motivated a priori—to warrant publication in spite of these results.
Decision: Reject