Situating Sentence Embedders with Nearest Neighbor Overlap

1 INTRODUCTION . Continuous embeddings—of words and of larger linguistic units—are now ubiquitious in NLP . The success of self-supervised pretraining methods that deliver embeddings from raw corpora has led to a proliferation of embedding methods , with an eye toward “ universality ” across NLP tasks . Our focus here is on sentence embedders , and specifically their evaluation . As with most NLP components , intrinsic ( e.g. , Conneau et al. , 2018 ) and extrinsic ( e.g. , GLUE ; Wang et al. , 2019 ) evaluations have emerged for sentence embedders . Our approach , nearest neighbor overlap ( N2O ) , is different : it compares a pair of embedders in a linguistics- and task-agnostic manner , using only a large unannotated corpus . The central idea is that two embedders are more similar if , for a fixed query sentence , they tend to find nearest neighbor sets that overlap to a large degree . By drawing a random sample of queries from the corpus itself , N2O can be computed on in-domain data without additional annotation , and therefore can help inform embedder choices in applications such as text clustering ( Cutting et al. , 1992 ) , information retrieval ( Salton & Buckley , 1988 ) , and open-domain question answering ( Seo et al. , 2018 ) , among others . After motivating and explaining the N2O method ( §2 ) , we apply it to 21 sentence embedders ( §3-4 ) . Our findings ( §5 ) reveal relatively high functional similarity among averaged static ( noncontextual ) word type embeddings , a strong effect of the use of subword information , and that BERT and GPT are distant outliers . In §6 , we demonstrate the robustness of N2O across different query samples and probe sizes . We also illustrate additional analyses made possible by N2O : identifying embeddingspace neighbors of a query sentence that are stable across embedders , and those that are not ( §7 ) ; and probing the abilities of embedders to find a known paraphrase ( §8 ) . The latter reveals considerable variance across embedders ’ ability to identify semantically similar sentences from a broader corpus . 2 CORPUS-BASED EMBEDDING COMPARISON . We first motivate and introduce our nearest neighbor overlap ( N2O ) procedure for comparing embedders ( maps from objects to vectors ) . Although we experiment with sentence embedders in this paper , we note that this comparison procedure can be applied to other types of embedders ( e.g. , phrase-level or document-level ) .1 1We also note that nearest neighbor search has been frequently used on word embeddings ( e.g. , word analogy tasks ) . function N2O ( eA , eB , C , k ) for each query qj ∈ { qi } ni=1 do neighborsA ← nearest ( eA , qj , C , k ) neighborsB ← nearest ( eB , qj , C , k ) o [ j ] ← |neighborsA ∩ neighborsB | end for return ∑ j o [ j ] / ( k × n ) end function Figure 2 : Computation of N2O for two embedders , eA and eB , using a corpus C ; the number of nearest neighbors is given by k. n is the number of queries ( q1 . . .qn ) , which are sampled uniformly from the corpus without replacement . The output is in [ 0 , 1 ] , where 0 indicates no overlap between nearest neighbors for all queries , and 1 indicates perfect overlap . 2.1 DESIDERATA . We would like to quantify the extent to which sentence embedders vary in their treatment of “ similarity. ” For example , given the sentence Mary gave the book to John , embedders based on bag-of-words will treat John gave the book to Mary as being maximally similar to the first sentence , whereas other embedders may treat Mary gave the dictionary to John as more similar ; our comparison should reflect this intuition . We would also like to focus on using naturally-occuring text for our comparison . Although there is merit in expert-constructed examples ( see linguistic probing tasks referenced in §9 ) , we have little understanding of how these models will generalize to text from real documents ; many application settings involve computing similarity across texts in a corpus . Finally , we would like our evaluation to be task-agnostic , since we expect embeddings learned from large unannotated corpora in a self-supervised ( and task-agnostic ) manner to continue to play an important role in NLP . As a result , we base our comparison on nearest neighbors : first , because similarity is often assumed to correspond to nearness in embedding space ( e.g. , Figure 1 ) ; second , because nearest neighbor methods are used directly for retrieval and other applications ; and finally , because the nearest neighbors of a sentence can be computed for any embedder on any corpus without additional annotation . 2.2 ALGORITHM . Suppose we want to compare two sentence embedders , eA ( · ) and eB ( · ) , where each embedding method takes as input a natural language sentence s and outputs a d-dimensional vector . For our purposes , we consider variants trained on different data or using different hyperparameters , even with the same parameter estimation procedure , to be different sentence embedders . Take a corpus C , which is likely to have some semantic overlap in its sentences , and segment it into sentences s1 , . . . , s|C| . Randomly select a small subset of the sentences in C as “ queries ” ( q1 , . . . , qn ) . To see how similar eA and eB are , we compute the overlap in nearest neighbor sentences , averaged across multiple queries ; the algorithm is in Figure 2. nearest ( ei , qj , C , k ) returns the k nearest neighbor sentences in corpus C to the query sentence qj , where all sentences are embedded with ei.2 There are different ways to define nearness and distance in embedding spaces ( e.g. , using cosine similarity or Euclidean distance ) ; in this paper we use cosine similarity . We can think about this procedure as randomly probing the sentence vector space ( through the n query sentences ) from the larger space of the embedded corpus , under a sentence embedder ei ; in some sense , k controls the depth of the probe . The N2O procedure then compares the sets of sentences recovered by the probes . 2One of these will be the query sentence itself , since we sampled it from the corpus ; we assume nearest ignores it when computing the k-nearest-neighbor lists . 3 SENTENCE EMBEDDING METHODS . In the previous section , we noted that we consider a “ sentence embedder ” to encompass how it was trained , which data it was trained on , and any other hyperparameters involved in its creation . In this section , we first review the broader methods behind these embedders , turning to implementation decisions in §4 . 3.1 TF-IDF . We consider tf-idf , which has been clasically used in information retrieval settings . The tf-idf of a word token is based off two statistics : term frequency ( how often a term appears in a document ) and inverse document frequency ( how rare the term is across all documents ) . The vector representation of the document is the idf-scaled term frequencies of its words ; in this work we treat each sentence as a “ document ” and the vocabulary-length tf-idf vector as its embedding . 3.2 WORD EMBEDDINGS . Because sentence embeddings are often built from word embeddings ( through initialization when training or other composition functions ) , we briefly review notable word embedding methods . Static embeddings . We define “ static embeddings ” to be fixed representations of every word type in the vocabulary , regardless of its context . We consider three popular methods : word2vec ( Mikolov et al. , 2013 ) embeddings optimized to be predictive of a word given its context ( continuous bag of words ) or vice versa ( skipgram ) ; GloVe ( Pennington et al. , 2014 ) embeddings learned based on global cooccurrence counts ; and FastText ( Conneau et al. , 2017 ) , an extension of word2vec which includes character n-grams ( for computing representations of out-of-vocabulary words ) . Contextual embeddings . Contextual word embeddings , where a word token ’ s representation is dependent on its context , have become popular due to improvements over state-of-the-art on a wide variety of tasks . We consider : • ELMo ( Peters et al. , 2018 ) embeddings are generated from a multi-layer , bidirectional recurrent language model that incorporates character-level information . • GPT ( Radford et al. , 2018 ) embeddings are generated from a unidirectional language model with multi-layer transformer decoder ; subword information is included via byte-pair encoding ( BPE ; Sennrich et al. , 2016 ) . • BERT ( Devlin et al. , 2019 ) embeddings are generated from a transformer-based model trained to predict ( a ) a word given both left and right context , and ( b ) whether a sentence is the “ next sentence ” given a previous sentence . Subword information is incorporated using the WordPiece model ( Schuster & Nakajima , 2012 ) . Composition of word embeddings . The simplest way to obtain a sentence ’ s embedding from its sequence of words is to average the word embeddings.3 Despite the fact that averaging discards word order , it performs surprisingly well on sentence similarity , NLI , and other downstream tasks ( Wieting et al. , 2016 ; Arora et al. , 2017 ) .4 In the case of contextual embeddings , there may be other conventions for obtaining the sentence embedding , such as using the embedding for a special token or position in the sequence . With BERT , the [ CLS ] token representation ( normally used as input for classification ) is also sometimes used as a sentence representation ; similarly , the last token ’ s representation may be used for GPT . 3In the case of GPT and BERT , which yield subword embeddings , we treat those as we would standard word embeddings . 4Arora et al . ( 2017 ) also suggest including a PCA-based projection with word embedding averaging to further improve downstream performance . However , because our focus is on behavior of the embeddings themselves , we do not apply this projection here . 3.3 ENCODERS . A more direct way to obtain sentence embeddings is to learn an encoding function that takes in a sequence of tokens and outputs a single embedding ; often this is trained using a relevant supervised task . We consider two encoder-based methods : • InferSent ( Conneau et al. , 2017 ) : supervised training on the Stanford Natural Language Inference ( SNLI ; Bowman et al. , 2015 ) dataset ; the sentence encoder provides representations for the premise and hypothesis sentences , which are then fed into a clasifier . • Universal Sentence Encoder ( USE ; Cer et al. , 2018 ) : supervised , multi-task training on several semantic tasks ( including semantic textual similarity ) ; sentences are encoded either with a deep averaging network or a transformer . 4 EXPERIMENTAL DETAILS . Our main experiment is a broad comparison , using N2O , of the embedders discussed above and listed in Table 1 . Despite the vast differences in methods , N2O allows us to situate each in terms of its functional similarity to the others . N2O computation . We describe a N2O sample as , for a given random sample of n queries , the computation of N2O ( eA , eB , C , k ) for every pair of sentence embedders as described in §2 , using cosine similarity to determine nearest neighbors . The results in §5 are with k ( the number of sentences retrieved ) set to 50 , averaged across five samples of n = 100 queries . We illustrate the effects of different k and N2O samples in §6 . Corpus . For our corpus , we draw from the English Gigaword ( Parker et al. , 2011 ) , which contains newswire text from seven news sources . For computational feasibility , we use the articles from 2010 , for a total of approximately 8 million unique sentences.5 We note preprocessing details ( segmentation , tokenization ) in Appendix A. Queries . For each N2O sample , we randomly select 100 ledes ( opening sentences ) from the news articles of our corpus , and use the same ones across all embedders . Because the Gigaword corpus contains text from multiple news sources covering events over the same time period , it is likely that the corpus will contain semantically similar sentences for a given lede . The average query length is 30.7 tokens ( s.d . 10.2 ) ; an example query is : “ Sandra Kiriasis and brakewoman Stephanie Schneider of Germany have won the World Cup bobsled race at Lake Placid. ” Sentence embedders . Table 1 details the sentence embedders we use in our experiments . In general , we use popular pretrained versions of the methods described in §3 . We also select pretrained variations of the same method ( e.g. , FastText embeddings trained from different corpora ) to permit more controlled comparisons . In a couple of cases , we train/finetune models of our own : for tf-idf , we compute frequency statistics using our corpus , with each sentence as its own “ document ” ; for BERT , we use the Hugging Face implementation with default hyperparameters,6 and finetune using the matched subset of MultiNLI ( Williams et al. , 2018 ) for three epochs ( dev . accuracy 84.1 % ) . We note that additional embedders are easily situated among the ones tested in this paper by first computing nearest neighbors of the same query sentences , and then computing overlap with the nearest neighbors obtained in this paper . To enable this , the code , query sentences , and nearest neighbors per embedder and query will be publicly available .

Summary: The paper proposes a method to estimate the similarity of sentence embedders called N2O with the goal to better inform embedder choice in downstream applications. For two embedders A and B, N2O samples sentences called queries from a corpus, uses A and B to compute embeddings for each sentence, determines the k nearest neighbors (= other sentences from the corpus) for each sentence, and computes the overlap of the resulting sets of neighbors. Nearest neighbors are computed with Cosine similarity.