Bridging the Gap: Incorporating a Semantic Similarity Measure for Effectively Mapping PubMed Queries to Documents

Abstract: The main approach of traditional information retrieval (IR) is to examine how many words from a query appear in a document. A drawback of this approach, however, is that it may fail to detect relevant documents where no or only few words from a query are found. The semantic analysis methods such as LSA (latent semantic analysis) and LDA (latent Dirichlet allocation) have been proposed to address the issue, but their performance is not superior compared to common IR approaches. Here we present a query-document similarity measure motivated by the Word Mover's Distance. Unlike other similarity measures, the proposed method relies on neural word embeddings to calculate the distance between words. Our method is efficient and straightforward to implement. The experimental results on TREC and PubMed show that our approach provides significantly better performance than BM25. We also discuss the pros and cons of our approach and show that there is a synergy effect when the word embedding measure is combined with the BM25 function.

In information retrieval (IR), queries and documents are typically represented by term vectors where each term is a content word and weighted by tf-idf, i.e. the product of the term frequency and the inverse document frequency, or other weighting schemes BIBREF0 . The similarity of a query and a document is then determined as a dot product or cosine similarity. Although this works reasonably, the traditional IR scheme often fails to find relevant documents when synonymous or polysemous words are used in a dataset, e.g. a document including only “neoplasm" cannot be found when the word “cancer" is used in a query. One solution of this problem is to use query expansion BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 or dictionaries, but these alternatives still depend on the same philosophy, i.e. queries and documents should share exactly the same words.
While the term vector model computes similarities in a sparse and high-dimensional space, the semantic analysis methods such as latent semantic analysis (LSA) BIBREF5 , BIBREF6 and latent Dirichlet allocation (LDA) BIBREF7 learn dense vector representations in a low-dimensional space. These methods choose a vector embedding for each term and estimate a similarity between terms by taking an inner product of their corresponding embeddings BIBREF8 . Since the similarity is calculated in a latent (semantic) space based on context, the semantic analysis approaches do not require having common words between a query and documents. However, it has been shown that LSA and LDA methods do not produce superior results in various IR tasks BIBREF9 , BIBREF10 , BIBREF11 and the classic ranking method, BM25 BIBREF12 , usually outperforms those methods in document ranking BIBREF13 , BIBREF14 .
Neural word embedding BIBREF15 , BIBREF16 is similar to the semantic analysis methods described above. It learns low-dimensional word vectors from text, but while LSA and LDA utilize co-occurrences of words, neural word embedding learns word vectors to predict context words BIBREF10 . Moreover, training of semantic vectors is derived from neural networks. Both co-occurrence and neural word embedding approaches have been used for lexical semantic tasks such as semantic relatedness (e.g. king and queen), synonym detection (e.g. cancer and carcinoma) and concept categorization (e.g. banana and pineapple belong to fruits) BIBREF10 , BIBREF17 . But, Baroni et al. Baroni2014 showed that neural word embedding approaches generally performed better on such tasks with less effort required for parameter optimization. The neural word embedding models have also gained popularity in recent years due to their high performance in NLP tasks BIBREF18 .
Here we present a query-document similarity measure using a neural word embedding approach. This work is particularly motivated by the Word Mover's Distance BIBREF19 . Unlike the common similarity measure taking query/document centroids of word embeddings, the proposed method evaluates a distance between individual words from a query and a document. Our first experiment was performed on the TREC 2006 and 2007 Genomics benchmark sets BIBREF20 , BIBREF21 , and the experimental results showed that our approach was better than BM25 ranking. This was solely based on matching queries and documents by the semantic measure and no other feature was used for ranking documents.
In general, conventional ranking models (e.g. BM25) rely on a manually designed ranking function and require heuristic optimization for parameters BIBREF22 , BIBREF23 . In the age of information explosion, this one-size-fits-all solution is no longer adequate. For instance, it is well known that links to a web page are an important source of information in web document search BIBREF24 , hence using the link information as well as the relevance between a query and a document is crucial for better ranking. In this regard, learning to rank BIBREF22 has drawn much attention as a scheme to learn how to combine diverse features. Given feature vectors of documents and their relevance levels, a learning to rank approach learns an optimal way of weighting and combining multiple features.
We argue that the single scores (or features) produced by BM25 and our proposed semantic measure complement each other, thus merging these two has a synergistic effect. To confirm this, we measured the impact on document ranking by combining BM25 and semantic scores using the learning to rank approach, LamdaMART BIBREF25 , BIBREF26 . Trained on PubMed user queries and their click-through data, we evaluated the search performance based on the most highly ranked 20 documents. As a result, we found that using our semantic measure further improved the performance of BM25.
Taken together, we make the following important contributions in this work. First, to the best of our knowledge, this work represents the first investigation of query-document similarity for information retrieval using the recently proposed Word Mover's Distance. Second, we modify the original Word Mover's Distance algorithm so that it is computationally less expensive and thus more practical and scalable for real-world search scenarios (e.g. biomedical literature search). Third, we measure the actual impact of neural word embeddings in PubMed by utilizing user queries and relevance information derived from click-through data. Finally, on TREC and PubMed datasets, our proposed method achieves stronger performance than BM25.
A common approach to computing similarity between texts (e.g. phrases, sentences or documents) is to take a centroid of word embeddings, and evaluate an inner product or cosine similarity between centroids BIBREF14 , BIBREF27 . This has found use in classification and clustering because they seek an overall topic of each document. However, taking a simple centroid is not a good approximator for calculating a distance between a query and a document BIBREF19 . This is mostly because queries tend to be short and finding the actual query words in documents is feasible and more accurate than comparing lossy centroids. Consistent with this, our approach here is to measure the distance between individual words, not the average distance between a query and a document.
Our work is based on the Word Mover's Distance between text documents BIBREF19 , which calculates the minimum cumulative distance that words from a document need to travel to match words from a second document. In this subsection, we outline the original Word Mover's Distance algorithm, and our adapted model is described in Section 2.2.
First, following Kusner et al. Kusner2015, documents are represented by normalized bag-of-words (BOW) vectors, i.e. if a word INLINEFORM0 appears INLINEFORM1 times in a document, the weight is DISPLAYFORM0 
where INLINEFORM0 is number of words in the document. The higher the weight, the more important the word. They assume a word embedding so that each word INLINEFORM1 has an associated vector INLINEFORM2 . The dissimilarity INLINEFORM3 between INLINEFORM4 and INLINEFORM5 is then calculated by DISPLAYFORM0 
The Word Mover's Distance makes use of word importance and the relatedness of words as we now describe.
Let INLINEFORM0 and INLINEFORM1 be BOW representations of two documents INLINEFORM2 and INLINEFORM3 . Let INLINEFORM4 be a flow matrix, where INLINEFORM5 denotes how much it costs to travel from INLINEFORM6 in INLINEFORM7 to INLINEFORM8 in INLINEFORM9 , and INLINEFORM10 is the number of unique words appearing in INLINEFORM11 and/or INLINEFORM12 . To entirely transform INLINEFORM13 to INLINEFORM14 , we ensure that the entire outgoing flow from INLINEFORM15 equals INLINEFORM16 and the incoming flow to INLINEFORM17 equals INLINEFORM18 . The Word Mover's Distance between INLINEFORM19 and INLINEFORM20 is then defined as the minimum cumulative cost required to move all words from INLINEFORM21 to INLINEFORM22 or vice versa, i.e. DISPLAYFORM0 
The solution is attained by finding INLINEFORM0 that minimizes the expression in Eq. ( EQREF5 ). Kusner et al. Kusner2015 applied this to obtain nearest neighbors for document classification, i.e. k-NN classification and it produced outstanding performance among other state-of-the-art approaches. What we have just described is the approach given in Kusner et al. We will modify the word weights and the measure of the relatedness of words to better suit our application.
While the prior work gives a hint that the Word Mover's Distance is a reasonable choice for evaluating a similarity between documents, it is uncertain how the same measure could be used for searching documents to satisfy a query. First, it is expensive to compute the Word Mover's Distance. The time complexity of solving the distance problem is INLINEFORM0 BIBREF28 . Second, the semantic space of queries is not the same as those of documents. A query consists of a small number of words in general, hence words in a query tend to be more ambiguous because of the restricted context. On the contrary, a text document is longer and more informational. Having this in mind, we realize that ideally two distinctive components could be employed for query-document search: 1) mapping queries to documents using a word embedding model trained on a document set and 2) mapping documents to queries using a word embedding model obtained from a query set. In this work, however, we aim to address the former, and the mapping of documents to queries remains as future work.
For our purpose, we will change the word weight INLINEFORM0 to incorporate inverse document frequency ( INLINEFORM1 ), i.e. DISPLAYFORM0 
where INLINEFORM0 . INLINEFORM1 is the size of a document set and INLINEFORM2 is the number of documents that include the INLINEFORM3 th term. The rationale behind this is to weight words in such a way that common terms are given less importance. It is the idf factor normally used in tf-idf and BM25 BIBREF29 , BIBREF30 . In addition, our word embedding is a neural word embedding trained on the 25 million PubMed titles and abstracts.
Let INLINEFORM0 and INLINEFORM1 be BOW representations of a query INLINEFORM2 and a document INLINEFORM3 . INLINEFORM4 and INLINEFORM5 in Section 2.1 are now replaced by INLINEFORM6 and INLINEFORM7 , respectively. Since we want to have a higher score for documents relevant to INLINEFORM8 , INLINEFORM9 is redefined as a cosine similarity, i.e. DISPLAYFORM0 
In addition, the problem we try to solve is the flow INLINEFORM0 . Hence, Eq. ( EQREF5 ) is rewritten as follows. DISPLAYFORM0 
where INLINEFORM0 represents the word INLINEFORM1 in INLINEFORM2 . INLINEFORM3 in Eq. ( EQREF7 ) is unknown for queries, therefore we compute INLINEFORM4 based on the document collection. The optimal solution of the expression in Eq. ( EQREF9 ) is to map each word in INLINEFORM5 to the most similar word in INLINEFORM6 based on word embeddings. The time complexity for getting the optimal solution is INLINEFORM7 , where INLINEFORM8 is the number of unique query words and INLINEFORM9 is the number of unique document words. In general, INLINEFORM10 and evaluating the similarity between a query and a document can be implemented in parallel computation. Thus, the document ranking process can be quite efficient.
In our study, we use learning to rank to merge two distinctive features, BM25 scores and our semantic measures. This approach is trained and evaluated on real-world PubMed user queries and their responses based on click-through data BIBREF31 . While it is not common to use only two features for learning to rank, this approach is scalable and versatile. Adding more features subsequently should be straightforward and easy to implement. The performance result we obtain demonstrates the semantic measure is useful to rank documents according to users' interests.
We briefly outline learning to rank approaches BIBREF32 , BIBREF33 in this subsection. For a list of retrieved documents, i.e. for a query INLINEFORM0 and a set of candidate documents, INLINEFORM1 , we are given their relevancy judgements INLINEFORM2 , where INLINEFORM3 is a positive integer when the document INLINEFORM4 is relevant and 0 otherwise. The goal of learning to rank is to build a model INLINEFORM5 that can rank relevant documents near or at the top of the ranked retrieval list. To accomplish this, it is common to learn a function INLINEFORM6 , where INLINEFORM7 is a weight vector applied to the feature vector INLINEFORM8 . A part of learning involves learning the weight vector but the form of INLINEFORM9 may also require learning. For example, INLINEFORM10 may involve learned decision trees as in our application.
In particular, we use LambdaMART BIBREF25 , BIBREF26 for our experiments. LambdaMART is a pairwise learning to rank approach and is being used for PubMed relevance search. While the simplest approach (pointwise learning) is to train the function INLINEFORM0 directly, pairwise approaches seek to train the model to place correct pairs higher than incorrect pairs, i.e. INLINEFORM1 , where the document INLINEFORM2 is relevant and INLINEFORM3 is irrelevant. INLINEFORM4 indicates a margin. LambdaMART is a boosted tree version of LambdaRank BIBREF26 . An ensemble of LambdaMART, LambdaRank and logistic regression models won the Yahoo! learning to rank challenge BIBREF23 .
Our resulting formula from the Word Mover's Distance seeks to find the closest terms for each query word. Figure FIGREF11 depicts an example with and without using our semantic matching. For the query, “negative pressure wound therapy", a traditional way of searching documents is to find those documents which include the words “negative", “pressure", “wound" and “therapy". As shown in the figure, the words, “pressure" and “therapy", cannot be found by perfect string match. On the other hand, within the same context, the semantic measure finds the closest words “NPWT" and “therapies" for “pressure" and “therapy", respectively. Identifying abbreviations and singular/plural would help match the same words, but this example is to give a general idea about the semantic matching process. Also note that using dictionaries such as synonyms and abbreviations requires an additional effort for manual annotation.
In the following subsections, we describe the datasets and experiments, and discuss our results.
To evaluate our word embedding approach, we used two scientific literature datasets: TREC Genomics data and PubMed. Table TABREF13 shows the number of queries and documents in each dataset. TREC represents the benchmark sets created for the TREC 2006 and 2007 Genomics Tracks BIBREF20 , BIBREF21 . The original task is to retrieve passages relevant to topics (i.e. queries) from full-text articles, but the same set can be utilized for searching relevant PubMed documents. We consider a PubMed document relevant to a TREC query if and only if the full-text of the document contains a passage judged relevant to that query by the TREC judges. Our setup is more challenging because we only use PubMed abstracts, not full-text articles, to find evidence.
This is the number of PubMed documents as of Apr. 6, 2017. This number and the actual number of documents used for our experiments may differ slightly.
Machine learning approaches, especially supervised ones such as learning to rank, are promising and popular nowadays. Nonetheless, they usually require a large set of training examples, and such datasets are particularly difficult to find in the biomedical domain. For this reason, we created a gold standard set based on real (anonymized) user queries and the actions users subsequently took, and named this the PubMed set.
To build the PubMed set, we collected one year's worth of search logs and restricted the set of queries to those where users requested the relevance order and which yielded at least 20 retrieved documents. This set contained many popular but duplicate queries. Therefore, we merged queries and summed up user actions for each of them. That is, for each document stored for each query, we counted the number of times it was clicked in the retrieved set (i.e. abstract click) and the number of times users requested full-text articles (i.e. full-text click). We considered the queries that appeared less than 10 times to be less informative because they were usually very specific, and we could not collect enough user actions for training. After this step, we further filtered out non-informational queries (e.g. author and journal names). As the result, 27,870 queries remained for the final set.
The last step for producing the PubMed set was to assign relevance scores to documents for each query. We will do this based on user clicks. It is known that click-through data is a useful proxy for relevance judgments BIBREF34 , BIBREF35 , BIBREF36 . Let INLINEFORM0 be the number of clicks to the abstract of a document INLINEFORM1 from the results page for the query INLINEFORM2 . Let INLINEFORM3 be the number of clicks from INLINEFORM4 's abstract page to its full-text, which result from the query INLINEFORM5 . Let INLINEFORM6 be the boost factor for documents without links to full-text articles. INLINEFORM7 is the indicator function such that INLINEFORM8 if the document INLINEFORM9 includes a link to full-text articles and INLINEFORM10 otherwise. We can then calculate the relevance, INLINEFORM11 , of a document for a given query: DISPLAYFORM0 
 INLINEFORM0 is the trade-off between the importance of abstract clicks and full-text clicks. The last term of the relevance function gives a slight boost to documents without full-text links, so that they get a better relevance (thus rank) than those for which full-text is available but never clicked, assuming they all have the same amount of abstract clicks. We manually tuned the parameters based on user behavior and log analyses, and used the settings, INLINEFORM1 and INLINEFORM2 .
Compared to the TREC Genomics set, the full PubMed set is much larger, including all 27 million documents in PubMed. While the TREC and PubMed sets share essentially the same type of documents, the tested queries are quite different. The queries in TREC are a question type, e.g. “what is the role of MMS2 in cancer?" However, the PubMed set uses actual queries from PubMed users.
In our experiments, the TREC set was used for evaluating BM25 and the semantic measure separately and the PubMed set was used for evaluating the learning to rank approach. We did not use the TREC set for learning to rank due to the small number of queries. Only 62 queries and 162,259 documents are available in TREC, whereas the PubMed set consists of many more queries and documents.
We used the skip-gram model of word2vec BIBREF16 to obtain word embeddings. The alternative models such as GloVe BIBREF11 and FastText BIBREF37 are available, but their performance varies depending on tasks and is comparable to word2vec overall BIBREF38 , BIBREF39 . word2vec was trained on titles and abstracts from over 25 million PubMed documents. Word vector size and window size were set to 100 and 10, respectively. These parameters were optimized to produce high recall for synonyms BIBREF40 . Note that an independent set (i.e. synonyms) was used for tuning word2vec parameters, and the trained model is available online (https://www.ncbi.nlm.nih.gov/IRET/DATASET).
For experiments, we removed stopwords from queries and documents. BM25 was chosen for performance comparison and the parameters were set to INLINEFORM0 and INLINEFORM1 BIBREF41 . Among document ranking functions, BM25 shows a competitive performance BIBREF42 . It also outperforms co-occurrence based word embedding models BIBREF13 , BIBREF14 . For learning to rank approaches, 70% of the PubMed set was used for training and the rest for testing. The RankLib library (https://sourceforge.net/p/lemur/wiki/RankLib) was used for implementing LambdaMART and the PubMed experiments.
Table TABREF17 presents the average precision of tf-idf (TFIDF), BM25, word vector centroid (CENTROID) and our embedding approach on the TREC dataset. Average precision BIBREF43 is the average of the precisions at the ranks where relevant documents appear. Relevance judgements in TREC are based on the pooling method BIBREF44 , i.e. relevance is manually assessed for top ranking documents returned by participating systems. Therefore, we only used the documents that annotators reviewed for our evaluation BIBREF1 .
As shown in Table TABREF17 , BM25 performs better than TFIDF and CENTROID. CENTROID maps each query and document to a vector by taking a centroid of word embedding vectors, and the cosine similarity between two vectors is used for scoring and ranking documents. As mentioned earlier, this approach is not effective when multiple topics exist in a document. From the table, the embedding approach boosts the average precision of BM25 by 19% and 6% on TREC 2006 and 2007, respectively. However, CENTROID provides scores lower than BM25 and SEM approaches.
Although our approach outperforms BM25 on TREC, we do not claim that BM25 and other traditional approaches can be completely replaced with the semantic method. We see the semantic approach as a means to narrow the gap between words in documents and those in queries (or users' intentions). This leads to the next experiment using our semantic measure as a feature for ranking in learning to rank.
For the PubMed dataset, we used learning to rank to combine BM25 and our semantic measure. An advantage of using learning to rank is its flexibility to add more features and optimize performance by learning their importance. PubMed documents are semi-structured, consisting of title, abstract and many more fields. Since our interest lies in text, we only used titles and abstracts, and applied learning to rank in two different ways: 1) to find semantically closest words in titles (BM25 + SEMTitle) and 2) to find semantically closest words in abstracts (BM25 + SEMAbstract). Although our semantic measure alone produces better ranking scores on the TREC set, this does not apply to user queries in PubMed. It is because user queries are often short, including around three words on average, and the semantic measure cannot differentiate documents when they include all query words.
Table TABREF19 shows normalized discounted cumulative gain (NDCG) scores for top 5, 10 and 20 ranked documents for each approach. NDCG BIBREF45 is a measure for ranking quality and it penalizes relevant documents appearing in lower ranks by adding a rank-based discount factor. In the table, reranking documents by learning to rank performs better than BM25 overall, however the larger gain is obtained from using titles (BM25 + SEMTitle) by increasing NDCG@20 by 23%. NDCG@5 and NDCG@10 also perform better than BM25 by 23% and 25%, respectively. It is not surprising that SEMTitle produces better performance than SEMAbstract. The current PubMed search interface does not allow users to see abstracts on the results page, hence users click documents mostly based on titles. Nevertheless, it is clear that the abstract-based semantic distance helps achieve better performance.
After our experiments for Table TABREF19 , we also assessed the efficiency of learning to rank (BM25 + SEMTitle) by measuring query processing speed in PubMed relevance search. Using 100 computing threads, 900 queries are processed per second, and for each query, the average processing time is 100 milliseconds, which is fast enough to be used in the production system.
We presented a word embedding approach for measuring similarity between a query and a document. Starting from the Word Mover's Distance, we reinterpreted the model for a query-document search problem. Even with the INLINEFORM0 flow only, the word embedding approach is already efficient and effective. In this setup, the proposed approach cannot distinguish documents when they include all query words, but surprisingly, the word embedding approach shows remarkable performance on the TREC Genomics datasets. Moreover, applied to PubMed user queries and click-through data, our semantic measure allows to further improves BM25 ranking performance. This demonstrates that the semantic measure is an important feature for IR and is closely related to user clicks.
While many deep learning solutions have been proposed recently, their slow training and lack of flexibility to adopt various features limit real-world use. However, our approach is more straightforward and can be easily added as a feature in the current PubMed relevance search framework. Proven by our PubMed search results, our semantic measure improves ranking performance without adding much overhead to the system.
This research was supported by the Intramural Research Program of the NIH, National Library of Medicine.

Figure and table captions:
Table 1: Number of documents in the TREC and PubMed datasets.
Table 2: Average precision of BM25 and our embedding approach (EMBED) on the TREC set.
Table 3: Average precision of BM25 and our embedding approach (EMBED) on the PubMed set.
Table 4: Average precision of BM25 and our approach (EMBED) on the TREC set. All documents in the set are scored.
Figure 1: TREC performance changes depending on α.

Questions about this paper:
How do they propose to combine BM25 and word embedding similarity?

Answers:
They merge features of BM25 and semantic measures.