Question: Which benchmark does the first experiment perform on?
Evidence: In information retrieval (IR), queries and documents are typically represented by term vectors where each term is a content word and weighted by tf-idf, i.e. the product of the term frequency and the inverse document frequency, or other weighting schemes BIBREF0 . The similarity of a query and a document is then determined as a dot product or cosine similarity. Although this works reasonably, the traditional IR scheme often fails to find relevant documents when synonymous or polysemous words are used in a dataset, e.g. a document including only “neoplasm" cannot be found when the word “cancer" is used in a query. One solution of this problem is to use query expansion BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 or dictionaries, but these alternatives still depend on the same philosophy, i.e. queries and documents should share exactly the same words.
While the term vector model computes similarities in a sparse and high-dimensional space, the semantic analysis methods such as latent semantic analysis (LSA) BIBREF5 , BIBREF6 and latent Dirichlet allocation (LDA) BIBREF7 learn dense vector representations in a low-dimensional space. These methods choose a vector embedding for each term and estimate a similarity between terms by taking an inner product of their corresponding embeddings BIBREF8 . Since the similarity is calculated in a latent (semantic) space based on context, the semantic analysis approaches do not require having common words between a query and documents. However, it has been shown that LSA and LDA methods do not produce superior results in various IR tasks BIBREF9 , BIBREF10 , BIBREF11 and the classic ranking method, BM25 BIBREF12 , usually outperforms those methods in document ranking BIBREF13 , BIBREF14 .
Neural word embedding BIBREF15 , BIBREF16 is similar to the semantic analysis methods described above. It learns low-dimensional word vectors from text, but while LSA and LDA utilize co-occurrences of words, neural word embedding learns word vectors to predict context words BIBREF10 . Moreover, training of semantic vectors is derived from neural networks. Both co-occurrence and neural word embedding approaches have been used for lexical semantic tasks such as semantic relatedness (e.g. king and queen), synonym detection (e.g. cancer and carcinoma) and concept categorization (e.g. banana and pineapple belong to fruits) BIBREF10 , BIBREF17 . But, Baroni et al. Baroni2014 showed that neural word embedding approaches generally performed better on such tasks with less effort required for parameter optimization. The neural word embedding models have also gained popularity in recent years due to their high performance in NLP tasks BIBREF18 .
Here we present a query-document similarity measure using a neural word embedding approach. This work is particularly motivated by the Word Mover's Distance BIBREF19 . Unlike the common similarity measure taking query/document centroids of word embeddings, the proposed method evaluates a distance between individual words from a query and a document. Our first experiment was performed on the TREC 2006 and 2007 Genomics benchmark sets BIBREF20 , BIBREF21 , and the experimental results showed that our approach was better than BM25 ranking. This was solely based on matching queries and documents by the semantic measure and no other feature was used for ranking documents.
In general, conventional ranking models (e.g. BM25) rely on a manually designed ranking function and require heuristic optimization for parameters BIBREF22 , BIBREF23 . In the age of information explosion, this one-size-fits-all solution is no longer adequate. For instance, it is well known that links to a web page are an important source of information in web document search BIBREF24 , hence using the link information as well as the relevance between a query and a document is crucial for better ranking. In this regard, learning to rank BIBREF22 has drawn much attention as a scheme to learn how to combine diverse features. Given feature vectors of documents and their relevance levels, a learning to rank approach learns an optimal way of weighting and combining multiple features.
We argue that the single scores (or features) produced by BM25 and our proposed semantic measure complement each other, thus merging these two has a synergistic effect. To confirm this, we measured the impact on document ranking by combining BM25 and semantic scores using the learning to rank approach, LamdaMART BIBREF25 , BIBREF26 . Trained on PubMed user queries and their click-through data, we evaluated the search performance based on the most highly ranked 20 documents. As a result, we found that using our semantic measure further improved the performance of BM25.
Taken together, we make the following important contributions in this work. First, to the best of our knowledge, this work represents the first investigation of query-document similarity for information retrieval using the recently proposed Word Mover's Distance. Second, we modify the original Word Mover's Distance algorithm so that it is computationally less expensive and thus more practical and scalable for real-world search scenarios (e.g. biomedical literature search). Third, we measure the actual impact of neural word embeddings in PubMed by utilizing user queries and relevance information derived from click-through data. Finally, on TREC and PubMed datasets, our proposed method achieves stronger performance than BM25.
Our first experiment was performed on the TREC 2006 and 2007 Genomics benchmark sets, and the experimental results showed that our approach was better than BM25 ranking.
Answer: The first experiment was performed on the TREC 2006 and 2007 Genomics benchmark sets