Question: What ways do they use to do query reformulation?
Evidence: Rouge functions based on the assumption that in order for a summary to be of high quality, it has to share many words or phrases with a human gold summary. However, different terminology may be used to refer to the same concepts and thus relying only on lexical overlaps may underrate content quality scores. To overcome this problem, we propose an approach based on the premise that concepts take meanings from the context they are in, and that related concepts co-occur frequently.
Our proposed metric is based on analysis of the content relevance between a system generated summary and the corresponding human written gold-standard summaries. On high level, we indirectly evaluate the content relevance between the candidate summary and the human summary using information retrieval. To accomplish this, we use the summaries as search queries and compare the overlaps of the retrieved results. Larger number of overlaps, suggest that the candidate summary has higher content quality with respect to the gold-standard. This method, enables us to also reward for terms that are not lexically equivalent but semantically related. Our method is based on the well established linguistic premise that semantically related words occur in similar contexts BIBREF5 . The context of the words can be considered as surrounding words, sentences in which they appear or the documents. For scientific summarization, we consider the context of the words as the scientific articles in which they appear. Thus, if two concepts appear in identical set of articles, they are semantically related. We consider the two summaries as similar if they refer to same set of articles even if the two summaries do not have high lexical overlaps. To capture if a summary relates to a article, we use information retrieval by considering the summaries as queries and the articles as documents and we rank the articles based on their relatedness to a given summary. For a given pair of system summary and the gold summary, similar rankings of the retrieved articles suggest that the summaries are semantically related, and thus the system summary is of higher quality.
Based on the domain of interest, we first construct an index from a set of articles in the same domain. Since TAC 2014 was focused on summarization in the biomedical domain, our index also comprises of biomedical articles. Given a candidate summary INLINEFORM0 and a set of gold summaries INLINEFORM1 ( INLINEFORM2 ; INLINEFORM3 is the total number of human summaries), we submit the candidate summary and gold summaries to the search engine as queries and compare their ranked results. Let INLINEFORM4 be the entire index which comprises of INLINEFORM5 total documents.
Let INLINEFORM0 be the ranked list of retrieved documents for candidate summary INLINEFORM1 , and INLINEFORM2 the ranked list of results for the gold summary INLINEFORM3 . These lists of results are based on a rank cut-off point INLINEFORM4 that is a parameter of the system. We provide evaluation results on different choices of cut-off point INLINEFORM5 in the Section SECREF5 We consider the following two scores: (i) simple intersection and (ii) discounted intersection by rankings. The simple intersection just considers the overlaps of the results in the two ranked lists and ignores the rankings. The discounted ranked scores, on the other hand, penalizes ranking differences between the two result sets. As an example consider the following list of retrieved documents (denoted by INLINEFORM6 s) for a candidate and a gold summary as queries:
Results for candidate summary: INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 
Results for gold summary: INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 
These two sets of results consist of identical documents but the ranking of the retrieved documents differ. Therefore, the simple intersection method assigns a score of 1.0 while in the discounted ranked score, the score will be less than 1.0 (due to ranking differences between the result lists).
We now define the metrics more precisely. Using the above notations, without loss of generality, we assume that INLINEFORM0 . Sera is defined as follows: INLINEFORM1 
To also account for the ranked position differences, we modify this score to discount rewards based on rank differences. That is, in ideal score, we want search results from candidate summary ( INLINEFORM0 ) to be the same as results for gold-standard summaries ( INLINEFORM1 ) and the rankings of the results also be the same. If the rankings differ, we discount the reward by log of the differences of the ranks. More specifically, the discounted score (Sera-Dis) is defined as: INLINEFORM2 
where, as previously defined, INLINEFORM0 , INLINEFORM1 and INLINEFORM2 are total number of human gold summaries, result list for the candidate summary and result list for the human gold summary, respectively. In addition, INLINEFORM3 shows the INLINEFORM4 th results in the ranked list INLINEFORM5 and INLINEFORM6 is the maximum attainable score used as the normalizing factor.
We use elasticsearch[1], an open-source search engine, for indexing and querying the articles. For retrieval model, we use the Language Modeling retrieval model with Dirichlet smoothing BIBREF6 . Since TAC 2014 benchmark is on summarization of biomedical articles, the appropriate index would be the one constructed from articles in the same domain. Therefore, we use the open access subset of Pubmed[2] which consists of published articles in biomedical literature.
[1]https://github.com/elastic/elasticsearch [2]PubMed is a comprehensive resource of articles and abstracts published in life sciences and biomedical literature http://www.ncbi.nlm.nih.gov/pmc/
We also experiment with different query (re)formulation approaches. Query reformulation is a method in Information Retrieval that aims to refine the query for better retrieval of results. Query reformulation methods often consist of removing ineffective terms and expressions from the query (query reduction) or adding terms to the query that help the retrieval (query expansion). Query reduction is specially important when queries are verbose. Since we use the summaries as queries, the queries are usually long and therefore we consider query reductions.
In our experiments, the query reformulation is done by 3 different ways: (i) Plain: The entire summary without stopwords and numeric values; (ii) Noun Phrases (NP): We only keep the noun phrases as informative concepts in the summary and eliminate all other terms; and (iii) Keywords (KW): We only keep the keywords and key phrases in the summary. For extracting the keywords and keyphrases (with length of up to 3 terms), we extract expressions whose idf[1] values is higher than a predefined threshold that is set as a parameter. We set this threshold to the average idf values of all terms except stopwords. idf values are calculated on the same index that is used for the retrieval.
[1]Inverted Document Frequency
We hypothesize that using only informative concepts in the summary prevents query drift and leads to retrieval of more relevant documents. Noun phrases and keywords are two heuristics for identifying the informative concepts.
In our experiments, the query reformulation is done by 3 different ways: (i) Plain: The entire summary without stopwords and numeric values; (ii) Noun Phrases (NP): We only keep the noun phrases as informative concepts in the summary and eliminate all other terms; and (iii) Keyword (KW): We only keep the keywords and key phrases in the summary.
Answer: The entire summary without stopwords and numeric values; (ii) Noun Phrases (NP): They only keep the noun phrases as informative concepts in the summary and eliminate all other terms; and (iii) Keyword (KW): They only keep the keywords and key phrases in the summary.