# Improving Sentence Retrieval Using Sequence Similarity
## 5 Experiments and Results  
The experiment was conducted using data from the novelty tracks of the Text Retrieval Conference (TREC). There were three TREC novelty tracks in the years from 2002 to 2004: TREC 2002, TREC 2003, and TREC 2004 [9; 10; 11]. The task was novelty detection, which consists of two subtasks: Finding relevant sentences and finding novel sentences. Our experiment was entirely focused on sentence retrieval, which represents the first task of novelty detection. Three data collections were used, each consisting of 50 topics (queries) and 25 documents per topic, with multiple sentences (Table 2) [30]. When it comes to TREC topics, it must be emphasized that each one has a _title_, a _description_, and _narrative_. These three parts represent three versions of the same query but with different lengths. The _title_ is the shortest query and _narrative_ the longest. In our tests, we used the shortest version called _\"title.\"_ Figure 1 depicts the _title_ of topic N1 from TREC 2003. \begin{table}
\begin{tabular}{c c} \hline \hline
**Method** & **Ranking Function** \\ \hline TF-ISF & TF-ISF\((s,\ q)\) \\ TF-ISF\({}_{part}\) & TF-ISF\({}_{partial_{(s,\ q)}}\) \\ BM25 & BM25\((s,\ q)\) \\ BM25\({}_{part}\) & BM25\({}_{partial_{(s,\ q)}}\) \\ LM & LM(\(q,\ s)\) \\ LM\({}_{part}\) & LM\({}_{partial_{(s)}}\)(\(q,\ s)\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Overview of all sentence retrieval methods tested in this paper. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Name of the** & **Number of Topics** & **Number of Documents** & **Number of** \\
**Collection** & **(Queries)** & **per Topic** & **Sentences** \\ \hline TREC 2002 & 50 & 25 & 57,792 \\ TREC 2003 & 50 & 25 & 39,820 \\ TREC 2004 & 50 & 25 & 52,447 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Description of dataset characteristics. Every query was executed on 25 documents. Each of the documents consisted of multiple sentences as described in Table 2. Each sentence was marked with a beginning and ending tag. Each sentence had a number. Figure 2 shows an extract form document NYT19980629.0465 with sentence number 11. Each TREC data collection also contains a list of relevant sentences. Figure 3 depicts an excerpt from the list of relevant sentences of TREC 2003. The marked line in Figure 3 defines sentence 11 from the document NYT19980629.0465 as relevant to the topic (query) N1 (\"partial birth abortion ban\"). To test whether the new methods provide better sentence retrieval results than the existing methods, we compared the performances of the new methods in relation to the existing methods (baseline methods) using the following standard measures: P@10, the MAP, and the _R-precision_[1; 6]. Figure 1: Example of query from Text Retrieval Conference (TREC) 2003 novelty track. Figure 3: An excerpt in the dataset that contains a list of relevant sentences. Figure 2: Example of the sentence part within the document from TREC 2003 novelty track. The precision at \(x\) or \(P@x\) can be defined as:  
\[P@x\Big{(}q_{j}\Big{)}=\frac{\text{number of relevant sentences}}{\text{ within top $x$ retrieved}} \tag{12}\]  
The P@10 values shown in this paper refer to average P@10 for 50 queries [30]. _R-precision_ can be defined as [1]:  
\[R\text{-}precision=\frac{r}{|Rel|} \tag{13}\]  
where  
* \(|Rel|\) is the number of relevant sentences to the query, and
* \(r\) is the number of relevant sentences in top \(|Rel|\) sentences of the result. The _R-precision_ values shown in this paper are (analogous to P@10) averages for 50 queries [30]. The Mean Average Precision and _R-precision_ gave similar results and were used to test high recall. High recall means it is more important to find all of the relevant sentences, even if it means searching through many sentences including many that are nonrelevant. Meanwhile, P@10 is used for testing precision [30]. In terms of information retrieval, precision means it is more important to get only relevant sentences than finding all of the relevant sentence [30].