## 4 Evaluation
### Experimental Results  
The first experiment was conducted to evaluate the effectiveness of the additional input embeddings (i.e., position embedding, overlapped position embedding, and IDF embedding) and the phrase modeling layer by comparing the changes in performance, as presented in Table 1.  
In Table 1, \"w/o additional input embeddings\" refers to a modified model in which only word embeddings are used as input units. Therefore, \"w/o phrase modeling layer\" means a modification of our model in which the phrase modeling layer is excluded. As presented in Table 1, the additional input embeddings and phrase modeling layer contribute to the improvement of MRR@10 by 8%p and 0.8%p, respectively. In addition, to check whether the word \(n\)-gram features can effectively contain phrase information or not, we visualized the degrees of associations between the \(n\)-gram phrases in a query and those in a passage (i.e., P-Q attention scores in Equation (2)) through 2-dimensional heat maps, as shown in Figure 6.  
In Figure 6, the \(n\)-gram phrases with higher attention scores were colored in bluer. As illustrated in Figure 6, the uni-gram features were colored in bluer between single words or short phrases, and the tri-gram features were colored in bluer between long phrases. It reveals that each \(n\)-gram feature differently contributes to capturing associations between a query and a passage. The second experiment was conducted to compare the proposed model with the earlier models, as presented in Table 2.  
Referring to Table 2, BM25 is a traditional retrieval model termed Okapi BM25, and Duet v2 is a joint ANN model comprising a local model based on term frequencies and a distributed \begin{table}
\begin{tabular}{c c} \hline \hline
**Model** & **MRR@10** \\ \hline Proposed model & 0.303 \\ w/o additional input embeddings & 0.223 \\ w/o phrase modeling layer & 0.295 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Performance changes in development set. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **MRR@10** \\ \hline Proposed model & 0.303 \\ w/o additional input embeddings & 0.223 \\ w/o phrase modeling layer & 0.295 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Performance changes in development set. Figure 6: Heat map for visualizing the degrees of associations between a query and a passage. model based on word vectors. Conv-KNRM is an ANN model in which queries and documents are encoded using CNNs. [PERSON] et al. [11] refer to a bidirectional long short-term memory network model with a co-attention mechanism between query and passage representations. BERT-Base and BERT-Large are fine-tuned classification models based on the base and large models of BERT [25], which is a pre-trained language model with state-of-the-art performance in several downstream natural language processing tasks such as span prediction, sequence labeling, and text classification [26]. As presented in Table 2, the proposed model outperformed all the previous models except the BERT-based models (The proposed model named \"n-gram co-attention\" can be found in the official rankings: [[https://microsoft.github.io/msmarco/](https://microsoft.github.io/msmarco/)]([https://microsoft.github.io/msmarco/](https://microsoft.github.io/msmarco/))). Table 3 presents the memory usages and the response times of the proposed and BERT-based models. In Table 3, the response time is the average time per query that is spent to rank the top 1000 passages retrieved with BM25 [22]. In order to re-rank the passages retrieved against 6900 queries in the development set, BERT-Base spent approximately 2.108 h, but the proposed model spent approximately 0.575 h. As presented in Table 3, although the proposed model demonstrated lower performance than the BERT models, it demonstrated significantly less memory usage (about 8.0 times less) and faster response time (about 3.7 times faster) than the latter.