# An Efficient Document Retrieval for Korean Open-Domain Question Answering Based on ColBERT
## 5 Experimental Setup and Results
### Results
SBERT is trained to discern the semantic similarity between two sentences by inputting each sentence into BERT separately and using the obtained representations. Therefore, SBERT's representations may be more suitable for similarity-based tasks compared to those of standard BERT. This is evidenced by KoSBERT exhibiting superior performance over KLUE-BERT. These findings can be instrumental in guiding the selection of pre-trained language models for future ODQA tasks. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Method** & **MRR@10** & **MRR@100** & **Recall@50** & **Recall@200** & **Latency (ms)** \\ \hline \multicolumn{5}{c}{**Baseline**} \\ \hline BM25 & 55.25 & 55.77 & 81.19 & 86.96 & **5883** \\ KoBERT & 68.09 & 68.38 & 88.49 & 90.76 & 16,466 \\ KoSBERT & **73.71** & **73.92** & **90.63** & **91.39** & 20,411 \\ \hline \multicolumn{5}{c}{**Ours**} \\ \hline [PERSON] & 62.55 & 62.98 & 86.29 & 89.99 & 1578 \\ [PERSON] & & & & \\ w/[PERSON]- & 66.04 & 66.40 & 86.66 & 90.26 & 886 \\ [PERSON] & & & & \\ [PERSON] & & & & \\ w/[PERSON] & 67.29 & 68.07 & 87.16 & 90.50 & 1240 \\ [PERSON] & & & & & \\ w/[PERSON]- & 67.91 & 68.24 & 86.99 & **90.79** & 1322 \\ [PERSON] & & & & & \\ [PERSON] & & & & & \\ w/[PERSON] & **68.15** & **68.48** & **87.26** & 90.63 & **873** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison of re-ranking performance using various pre-trained Korean language models. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Method** & **MRR@10** & **MRR@100** & **Recall@50** & **Recall@200** & **Latency (ms)** \\ \hline
**Max Query Length** & & & **20** & & \\ \hline BM25 & 23.18 & 23.91 & 51.81 & 64.48 & 5586 \\ [PERSON] & 20.05 & 21.09 & 55.01 & 70.29 & 16,328 \\ \hline [PERSON] w/ [PERSON] (Ours) & **41.15** & **41.79** & **69.22** & **74.99** & **1581** \\ \hline
**Max Query Length** & & & & **50** & \\ \hline BM25 & 51.34 & 51.9 & 79.52 & 85.66 & 5869 \\ [PERSON] & **65.05** & **65.46** & **86.89** & **89.76** & 16,465 \\ \hline [PERSON] w/ & & & & & \\ [PERSON] & 58.95 & 59.46 & 84.39 & 89.32 & **1538** \\ (Ours) & & & & & \\ \hline
**Max Query Length** & & & & & \\ \hline BM25 & 55.25 & 55.77 & 81.19 & 86.96 & 5883 \\ [PERSON] & **68.09** & **68.38** & **88.49** & **90.76** & 16,466 \\ \hline [PERSON] w/ [PERSON] (Ours) & 62.55 & 62.98 & 86.29 & 89.99 & **1578** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Comparison of re-ranking performance by query length. Table 5 presents additional comparative experimental results examining the re-ranking performance of KoSBERT for each task. In [39], BERT's sentence embeddings are improved by mapping semantically similar sentences close together in the vector space through the natural language inference (NLI) task, which discriminates between implication, contradiction, and neutrality by calculating embeddings for two sentences separately, and the semantic textual similarity (STS) task, which predicts similarity scores between 0 and 5 using the cosine similarity of two sentence embeddings. KoSBERT is a pre-trained Korean language model trained using the SBERT structure proposed in [39] on the KLUE-BERT model. KoSBERT-STS is a model fine-tuned with the KorSTS dataset [41], KoSBERT-NLI is a model fine-tuned with the KorNLI dataset [42], and KoSBERT-Multi is a model fine-tuned with KorNLI and further fine-tuned with KorSTS. The experiments show that the best performance is achieved when using KoSBERT-NLI among the various KoSBERT models. This implies that the model trained on inferring semantic relationships between sentences has a significant impact on exhibiting good search performance.