# An Efficient Document Retrieval for Korean Open-Domain Question Answering Based on ColBERT
## 4 Korean Open-Domain Question Answering Dataset Construction
### Evaluation Dataset  
In our study, we employ the BM25 algorithm as a preliminary step in constructing the test set for evaluating our proposed methodology. BM25 identifies and ranks documents based on their relevance to a query, which is achieved by balancing the term frequency with the term's inverse document frequency, adjusted for document length. The score of a document \(D\) for a query \(Q\) using BM25 is computed as follows:  
\[\text{Score}(D,Q)=\sum_{i=1}^{n}IDF(q_{i})\times\frac{f(q_{i},D)\times(k_{1}+1)} {f(q_{i},D)+k_{1}\times(1-b+b\times\frac{|D|}{\text{avgdl}})} \tag{2}\] \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Divisions**} & **Min, Max** & \multirow{2}{*}{\begin{tabular}{c} **Avg Query** \\ **Length** \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} **Avg Query** \\ **Length** \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} **Avg** \\ **Document** \\ **Length** \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} **Queries** \\ **Length** \\ \end{tabular} } & \multirow{2}{*}{\
\begin{tabular}{c} **d+ Docu-** \\ **ments** \\ \end{tabular} } & \multirow{2}{*}{
\begin{tabular}{c} **d– Docu-** \\ **ments** \\ \end{tabular} } \\ \hline
20 & 5–20 & 16.7 & 582.2 & 122,328 & 69,001 & 413,711 \\
50 & 5–50 & 28.8 & 544.9 & 126,574 & 97,254 & 460,440 \\
200 & 5–183 & 31.3 & 544.7 & 126,725 & 98,714 & 461,452 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: ColBERT training triple <q, d+, d–> dataset statistics information. Figure 3: Process of constructing a dataset for open-domain question answering. This formula incorporates the term frequency \(f(q_{i},D)\) and inverse document frequency \(IDF(q_{i})\), and adjustments for the document's length and the average document length in the corpus, with the hyperparameters \(k_{1}\) and \(b\), respectively. Using BM25, we generated a pre-indexed set of documents for each of the 2999 queries in our dataset. This involves selecting the top 1000 documents based on their BM25 relevance scores. These pre-selected documents constituted the test set for our subsequent evaluation with ColBERT. Among the 1000 documents, the gold-label ranking for the relevant document d+ in each query is set to 1. The remaining 999 documents are considered low-relevance documents d\(-\). This configuration aims to assess the model's ability to rank the most relevant document as the top result. The statistical information of the dataset used for the re-ranking performance evaluation experiment is shown in Table 2.  
We analyzed our constructed dataset from the perspectives of question types and reasoning types. The distribution of these question and reasoning types is visualized in Figure 4. We manually annotated the question and reasoning types for a randomly sampled 100 questions. We adopted the classification scheme of CMRC [34], categorizing them into seven types: Who, What, When, Where, Why, How, and Others. The 'What' type constitutes approximately 60%, which is similar to the 54% for 'What' questions in SQuAD [35]. For reasoning types, following the criteria set by [PERSON] et al. [36] and [PERSON] et al. [37], the questions were annotated into five different categories: word matching (WM), paraphrasing (PP), single-sentence reasoning (SSR), multi-sentence reasoning (MSR), and ambiguous/insufficient (AoI). The proportion of reasoning types in our dataset stands at 48%, which is higher than the 21% in SQuAD and 34% in NewsQA [38]. This highlights that our dataset is more challenging compared to SQuAD and NewsQA. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Divisions**} & **Min, Max** & \multirow{2}{*}{**Avg Query**} & \multicolumn{2}{c}{**Avg**} & \multirow{2}{*}{**Queries**} & \multirow{2}{*}{**k**} \\  & **Query Length** & & & **Document Length** & \\ \hline
20 & 5–20 & 16.7 & 601.6 & 2999 & 1000 \\
50 & 5–50 & 28.9 & 608.9 & 2999 & 1000 \\
200 & 5–124 & 31.3 & 609.7 & 2999 & 1000 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: ColBERT top-k re-ranking evaluation dataset statistics information.