# Multi Stage Retrieval for Web Search During Crisis
## 4 Results
### Results at the Event Level  
This section reports detailed information about the performance for each of the test events using BERT-Score (see Table 6). The BERT-Score metrics are generally higher for the NIST-based ground truth than for the Wikipedia-based one. This suggests that the model performs better on NIST in terms of capturing semantic similarity between the generated and reference summaries. This is expected since Wikipedia summaries are generic and not built from the corpus [36]. The scores for NIST are relatively consistent across different events, with F1 scores ranging from 76.19 to 81.72. This indicates stable performance across various scenarios. In contrast, the Wikipedia-based results show more variability in precision. Precision (P) and Recall (R) are balanced for NIST, with recall generally being slightly higher. This suggests that the model is capturing most of the relevant information (high recall) while maintaining good precision. Precision is generally lower than recall for Wikipedia, indicating that the model might include some irrelevant information in the summaries.  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Event** & \multicolumn{3}{c}{**NIST**} & \multicolumn{3}{c}{**Wikipedia**} \\ \hline  & **F1** & **P** & **R** & **F1** & **P** & **R** \\ \hline
009 & 79.35 \(\pm\) 1.37 & 77.10 \(\pm\) 2.11 & 81.74 \(\pm\) 0.65 & 73.30 \(\pm\) 0.96 & 72.52 \(\pm\) 1.40 & 74.09 \(\pm\) 0.63 \\
010 & 76.17 \(\pm\) 2.00 & 74.63 \(\pm\) 2.92 & 77.81 \(\pm\) 1.09 & 72.72 \(\pm\) 2.27 & 70.70 \(\pm\) 3.06 & 74.88 \(\pm\) 1.37 \\
011 & 78.86 \(\pm\) 1.05 & 79.02 \(\pm\) 1.04 & 78.70 \(\pm\) 1.12 & — & — & — \\
012 & 78.52 \(\pm\) 1.29 & 77.63 \(\pm\) 2.01 & 79.43 \(\pm\) 0.77 & — & — & — \\
013 & 76.22 \(\pm\) 1.61 & 75.17 \(\pm\) 2.13 & 77.30 \(\pm\) 1.06 & 68.67 \(\pm\) 1.18 & 65.48 \(\pm\) 1.51 & 72.19 \(\pm\) 0.77 \\
014 & 76.41 \(\pm\) 0.51 & 78.41 \(\pm\) 0.48 & 74.51 \(\pm\) 0.67 & 75.93 \(\pm\) 0.24 & 76.38 \(\pm\) 0.34 & 75.48 \(\pm\) 0.15 \\
015 & 78.48 \(\pm\) 0.48 & 78.30 \(\pm\) 0.82 & 78.67 \(\pm\) 0.14 & 73.39 \(\pm\) 0.19 & 70.27 \(\pm\) 0.50 & 76.80 \(\pm\) 0.24 \\
016 & 77.01 \(\pm\) 0.82 & 74.85 \(\pm\) 0.96 & 79.30 \(\pm\) 0.72 & 72.44 \(\pm\) 0.30 & 70.87 \(\pm\) 0.50 & 74.08 \(\pm\) 0.11 \\
017 & 80.26 \(\pm\) 0.38 & 78.22 \(\pm\) 0.47 & 82.40 \(\pm\) 0.28 & 71.04 \(\pm\) 0.47 & 65.27 \(\pm\) 0.53 & 77.95 \(\pm\) 0.49 \\
018 & 81.45 \(\pm\) 0.79 & 80.96 \(\pm\) 0.81 & 81.94 \(\pm\) 0.78 & 74.32 \(\pm\) 0.31 & 70.93 \(\pm\) 0.48 & 78.06 \(\pm\) 0.38 \\ \hline Mean & 78.27 \(\pm\) 1.71 & 77.43 \(\pm\) 1.92 & 79.18 \(\pm\) 2.29 & 72.73 \(\pm\) 2.03 & 70.30 \(\pm\) 3.38 & 75.44 \(\pm\) 1.92 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Comparison of NIST and Wiki BERScore scores for different events. Wikipedia summaries are missing for events 011 and 012.  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{**Search Mode**} & \multirow{2}{*}{**Time (s)**} & \multicolumn{3}{c}{**NIST**} & \multicolumn{3}{c}{**Wiki**} \\ \cline{3-8}  & & **F1** & **P** & **R** & **F1** & **P** & **R** \\ \hline Lexical (single-stage) & 913.50 \(\pm\) 35.83* & 77.47 \(\pm\) 3.56* & 76.54 \(\pm\) 4.17 & 78.46 \(\pm\) 3.22* & 72.62 \(\pm\) 3.10 & 70.00 \(\pm\) 4.15 & 75.52 \(\pm\) 2.59 \\ Dense (single-stage) & 1091.89 \(\pm\) 1.09* & 77.96 \(\pm\) 2.57 & 76.83 \(\pm\) 2.96 & 79.16 \(\pm\) 2.68 & 72.66 \(\pm\) 1.97 & 69.95 \(\pm\) 3.05 & 75.67 \(\pm\) 3.94 \\ \hline MSCR & 1023.99 \(\pm\) 4.16 & 77.76 \(\pm\) 2.54 & 76.71 \(\pm\) 2.93 & 78.89 \(\pm\) 2.58 & 72.69 \(\pm\) 2.09 & 70.02 \(\pm\) 3.19 & 75.64 \(\pm\) 1.72 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Execution time and BERT-Score. Comparison between multi-stage and single-stage approaches. * indicates statistically significant difference with \(p<0.05\) compared to MSCR.