## 3 The Proposed System
### Sentence Classification
#### 3.2.1 Target Sentence Labeling
We selected LS = 0.5 and HS = 0.8, as these values are the best parameters in the highest number of news events. Table 7 displays the F1 score performance of the test news for each fold. On average, our system achieved an F1 score of 0.77, with the lowest value being 0.67 and the highest reaching 0.87. To compare our system with state-of-the-art classification models, we utilized two BERT classifiers based on the bert-base-uncased model [3]. In the first comparison, we fine-tuned the model within the leave-one-out cross-validation setting. The fine-tuning parameters were set as follows: a maximum sequence length of 128, a learning rate of 2 \(\times\) 10\({}^{-5}\), and a batch size of 8 (due to computational constraints). The model was trained for 5 epochs in each fold, with the remaining parameters following the standard settings of the Hugging Face implementation [[https://huggingface.co/bert-base-uncased](https://huggingface.co/bert-base-uncased)]([https://huggingface.co/bert-base-uncased](https://huggingface.co/bert-base-uncased)) (accessed on 3 November 2023). In the second comparison, we employed prompt tuning with the following parameters: a maximum sequence length of 128, a learning rate of 3 \(\times\) 10\({}^{-2}\), and a batch size of 8 (again, due to computational limitations). The model was trained for 30 epochs in each fold. \begin{table}
\begin{tabular}{c c|c c c c c c} \hline
**LS** & **HS** & **M.L.** & **I.L.** & **I.E.** & **G.W.** & **G.F.** & **E.P.** & **C.L.** \\ \hline
0.4 & 0.6 & 0.46 & 0.61 & 0.62 & 0.61 & 0.47 & 0.87 & 0.59 \\
0.4 & 0.7 & 0.52 & 0.58 & 0.63 & 0.62 & 0.47 & **0.89** & 0.61 \\
0.4 & 0.8 & 0.54 & 0.58 & 0.65 & 0.62 & 0.47 & **0.89** & 0.59 \\
0.5 & 0.6 & 0.6 & 0.75 & 0.75 & 0.7 & **0.79** & 0.85 & 0.75 \\
0.5 & 0.7 & 0.65 & **0.79** & 0.76 & 0.71 & **0.79** & 0.87 & 0.77 \\
0.5 & 0.8 & **0.67** & **0.79** & 0.78 & **0.72** & **0.79** & 0.87 & 0.75 \\
0.6 & 0.7 & 0.56 & 0.72 & 0.82 & 0.71 & **0.79** & 0.87 & **0.88** \\
0.6 & 0.8 & 0.58 & 0.72 & **0.84** & **0.72** & **0.79** & 0.87 & 0.86 \\ \hline \end{tabular}
\end{table} 
Table 6: F1 scores of different threshold configurations in an example fold. \begin{table}
\begin{tabular}{c c} \hline
**News** & **F1 score** \\ \hline Mars Landing & 0.67 \\ Italy Lockdown & 0.79 \\ Italy Euro 2020 & 0.78 \\ Global Warming & 0.72 \\ [PERSON] Death & 0.79 \\ [PERSON] & 0.87 \\ China Landed on Mars & 0.75 \\ Beirut Explosion & 0.81 \\ \hline
**Average** & **0.772** \\ \hline \end{tabular}
\end{table} 
Table 7: F1 score for the single news and the average F1 score. We present the results of the two BERT classifiers and compare them with our system in Table 8. The table demonstrates that, on average, fine-tuned BERT performs slightly better than our system, which, in turn, outperforms the model tuned with prompts. It is worth noting that fine-tuning a classification model based on BERT can be expensive in terms of both time and computational costs. In contrast, our system is significantly more efficient in both the training and classification phases, requiring only the extraction of sentence-level embeddings and the computation of the similarity matrix. Prompt tuning, on the other hand, tends to work better with larger models and demands substantial computational resources. We computed the Wilcoxon signed-rank test to verify if the differences between the performances of our system and the two BERT classifiers are statistically significant. For the computation of the statistics, we used the SciPy method [[https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html)]([https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html)) (accessed on 3 November 2023), maintaining the default parameters.