## 4 Experimental Results
### Preliminary System Evaluation on Real-World Data  
During the evaluation process, the need for a news database was raised. Due to the inexistence of a public and open Greek media news database, a custom one was constructed and populated, consisting of 38,759 articles, gathered in 175 days from 412 online news platforms. A custom tool for news-feed collection from real-life aggregators was developed, aiming to provide the deployed database with necessary data. Each article was classified as \"breaking\" or \"not breaking\" by assigning the respective label to it. Classification could only be carried out by humans; thus, three experts were invited to process the article titles of two whole dates (526 articles) and place a \"vote\" for each one they perceived as breaking [61]. Two testing sets were created, namely _three-vote unanimous_ and the _two-vote majority_. The breaking events of the former set were voted by all three experts as such, whilst the breaking events of the latter were voted by at least two experts. The dataset is publicly available on [[http://m3c.web.auth.gr/research/datasets/bndb/](http://m3c.web.auth.gr/research/datasets/bndb/)]([http://m3c.web.auth.gr/research/datasets/bndb/](http://m3c.web.auth.gr/research/datasets/bndb/)) (accessed on 9 November 2021). After constructing this first testing dataset and running the detection tool on it, we determined the evaluation metrics that should be used. Precision and recall are two of the most commonly used metrics in information retrieval and classification [62]. Precision is the fraction of the documents retrieved that are relevant to the user's information need:  
\[\text{precision}\;=\frac{|\{\text{relevant documents}\}|\cap|\{\text{retrieved documents}\}|}{|\{\text{retrieved documents}\}|} \tag{2}\]  
Recall is the fraction of the documents that are relevant to the query that is successfully retrieved:  
\[\text{recall}\;=\frac{|\{\text{relevant documents}\}|\cap|\{\text{retrieved documents}\}|}{|\{\text{relevant documents}\}|} \tag{3}\] \begin{table}
\begin{tabular}{c l c c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**Question**} & **E1** & **E2** & **E3** & **E4** & **E5** & **E6** & **E7** & **E8** & **E9** & **E10** \\ \hline EQ1 & \begin{tabular}{c} Would you be interested in a system that collects all the news articles to \\ create a data set? \\ \end{tabular} & 4 & 5 & 4 & 3 & 3 & 4 & 5 & 5 & 4 & 5 \\ \hline EQ2 & \begin{tabular}{c} Would you be interested in a system to look up information? \\ \end{tabular} & 2 & 3 & 2 & 2 & 3 & 3 & 3 & 4 & 4 & 5 \\ \hline EQ3 & \begin{tabular}{c} Would you be interested in a system that would allow you to compare \\ articles with each other? \\ \end{tabular} & 5 & 5 & 4 & 4 & 4 & 5 & 4 & 5 & 5 & 5 \\ \hline EQ4 & \begin{tabular}{c} Would you be interested in an automatic emergency detection system? \\ \end{tabular} & 3 & 5 & 5 & 4 & 4 & 3 & 4 & 4 & 3 & 4 \\ \hline EQ5 & \begin{tabular}{c} An automatic breaking news system is useful. \\ \end{tabular} & 3 & 4 & 4 & 4 & 4 & 4 & 4 & 3 & 4 \\ \hline EQ6 & \
\begin{tabular}{c} I would use an automatic emergency detection system. \\ \end{tabular} & 3 & 5 & 5 & 4 & 4 & 4 & 4 & 4 & 4 & 5 \\ \hline EQ7 &
\begin{tabular}{c} I would trust an automatic emergency detection system. \\ \end{tabular} & 2 & 3 & 3 & 3 & 3 & 3 & 4 & 4 & 4 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Questionnaire answered, evaluating the acceptance of presented work by experts (E1–E10) (1: strongly disagree, 2: disagree, 3: neutral, 4: agree, 5: strongly agree). Figure 6: Mean and standard deviation of the answers given in the validation questionnaire. The Precision-Recall curve is one of the most used metrics for the evaluation of binary classifiers.