# AI-Enhanced Disaster Management: A Modular OSINT System for Rapid Automated Reporting
## 5 Results and Discussion
### AlignScore  
To evaluate the quality of the reports generated by ODET in our case studies, the metric AlignScore was used to show the validity of the results. This metric makes it possible to compare machine-generated reports with an established reference source and to measure the alignment of the content elements. AlignScore provides a systematic and objective metric for evaluating the accuracy and completeness of automatically generated content compared to a reliable source [9] by measuring its structural and semantic alignment. The advantage of this metric is that it is possible to measure both content scope and semantic accuracy, which provides a more comprehensive insight into the system's performance. AlignScore considers not only the surface area but also the depth and context of the information. AlignScore is mathematically described as:  
\[\textsc{AlignScore}(o,l)=\underset{j}{\text{mean}}\ \underset{i}{\text{max}}\ \text{ alignment}(o^{\prime}_{i},l^{\prime}_{j}), \tag{1}\]  
where \(o\) refers to the context, \(l\) represents the claim, \(o^{\prime}_{i}\) is the collection of context chunks, \(l^{\prime}_{j}\) is the collection of claim sentences, and alignment\((\cdot)\) denotes the likelihood that the model assigns to the Aligned label in a 3-class classification setup [9].  
The goal of ODET is to generate situational awareness reports from unstructured disaster data. This task requires a robust evaluation platform that can assess how well generated outputs reflect real-world events. Compared to traditional metrics such as F1 score, AlignScore is well suited for ODET evaluation because it focuses on factual consistency rather than discrete classification accuracy. AlignScore addresses this need by holistically evaluating the semantic fidelity of the generated text to the reference material to identify nuanced discrepancies such as omissions, factual distortions, or misrepresentations [9].  
Throughout the case studies of the 2023 Turkey Earthquake and Hurricane Harvey, Wikipedia was a valuable source for evaluating reports. A direct timeline of events was not available; therefore, Wikipedia pages had to be relied upon as a comprehensive source of information. These pages contain well-researched and detailed neutral information that has been edited and added by numerous authors over the years. Although Wikipedia contains retrospective insights and information that only became available after the event, it provides a reliable basis for comparison.  
To evaluate the case studies, each of the 58 ODET-generated reports for the Hurricane Harvey case study and 131 reports for the 2023 Turkey Earthquake case study were analyzed in a final summary report and then compared to the corresponding Wikipedia article. The high level of agreement indicates that the ODET-generated reports were largely accurate despite the limitations of live data and the limited database.  
A final result of 0.89 (Hurricane Harvey) and 0.84 (2023 Turkey earthquake) was achieved, matching the generated ODET reports and the corresponding Wikipedia articles. These values confirm a remarkable agreement between the two sources, especially considering the different natures of the data sources. While Wikipedia is based on retrospective research and extensive source studies, ODET reports reflect information gathered in real time during the event.