# Multi Stage Retrieval for Web Search During Crisis
## 3 Materials and Methods
### Dataset  
The CrisisFACTS 2023 dataset [35; 36] comprises a series of documents from different sources for different catastrophic events (see Table 1). The events are tornadoes, wildfires, hurricanes, floods, and various accidents. They last for more than one day, and a minimum of 20,000 documents are associated. These documents are produced and deposited on the Internet daily during an event. The data in the collection is only weakly filtered (e.g., based on hashtags and keywords). For this reason, there is a lot of noise and irrelevant documents. The following streams are available:  
Figure 1: During each day of an event, the system has to select all relevant documents (blue squares in the figure) from the streams (News, Twitter, Reddit, and Facebook) and, at the end of each day, compose a summary using the selected documents. 1. _News_: News articles are an excellent source of information during catastrophic events, and a small number of pieces were included in the dataset. 2. _Twitter_: Twitter posts were collected based on keywords relevant to the events in the analysis. 3. _Reddit_: Top-level posts with all subsequent comments were extracted from Reddit threads relevant to the emergency and included in the dataset. 4. _Facebook_: Facebook posts from public pages are provided based on relevance to each disaster event. The first eight events were used during the 2022 competitions and can be used for parameter tuning during the 2023 edition. The evaluation set is composed of events from 009 to 018 (see Table 1). The dataset is also supplemented with queries (set \(Q\)) defining the information needs of disaster-response stakeholders extracted from FEMA ICS 209 forms. As reported in the problem statement, the set \(Q\) is optionally used. These queries capture what a responder might consider important, such as emerging/approaching threats, risks from hazardous materials, damage to key infrastructure, evacuations or emerging threats, statistics on casualties or numbers missing, weather concerns, and restrictions on or availability of resources. Some example queries and some related relevant documents are reported in Table 2.  
As ground truths, two summaries for each event are provided: \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Event ID** & **Name** & **Queries** & **Texts** & **Days** \\ \hline
001 & Lilac Wildfire 2017 & 52 & 51,015 & 9 \\
002 & Cranston Wildfire 2018 & 52 & 30,535 & 6 \\
003 & Holy Wildfire 2018 & 52 & 32,489 & 7 \\
004 & Hurricane Florence 2018 & 51 & 376,537 & 15 \\
005 & Maryland Flood 2018 & 48 & 41,770 & 4 \\
006 & Saddleridge Wildfire 2019 & 52 & 38,369 & 4 \\
007 & Hurricane Laura 2020 & 51 & 62,182 & 2 \\
008 & Hurricane Sally 2020 & 51 & 116,303 & 8 \\ \hline
009 & Beirut Explosion, 2020 & 56 & 468,178 & 7 \\
010 & Houston Explosion, 2020 & 56 & 72,530 & 7 \\
011 & Rutherford TN Floods, 2020 & 48 & 20,868 & 5 \\
012 & TN Derecho, 2020 & 49 & 79,688 & 7 \\
013 & Edenville Dam Fail, 2020 & 48 & 28,185 & 7 \\
014 & Hurricane Dorian, 2019 & 51 & 556,233 & 7 \\
015 & Kincade Wildfire, 2019 & 52 & 137,072 & 7 \\
016 & Easter Tornado Outbreak, 2020 & 51 & 131,954 & 5 \\
017 & Beirut Explosion, 2020 & 51 & 120,899 & 6 \\
018 & Tornado Outbreak, 2020 March & 51 & 200,008 & 6 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Description of CrisisFACTS events. Only the facts from 009 were evaluated in the 2023 edition. \begin{table}
\begin{tabular}{l l} \hline \hline
**Query** & **Relevant Document** \\ \hline What roads are closed? & Hwy 76 is closed both directions... \\ \hline How many people have been injured? & Two civilians are being treated for burn injuries... \\ \hline Where are wind speeds expected to be high? &...San Diego County as winds gust to 75 mph. \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Sample queries and relevant documents from the corpus. Wikipedia. It is the Wikipedia page for the given event, providing a high-level description. 2. NIST Summary. It is created by NIST assessors from the documents present in the corpus.