IDRISI-RA: The First Arabic Location Mention Recognition Dataset of Disaster Tweets
Year: 2023

Abstract:
Extracting geolocation information from social media data enables effective disaster management, as it helps response authorities; for example, in locating incidents for planning rescue activities, and affected people for evacuation. Nevertheless, geolocation extraction is greatly understudied for the low resource languages such as Arabic. To fill this gap, we introduce IDRISI-RA, the first publicly-available Arabic Location Mention Recognition (LMR) dataset that provides humanand automatically-labeled versions in order of thousands and millions of tweets, respectively. It contains both location mentions and their types (e.g., district, city). Our extensive analysis shows the decent geographical, domain, location granularity, temporal, and dialectical coverage of IDRISI-RA. Furthermore, we establish baselines using the standard Arabic NER models and build two simple, yet effective, LMR models. Our rigorous experiments confirm the need for developing specific models for Arabic LMR in the disaster domain. Moreover, experiments show the promising domain and geographical generalizability of IDRISI-RA under zero-shot learning.

heading: 1 Introduction
text: Worldwide, and in the Arab world, Twitter has played a critical operational role in crisis management. The Beirut explosion in 2020 is an excellent case in point, where on-site individuals started intuitively responding to each other using geotagged tweets. What makes tweets invaluable is the presence of location mentions at different granularity (Grace et al., 2018; McCormick, 2016; Reuter et al., 2016; Kropczynski et al., 2018). Response authorities exploit this geographical information to effectively manage emergencies using Crisis Maps. Although the geographical dimension adds situational and operational values to Twitter data, on 18 June 2019 Twitter discontinued the geotagging
feature in tweets.1 This necessitates the need to develop automatic geolocation tools. Nevertheless, the main obstacle for the Arabic language, which is a low-resource language, the LMR task is severely understudied due to several factors, including the absence of a unified evaluation framework constituting annotated datasets, a representative set of baselines, and fair evaluation metrics.
To address these barriers, we focus on the Location Mention Recognition (LMR) task and introduce IDRISI-RA,2 the first human-labeled dataset comprising Arabic tweets from 7 disaster events (gold annotations). We also introduce the first largescale automatically-labeled tweets (silver annotations) from 22 disaster events that cover all Arab world. IDRISI-RA covers the most occurring disaster types that happened in the Arab world. More importantly, it is labeled for two annotation types that are location mentions (i.e., toponym textual spans) and their types (e.g., city, street, POIs, etc.). Hence, it supports type-less LMR, where the model detects toponyms at any granularity, and type-based LMR, where the model distinguishes LMs types while detecting them (Suwaileh et al., 2023).
Although adapting Named Entity Recognition (NER) models and English datasets goes a long way towards tackling the LMR task, researchers have empirically shown that specialized LMR models yield better performance and are more effective for emergency management tasks (Suwaileh et al., 2022). For the Arabic language, only a few datasets exist yet suffer from their limited geographical, domain, and dialectal coverage. What exacerbates the low resources issue for the Arabic language is the unavailability of LMR datasets, except a few task-specific ones, including traffic surveillance
1https://twitter.com/TwitterSupport/status/ 1141039841993355264
2Named after Muhammad Al-Idrisi, who is one of the pioneers and founders of advanced geography: https:// en.wikipedia.org/wiki/Muhammad_al-Idrisi. The “R" refers to the recognition task.
16298
and event detection (Al Emadi et al., 2017; Alkouz and Al Aghbari, 2018; Alkouz and Al Aghbari, 2020; Bahnasy et al., 2020). Therefore, in an effort to expedite the development of Arabic LMR models and shape future directions, we perform extensive experiments to empirically answer the following research questions: RQ1: Are standard Arabic NER models sufficient for effective LMR over disaster tweets? RQ2: Can LMR models trained on IDRISI-RA generate generalizable LMR models that reasonably perform on unseen disaster events? RQ3: Can LMR models trained on IDRISI-RA generate domain generalizable LMR models that reasonably perform on unseen disaster events of the same or different types? RQ4: Can LMR models trained on IDRISI-RA generate geographically generalizable LMR models that reasonably perform on unseen disaster events that happened in different countries?
Our rigorous analyses and experiments necessitate the development of specialized LMR models for the disaster domain. Additionally, the experiments demonstrate promising domain and geographic generalizability of IDRISI-RA under zeroshot learning.
The contributions of this paper are as follows:
• We present IDRISI-RA,3 the first public human-labeled Arabic LMR dataset (gold version) of about 4.6k tweets. The dataset covers diverse disaster types and countries. • We release the largest automatically-labeled Arabic LMR dataset (silver version), constituting about 1.2M tweets. • We annotate the location mentions into coarseand fine-grained location types to enable hierarchical LM recognition, disambiguation, and evaluation. • We benchmark IDRISI-RA using the standard Arabic NER models and our own simple yet competitive LMR models to establish a set of baselines for the community. • We empirically demonstrate that IDRISI-RA is a reasonably generalizable dataset.
The rest of the paper is organized as follows. Section 2 discusses the related work. Section 3 presents an overview of the LMR problem. Section 4 describes the design objective, dataset creation, and annotation. Section 5 analyzes the relia-
3https://github.com/rsuwaileh/IDRISI/
bility, coverage, and diversity of IDRISI-RA. Sections 6-7 present benchmarks and generalizability study on IDRISI-RA. Section 8 presents the silver version of IDRISI-RA. Sections 9-10 discuss a few use cases for utilizing IDRISI-RA and the ethical considerations. We finally conclude and list a few future directions in Section 11.

heading: 2 Related Work
text: In this section, we review the Arabic Twitter NER and LMR datasets. We present their characteristics and issues and discuss how IDRISI-RA dataset overcomes these limitations. In Table 1, we summarize the existing NER and LMR datasets.

heading: 2.1 Twitter NER Datasets
text: An intuitive solution to tackle the LMR task is to use general-purpose NER models for LM detection or train their LMR models using Twitter NER datasets. For example, Chen et al. (2019) built a multilingual LMR system that employs a translation module at its core to process Arabic documents. The study focuses on detecting the location mentions in newswire, broadcast news and conversations, and conversational telephone speeches.
Although the Arabic NER datasets could be sufficient for training acceptable LMR models at the onset of disaster events, several challenges are associated with this line of research. Despite the fact that the NER models trained on web data perform poorly on tweets (Darwish and Gao, 2014), the Arabic NER studies have a limited focus on Twitter. While this requires creating domainspecific datasets, existing public Arabic Twitter NER datasets suffer from the limited size, domain, and geographic coverage (Darwish, 2013; Aguilar et al., 2018; Jarrar et al., 2022). Darwish and Gao (2014) introduced the first Arabic Twitter NER dataset that contains 5,069 tweets and 1,300 LMs, 299 of which are unique. Aguilar et al. (2018) created a dataset that constitutes 12,334 tweets and 5,306 LMs. However, only the training and development sets are public to the research community. As of Jan 2023, we managed to crawl 11,155 tweets containing 1,412 LMs, 440 of which are unique. Recently, Jarrar et al. (2022) created the first Arabic Multi-domain nested NER dataset. It contains a subset of 5,653 tweets containing 435 entities of types LOC and GPE, 175 of which are unique.
A major challenge in processing Arabic documents is handling the dialectical (colloquial) text
commonly used over Twitter. Although the MSAEGY (Aguilar et al., 2018) dataset is of reasonable size, it is limited to only MSA and Egyptian dialects, suffering from its limited geographical coverage. Other datasets do not report their dialectical distributions.
Furthermore, the Arabic NER datasets are randomly filtered using the sampling Twitter API; they are not disaster-specific datasets. These datasets could serve at the onset of disaster events for deploying acceptable LMR models but should be augmented with disaster-specific Twitter data for developing robust LMR models (Suwaileh et al., 2022).
IDRISI-RA addresses these limitations by being an event-centric dataset that geographically covers all Arab countries and reasonably represents their dialects.

heading: 2.2 Twitter LMR Twitter
text: Alkouz and Al Aghbari (2018) adopted an English LMR system (Malmasi and Dras, 2015) to extract LMs from English and Arabic traffic-related tweets (filtered using traffic keywords such as “traffic” and “jam”). The system issues the n-grams extracted from the tweet text against Google Place API and assigns the latitude and longitude coordinates to n-grams that obtained results from the API. The resultant data, however, is geographically limited to United Arab Emirates (UAE). There are a couple of other cross-lingual traffic monitoring systems for English and Arabic languages (Al Emadi et al., 2017; Alkouz and Al Aghbari, 2020). However, these datasets are not public and limited in size as they contain around 500-600 tweets. Bahnasy et al. (2020) employed LM extraction to aid event
detection over Arabic tweets. Although the dataset is large in size, its geographical coverage is limited to Egypt, dialectical coverage is limited to Egyptian dialect, and its disaster domain is limited to fire, flood, and pandemic disaster types only. In contrast, IDRISI-RA dataset contains other disaster events that represent the most happening disaster types in Arabic-speaking countries. These events happened in 22 different countries. IDRISI-RA also captures a good coverage of Arabic dialects.

heading: 3 Problem Overview
text: In this work, we focus on the Location Mention Recognition (LMR) task that aims to automatically recongnize and extract toponyms (places or location names) from text. To distinguish the LMR from other tasks, we emphasize that the LMR task aims at removing geo/non-geo ambiguity of tokens in text. It is also known as location extraction or geoparsing in the literature. Differently, the Location Mention Disambiguation (LMD), which is a consecutive task for LMR, aims at removing geo/geo ambiguity between candidate LMs extracted by LMR systems. The LMD task is also known as location resolution, location linking (looking up a geo-positioning database), or geocoding (assigning geo-coordinates to LMs) in the literature.
There are two types of LMR tasks. The first type recognizes toponyms (e.g., Paris, New York) without their types (e.g., city, state) and is denoted as “type-less LMR”. The second type recognizes toponyms and also distinguishes between location types (e.g., country, city, and street) and is denoted as “type-based LMR” (Suwaileh et al., 2023). The latter better serves the development and evaluation
of geolocation processing systems in light of the responders’ needs. It enables a variety of downstream tasks (e.g., crisis maps) at different location granularity, in addition to being crucial for accurately disambiguating the toponyms.
The LMR problem is formally defined as follows: given a tweet t, the LMR system aims to identify all location mentions Lt = {li; i ∈ [1, nt]} in the tweet t, where li is the ith location mention and nt is the total number of location mentions in t, if any. Each location mention may span one or more tokens. In this work, we follow the BILOU annotation scheme (bilou_tag) with 5 classes: “B” denotes the beginning token of an LM, “I” denotes a token inside an LM, “L” denotes the last token in an LM, “O" denotes a token outside of an LM, and “U” indicates that the LM has only one token such as “Doha". Therefore, we define the LMR as a multi-class classification task on the token level.
A type-less LMR model predicts whether a token is part of a location mention (<bilou_tag>-LOC) or not (“O”), while a type-based LMR model predicts whether a token is part of LM of specific type (<bilou_tag>-<location_type>) or not (“O”). Where <location_type> is one of the 9 types presented in Section 4.2.

heading: 4 Dataset Construction
text: This section discusses design objectives, data selection, and data annotation details.

heading: 4.1 Design Objectives and Data Selection
text: During the development of our Arabic LMR dataset, we established five key design objectives: (1) to expand geographical coverage by incorporating as many Arab countries as possible, (2) to incorporate a variety of recurrent disaster types specific to the Arab world, (3) to ensure a balance of location granularity (i.e., cities, districts), (4) to ensure broad temporal coverage, and (5) to increase relevance to disaster response and management tasks by discerning informative content, as social media event streams are often noisy. We analyzed the existing disaster-related Twitter datasets in Arabic and selected Kawarith (Alharbi and Lee, 2021), as it contains tweets from 22 disaster events from the Arab world. We selected tweets (ids) from seven events (listed in Table 2) labeled as relevant for humanitarian purposes. The selected tweet ids (6,182) are used to download full tweet content using the Twitter API, which resulted in 4,593 tweets.

heading: 4.2 Dataset Annotation
text: We perform two types of annotation on the selected data. The first involves human annotators identifying toponyms, such as geographical names of places, within the tweet text. In the second, the annotators assign location types to the identified toponyms. These location types include country, province, city/town, district, neighborhood, road/street, natural points of interest such as rivers and seas, and human-made points of interest such as schools and hospitals. Toponyms that do not belong to the defined location types (e.g., islands, villages, camps) are assigned the “other location" label.
Seven graduate-level students were trained4 to carry out the annotation task, voluntarily without any monetary or course credit benefits, using the WebAnno NLP annotation tool5.
We selected the WebAnno tool as it supports Unicode right-to-left languages (e.g., Arabic). To ensure the quality of annotations, we selected the annotators to be either a citizen or having a good familiarity with the country of the disaster event. All annotators had to pass a quiz of 20 tweets before being eligible to start the annotation task.
Disagreements between annotators were examined by an additional meta-annotator and resolved. In Table 2, columns “# LMs (unique)”, we show the total number of annotated LMs and the unique number of LMs after de-duplicating them per event in parentheses. The unique number of LMs varies according to the granularity of the affected area. On average, 26% of the LMs are unique. We further report the “Hapax” per data split.

heading: 5 Dataset Description and Quality
text: In this section, we evaluate IDRISI-RA datasets for reliability, diversity, and coverage.

heading: 5.1 Reliability
text: To evaluate the quality of the dataset, we compute the Inter-annotator Agreement (IAA) that quantifies the reliability of annotations. We compute Cohen’s Kappa (Cohen, 1960) for both annotation tasks separately and jointly. Results in Figure 1, show the average reliability achieved is 83% (almost perfect), 67% (substantial), 70% (substantial), for LOC (i.e., toponym identification task), TYPE
4We share the annotation guidelines publicly in the GitHub repository: https://github.com/rsuwaileh/IDRISI/
5https://webanno.github.io/
(i.e., location type assignment), and LOC+TYPE, respectively. All events show high-quality annotations, except the “Hafr Floods 2019" event with 12% agreement for the TYPE task (slight reliability) and 44% for LOC+TYPE (moderate reliability). Upon investigation, we found that “Hafr Albatten" is the most frequent LM in the dataset; one annotator assigns “city" type for all occurrences, and the other assigns “province". While both annotators are correct (as in the Arab world, both types are used interchangeably), we anticipate the agreement level to increase when accepting both types. Furthermore, the COVID-19 event shows slight agreement for the TYPE task due to similar reasons across Arab countries.

heading: 5.2 Coverage and Diversity
text: In this section, we discuss how IDRISI-RA satisfies the design objectives presented in Section 4.1.
Geographical Coverage: IDRISI-RA covers 5 distinct Arab countries, namely Jordan, Kuwait, Egypt, Saudi Arabia, Lebanon. Additionally, the whole Arab world is represented by COVID-19 pandemic event. Figure 2a shows the distribution of distinct LMs per location type.
Domain Coverage: IDRISI-RA represents the most happening disaster types in the Arab world that are discussed over Twitter (Alabbas et al., 2017; Alharbi and Lee, 2019; Ameen et al., 2020; Alharbi and Lee, 2021), including 3 floods, 2 explosions, 1 storm, and the global COVID-19 pandemic. Location Types Coverage: Figure 2b shows the distribution of the location types. It is evident that the coarse-grained (e.g., Country, State, and City) LMs dominate the dataset due to Kawarith collection strategy that depends on tracking relevant keywords, mostly hashtags which are the names of the coarse affected areas by the disaster event.
Temporal Coverage: IDRISI-RA covers recent disaster events happened between 2018-2020. The time span for events is approximately 8.8 days, on average (refer to Table 6 in Appendix A). In Figure 5 in Appendix A, we depict the number of tweets during two events showing the coverage over important developments. Dialectical Distribution: To analyze the distribution of dialects vs. MSA in IDRISI-RA, we employed the ASAD dialectical classifier (Hassan et al., 2021). We found around 86.8% of the tweets in the dataset are MSA. Table 3 shows the dialectical distribution of around 13% tweets. The largest portion goes to the Egyptian dialect as Cairo BMB 2019 and Dragon STR 2020 happened in Egypt. The next dialect is Kuwaiti as Kuwait FLD 2018 event contains the second top number of tweets. Qatari and Saudi dialects are very close to the Kuwaiti dialect which explains their prevalence.

heading: 6 Benchmarking Experiments
text: In this section, we discuss IDRISI-RA benchmarking experiments that we conducted to provide baselines for the research community.

heading: 6.1 Experimental Setup
text: We benchmark IDRISI-RA datasets under different tasks, data, and domain setups. The LMR task setups are type-less and type-based. The data setups are (i) random, where we ignore tweets posting timestamp, and (ii) time-based, where tweets are chronologically ordered. Tweets are randomly shuffled and split into 70% training, 10% development, and 20% test sets, per event. We report the detailed stats in Table 6 in Appendix A.

heading: 6.2 Learning Models
text: We employ one deep learning-based and three traditional machine learning models for benchmarking, as described below.
6en.wikipedia.org/wiki/ISO_3166-1_alpha-2
• CAMeLBERT-Mix (CML) (Inoue et al., 2021): An NER model that is trained on ANERcorp dataset, including MSA, Dialectal Arabic (DA), and Classical Arabic (CA) data. • Farasa (FRS) (Abdelali et al., 2016): A commonly used NER model in Farasa Arabic tool. • Conditional Random Fields (CRF) (Lafferty et al., 2001): We employ CRF due to its competitive performance in the NER task. We used word syntactic features, including the suffix, POS tag, and the context (adjacent words and their syntactical features). • BERT-based (BRT) (Abdul-Mageed et al., 2020): We selected MARBERT model for its superiority in Arabic Twitter NER when used for embeddings (Benali et al., 2022).
All these scientific artifacts are used according to their terms and conditions for research purposes.

heading: 6.3 Hyperparameter Tuning
text: During training, we tuned the hyperparameters of the BERT-based model including the sequence length, the batch size, the number of training epochs, and the learning rate in light of the recommended values by (Devlin et al., 2019). For the CRF-based models, we experimented with five available training algorithms and their hyperparameters. We report the full details on hyperparameters tuning in Appendix B.

heading: 6.4 Evaluation Measures
text: To evaluate the LMR models, we compute the harmonic mean (F1 score) of Precision (P) and Recall (R). We extended the seqeval (v1.2.2) package 7 to evaluate the models per tweet and report the average performance, and reward the models when correctly predicting no LMs for a tweet.

heading: 6.5 Results
text: Type-less LMR: Table 4 presents the F1 results of all type-less models. The detailed results including precision and recall are in Appendix B. MARBERT-based model (BRT) achieves the best performance for both random and time-based scenarios. Next in order are the CRF, FARASA (FRS), and CAMeLBERT-Mix (CML). Although the CAMeLBERT-Mix is considered a BERT-based model, it shows poor performance compared to MARBERT-based model, as it was fine-tuned on
7https://github.com/chakki-works/seqeval
news wire documents for the NER (entities include LOC, ORG, PER, and MISC) task. Type-based LMR: Results in Table 4 show that the MARBERT-based LMR is evidently the best model for the random data setup. We anticipate the reason behind the lower performance of the CRF model to be the limited features used to train the Arabic version (refer to Section 6.2). The CRF model exhibits comparable F1 scores to the MARBERTbased model in the time-based data setup. To answer RQ1, we confirm the need for specific-LMR datasets and models that can perform effectively over disaster tweets.

heading: 7 Dataset Generalizability
text: In this section, we empirically study the generalizability of IDRISI-RA dataset. For that, we employ the best LMR model, MARBERT-based (refer to Section 6); hereafter, we refer to it as “the model”. We study three dimensions: (i) generalizability to unseen events regardless of their type and geolocation, (ii) generalizability to unseen events of the same or different disaster types (domain generalizability), and (iii) generalizability to unseen events that happened in the same or different countries (geographical generalizability).

heading: 7.1 Experimental Setups
text: We run our experiments under both type-less and type-based task setups for only random data setup. We tune the hyperparameters of the model for every setup (refer to Section 6.3). We define the source dataset as the dataset (or the combination of datasets) used to train the model, and the target dataset as the dataset used to test it. Domain generalizability: We examine the model’s performance under cross- and in-domain transfer setups (Suwaileh et al., 2020). The “domain” in our experiments refers to the type of disaster event. IDRISI-RA dataset covers the four most occurring disaster types in the Arab world: flood, bombing, storm, and pandemic. A transfer data setup is composed of source-target pair, resulting in 16 runs. Geographical generalizability: We examine the model’s performance over events that took place in different countries than the source dataset. IDRISIRA covers five countries (refer to Table 6 in Appendix A), besides the global COVID-19. A transfer data setup is composed of source-target pair, resulting in 42 runs after excluding the target runs.

heading: 7.2 Results
text: Generalizability to unseen events: Table 5 shows the results of the model both with and without (i.e., zero-shot) the target event. The results for the typeless LMR demonstrate the potential of IDRISI-RA dataset under the zero-shot setting. The difference against the target runs is mostly negligible. Due to the difficulty of the type-based LMR, the performance under zero-shot learning is significantly lower than the target runs. However, the zero-shot results are still within a reasonable range (i.e., average F1 0.88), which demonstrates the effectiveness of models trained on IDRISI-RA. To answer RQ2, we confirm that training on IDRISI-RA generates generalizable Arabic LMR models that achieve, on average, around 0.75 and 0.88 F1 scores for the type-less and type-based LMR, respectively. Domain generalizability: Figure 3 illustrates the F1 scores of the models over the target sets.
In-Domain: Ideally, the best results should lay on the diagonal which depicts the in-domain setup. This assumption holds for all runs, except the STR-to-STR runs in the type-based LMR (Figure 3.b). Training on “bombing" data in the BMBto-STR setup achieves comparable results to training on “storm" data in the STR-to-STR, because both source and target data share the same or close affected areas (Egypt and Lebanon), which could imply the overlap of toponyms’ occurrences and patterns. The “bombing" (BMB) includes data from the Cairo Bombing 2019 in Egypt and the Beirut Explosion 2020 in Lebanon. The “storm" (STR) test data contains Dragon storms 2020 that affected Egypt and Jordan, among a few LMs from Levantine Arabic. Moreover, the FLD-to-STR run achieves 6.3% better performance compared to the STR-to-STR run, as the FLD source data is approximately 7.5 times larger in size than the “storm" STR source. The effect of training dataset size
on these results could be confirmed by the relatively low F1 scores when the model trained on the “storm" data that has the smallest training data. Cross-Domain: Generally, the right upper part above the diagonal shows better results than the counterpart, except for the BMB-to-FLD where the size of training data influences the results. We also note here that the model is tuned for every sourceto-target transfer setup over the development splits, hence, the poor results on the test splits could indicate overfitting that prevents generalizability. This
motivates the use of more advanced transfer learning techniques. To answer RQ3, we confirm that IDRISI-RA can generate acceptable domain generalizable models for the most disaster types. It also provides challenging examples for the LMR models.
Geographical generalizability: Figure 4 shows the F1 scores of the models over the target countries that are the same or different than the affected area of the source data. The model achieves approximately 0.61 and 0.84 F1 scores, on average, for type-less and type-based LMR, respectively. The top performance is achieved over “GL” target data that refers to the COVID-19 event due to its geographical coverage. To answer RQ4, we found that IDRISI-RA can generate reasonable geographically generalizable models.

heading: 8 Silver IDRISI-RA
text: Given the dearth of Arabic LMR datasets, we expand the size of IDRISI-RA dataset by employing the best-performing LMR model to infer labels for 1.2 million Arabic tweets posted during 22 disaster events. We call the resulting data as the silver version to indicate the reliability of annotations. The inference model is trained using the entire type-based and type-less train sets from all events. All development partitions are used for tuning the hyperparameters of the MARBERT-based model. The model applied to the tweets resulted in 884,217 LMs (18,609 distinct). We note that this large dataset can potentially be useful for a variety of LMR approaches, e.g., domain adaptation, transfer learning, and semi-supervised learning.

heading: EG & JO 0.48 0.39 0.40 0.38 0.59 0.82 0.51

heading: EG & JO 0.81 0.88 0.65 0.73 0.85 0.94 0.81

heading: 9 Research Use Cases
text: Releasing IDRISI-RA empowers research on different applications, other than the geolocation. In this sections, we discuss a few use cases.
Event/incident detection: People tend to mention where events/incidents take place when they report them (Hu and Wang, 2020). Following (Sankaranarayanan et al., 2009) and (Watanabe et al., 2011), IDRISI-RA can be employed to detect event/incident on Twitter by extracting LMs.
Relevance filtering: Prior studies show that the geographical references in social media messages could indicate their relevance and informativeness for a disaster event (Vieweg et al., 2010; De Albuquerque et al., 2015). Therefore, IDRISI-RA can be utilized to train relevance filtering models.
Displacement monitoring: Extracting LMs from tweets shared by displaced people and refugees can help predicting their future paths. In fact, monitor the population movement during emergencies is very useful for responders. Thus, IDRISI-RA can be exploited in modeling the population movement.
Geographical retrieval: IDRISI-RA could serve the GIR retrieval techniques that rely on detecting locations and spatial references in queries and documents (García-Cumbreras et al., 2009).

heading: 10 Ethical Considerations
text: While Twitter allows users to self-manage their privacy by disabling the geo-tagging functionalities, “even well informed and rational individuals cannot appropriately self-manage their privacy” (Solove, 2012) due to lack of awareness on how such data can be collected or commercially used. However, geolocation extraction could be justified in the context of social good during natural disaster events where protecting geographical privacy could be of least importance as affected people need to rescue their lives or get basic necessities of life (Crawford and Finn, 2015). The privacy of affected people during human-made disaster events is more critical because revealing their locations during conflicts and wars could risk their lives. Therefore, we limit our dataset to natural disaster events and COVID19 pandemic and apply a couple of de-identification steps to protect users’ privacy (refer to Section A). We further limit the dataset usage to research purposes only by releasing it under the Creative Commons Attribution 4.0 International License.8 Additionally, we emphasize that systems developed for LMR using IDRISI-RA dataset should implement proper mechanism for preserving user privacy.

heading: 11 Conclusion
text: We introduced IDRISI-RA, the first Arabic LMR Twitter dataset. It contains 22 disaster events of different types that happened in the Arab region. We manually- (gold) and automatically annotated (silver) about 4.6K and 1.2M tweets. Both versions are annotated for location mentions and location types which form the value and uniqueness of IDRISIRA. Our analysis showed that IDRISI-RA is second to none in empowering research for Arabic LMR. Additionally, the extensive experiments emphasize the need for developing LMR-specific models for the disaster domain. The developed LMR baselines are simple yet competitive ones. The results also demonstrated the decent generalizability of IDRISI-RA. For future work, we plan to extend the annotations for the Location Mention Disambiguation (LMD) task. We further plan to explore different transfer learning, domain adaptation, and active learning techniques to tackle the LMR task.
8https://creativecommons.org/licenses/by/4.0/ legalcode
Limitations
There are a few shortcomings that we discuss below: Underrepresented fine-grained LMs: Although we had chosen a careful sampling method focused of event-centric informative dataset aiming to increase the likelihood of fine-grained LMs’ occurrence (Kitamoto and Sagara, 2012), we think the low frequency of fine-grained LMs in IDRISI-RA is a major limitation as it contains solely 25.5% fine-gained LMs. Human errors: There are some human errors made during annotation due to the difficulty of the task.
• Annotators sometimes fail in distinguishing between Location and Organization entities (e.g., “Red Cross"). • Different location types could be used interchangeably for the same locations which forms a difficulty for annotators (refer to Section 5.1). • Annotators highlight the locations that are mentioned as descriptions within the context of the tweet.
We plan to overcome these errors as part of Location Mention Disambiguation (LMD) annotation that aim to remove ambiguity of geo/geo entities (as a sequel to the geo/non-geo LMR annotations). Temporary locations: Temporary facilities (i.e., medical camps, shelters, etc.) are constructed during emergencies to provide resources and support for the affected people. However, these facilities could be disassembled (e.q., quarantine centers) once the emergency event is over. Additionally, the names of some locations could change during emergencies. For example, allocating a specific school as a shelter and giving it a new expressive name (e.g., “main shelter”). Once the disaster event is over, the school will return to providing its original services. The difficulty of these temporary locations lies in their need for context when resolved. Although they are important for the affected people and response authorities, not all of them are labeled in IDRISI-RA. Generalizability: Due to the absence of public LMR datasets, we could not compare the generalizability of IDRISI-RA against existing LMR datasets. Hence, we study the generalizability within IDRISI-RA for domain and geographical aspects.

heading: Acknowledgements
text: This work was made possible by the Graduate Sponsorship Research Award (GSRA) #GSRA51-0527-18082 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors. We would like to thank the in-house annotators for their valuable help including Noura Abdullah, Aisha Suwaileh, Rasha Hamdoon, Najlaa Alfuhaida, Hana Shamayleh, Lamiaa Basyoni, Sara Alrasbi, and Nada Abo Eita.

heading: A Data Release
text: The IDRISI-RA dataset is released9 data setups that are random and Time-based. The location mention and location type annotations are made available for the community to enable development of type-less and type-based LMR models. The data is released in JSONL format where every lines
9This dataset is licensed under the Creative Commons Attribution 4.0 International License: https:// creativecommons.org/licenses/by/4.0/legalcode
corresponds to one tweet with the following properties: “text”, “created_at”, “info_class” adopted from Kwaraith dataset, and “location_mentions”.
We processed the data to de-identify it as follows:
In Figure 5, we depict the temporal coverage of COVID-19 and Kuwait FLD 2018 events.

heading: B Hyperparameter Tuning and Results
text: For the BERT-based models, we tuned the sequence length, the batch size, the number of training epochs, and the learning rate in light of the recommended values by (Devlin et al., 2019) as: batch size of 8, 16 or 32, number of epochs of 2, 3,
Event Arabic tweet English Translation (Google)
or 4, and learning rate of 5E-5, 3E-5, or 2E-5. We also experimented with a sequence length of 128 or 256.
For the CRF-based models, we tuned five algorithms, namely Gradient Descent using the LBFGS method (LBFGS), Stochastic Gradient Descent with L2 regularization term (L2EG), Averaged Perceptron (AP), Passive Aggressive (PA), and Adaptive Regularization Of Weight Vector (AROW). For LBFGS, we tuned the coefficients for L1 and L2 regularization parameters. For the L2EG, we tuned the the coefficient for L2 regularization and the initial value of learning rate used for calibration. For AP, we tuned the epsilon parameter that determines the condition of convergence. For PA, we tuned the strategy for updating feature weights and the sensitivity parameter that determine whether errors are considered in the objective function. For AROW, we tuned the initial variance of every feature weight and the tradeoff between loss function and changes of feature weights (gamma). The regularization parameters are tuned for values between 0.05 and 1 with step value of 0.05. The initial learning rate and epsilon are tuned using values {1 × 10i|i ∈ [2, 6]}. The PA sensitivity parameter is boolean and the updating strategy includes three types: without slack variables, type I, or type II. The variance and gamma parameters of AROW algorithm are tuned for values {2−i|i ∈ [0, 3]}.
We ran our hyperparameter tuning experiments for about 3 days on a cluster of 46 GPUs of different NVIDIA models including p100, v100, v100NVLINK, and T4.
Tables 7 and 8 show the best hyper-parameters and detailed results for the CRF and BERT-based LMR models for IDRISI-RA, respectively. Tables 9 shows the best hyper-parameters of the models under disaster domain transfer setting.
Tw ee
ts L
M s
E ve
nt C
ou nt
ry Ti
m e
Pe ri
od T
R N
D E
V T
ST A
ll L
M 0
T R
N D
E V
T ST
A ll
U ni q R an do m Jo rd an FL D 20 18 JO 10 /2 5 -1 1/ 18 37 1 53 10 3 52 7 10 7 61 4 89 19 4 89 7 10 8 K uw ai tF L D 20 18 K W 11 /0 4 -1 1/ 18 88 9 12 7 25 3 1, 26 9 50 3 82 0 93 22 4 1, 13 7 14 0 C ai ro B M B 20 19 E G 08 /0 4 -0 8/ 04 18 9 27 52 26 8 1 41 7 66 14 0 62 3 24 H af rF L D 20 19 SA 10 /2 5 -1 0/ 29 36 4 52 98 51 4 46 52 0 83 14 9 75 2 11 2 D ra go n ST R 20 20 E G & JO a 03 /1 1 -0 3/ 15 21 7 31 57 30 5 12 2 25 2 27 59 33 8 16 0 B ei ru tB M B 20 20 L B 08 /0 4 -0 8/ 07 24 5 35 69 34 9 63 37 8 49 12 3 55 0 61 C oV ID -1 9 G lo ba l 05 /1 7 -– /– b 95 9 13 7 26 5 1, 36 1 77 7 63 7 96 20 6 93 9 31 3 Ti m eba se d Jo rd an FL D 20 18 JO 10 /2 5 -1 1/ 18 37 1 53 10 3 52 7 10 7 63 1 79 18 7 89 7 10 8 K uw ai tF L D 20 18 K W 04 /1 1 -0 4/ 18 88 9 12 7 25 3 12 69 50 3 77 2 11 2 25 3 1, 13 7 14 0 C ai ro B M B 20 19 E G 08 /0 4 -0 8/ 04 18 9 27 52 26 8 1 45 8 53 11 2 62 3 24 H af rF L D 20 19 SA 10 /2 5 -1 0/ 29 36 4 52 98 51 4 46 52 6 79 14 7 75 2 11 2 D ra go n ST R 20 20 E G & JO a 03 /1 1 -0 3/ 15 21 7 31 57 30 5 12 2 24 8 39 51 33 8 16 0 B ei ru tB M B 20 20 L B 08 /0 4 -0 8/ 07 24 5 35 69 34 9 63 40 0 55 95 55 0 61 C oV ID -1 9 G lo ba l 05 /1 7 -– /– b 95 9 13 7 26 5 13 61 77 7 59 9 10 9 23 1 93 9 31 3 A ll - - 3, 23 4 46 2 89 7 4, 59 3 1, 61 9 3, 63 4 52 6 1, 07 6 5, 23 6 91 8
Ta bl
e 6:
D et
ai le
d in
fo rm
at io
n an
d st
at is
tic s
of ID
R IS
IR
A da
ta se
ts ,b
ot h
ra nd
om an
d tim
eba
se d
se tu
ps .“
T R
N ”,
“D E
V ”,
“T ST
” re
fe rt
o th
e tr
ai ni
ng ,d
ev el
op m
en t,
an d
te st
sp lit s, re sp ec tiv el y. L M 0 re fe rs to th e nu m be ro ft w ee ts w ith no L M s. a W hi le D ra go n st or m s ha ve af fe ct ed se ve ra lA ra b co un tri es an d LM s ar e fr om di ff er en tc ou nt rie s, K aw ar ith cr ea to rs ha d in te nt io na lly fo cu se d on th e Eg yp tia n tw ee ts .W e fo un d LM s fr om Jo rd an as w el l. b T he la st tw ee ti n th e ch ro no lo gi ca lly so rt ed tw ee ts w as pu bl is he d in 20 20 /0 6/ 15 bu tt he pa nd em ic w as on go in g.
R an
do m
Ti m
eba
se d
E ve
nt e
bs lr
sl P
R F1
e bs
lr sl
P R
F1
Ty pe
-le ss
Jo rd
an Fl
oo ds
3 8
3e -5
25 6
0. 95
4 0.
95 7
0. 95
3 3
8 3e
-5 12
8 0.
91 1
0. 91
6 0.
90 3
K uw
ai tF
lo od
s 4
16 3e
-5 25
6 0.
93 5
0. 92
5 0.
92 8
3 16
3e -5
12 8
0. 89
5 0.
90 5
0. 89 3 C ai ro B om bi ng 3 8 3e -5 25 6 0. 99 5 0. 98 6 0. 98 9 2 8 3e -5 12 8 0. 93 4 0. 93 9 0. 93 6 H af rF lo od s 4 8 3e -5 12 8 0. 88 3 0. 88 3 0. 87 9 4 8 3e -5 25 6 0. 87 8 0. 89 7 0. 87 8 D ra go n St or m s 3 8 3e -5 12 8 0. 87 8 0. 87 3 0. 87 0 4 8 4e -5 12 8 0. 88 2 0. 86 8 0. 86 9 B ei ru tE xp lo si on 4 8 3e -5 12 8 0. 88 5 0. 85 1 0. 85 5 4 8 3e -5 25 6 0. 60 1 0. 61 1 0. 58 2 C oV ID -1 9 3 8 3e -5 12 8 0. 88 9 0. 88 4 0. 88 1 3 8 3e -5 25 6 0. 89 6 0. 91 4 0. 89 7
Ty pe
-b as ed Jo rd an Fl oo
ds 4
8 3e
-5 12
8 0.
91 6
0. 90
7 0.
90 8
4 8
3e -5
25 6
0. 88
0 0.
87 2
0. 86 2 K uw ai tF lo od s 4 8 3e -5 12 8 0. 93 3 0. 92 5 0. 92 5 3 8 3e -5 25 6 0. 87 4 0. 89 2 0. 87 9 C ai ro B om bi ng 4 8 3e -5 12 8 0. 98 4 0. 97 0 0. 97 5 4 8 3e -5 25 6 0. 93 0 0. 93 5 0. 93 1 H af rF lo od s 4 8 3e -5 25 6 0. 87 0 0. 85 7 0. 85 6 3 8 3e -5 25 6 0. 84 1 0. 85 4 0. 83 8 D ra go n St or m s 4 8 3e -5 25 6 0. 79 8 0. 78 9 0. 78 7 4 8 3e -5 25 6 0. 72 6 0. 72 2 0. 71 4 B ei ru tE xp lo si on 4 8 4e -5 25 6 0. 85 4 0. 82 1 0. 81 3 4 8 5e -5 12 8 0. 61 6 0. 63 5 0. 59 6 C oV ID -1 9 4 8 3e -5 25 6 0. 89 5 0. 89 8 0. 89 3 4 8 3e -5 12 8 0. 88 8 0. 89 8 0. 88 6
Ta bl
e 8:
T he
be st
hy pe
rpa
ra m
et er
s an
d re
su lts
fo rt
he B
E R
Tba
se d
m od
el ov
er ID
R IS
IR
A .
e, bs
,l r,
an d
sl re
fe rt
o th
e hy
pe r-
pa ra
m et
er s,
nu m
be ro
fe po
ch s,
ba tc
h si
ze ,l
ea rn
in g
ra te
,a nd
se qu
en ce
le ng
th ,r
es pe
ct iv
el y.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations
3 A2. Did you discuss any potential risks of your work? Discussed two main risks: - Noisiness of SM data in Section 3.1 Design Objectives and Data Selection - Revealing individuals’ (potentially vulnerable ones) information in Appendix A Data Release
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3 Dataset Construction 5.2 Learning Models
3 B1. Did you cite the creators of artifacts you used? 5.2 Learning Models
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 5.2 Learning Models Appendix A Data Release
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 5.2 Learning Models
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Appendix A Data Release
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 4 Dataset Description and Quality
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3 Dataset Construction Appendix A Data Release
C 3 Did you run computational experiments? 5 Benchmarking Experiments 6 Dataset Generalizability
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 5.3 Hyperparameter Tuning Appendix B Hyperparameters Tuning and Results
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 5.1 Experimental Setup 5.3 Hyperparameter Tuning 6.1 Experimental Setups Appendix B Hyperparameters Tuning and Results
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5.5 Results 6.2 Results Appendix B Hyperparameters Tuning and Results We note that, as our aim in this paper is to provide baselines for the community and not to develop models that advance the SOTA, we extensively tune hyperparameters without repeating experiments with different seeds to report an anggregated (e.g., avg) results.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5.2 Learning Models 5.4 Evaluation Measures We use default models and parameters
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3.2 Dataset Annotation
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? 3.2 Dataset Annotation
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 3.2 Dataset Annotation
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? The annotators are volunteering students and were informed that their annotations will be used for research purposes in the humanitarian domain.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. We collect, annotate, and share the data under Developer Agreement and Policy. No approval from other ethics review board.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? 3.2 Dataset Annotation

heading: Future Work
text: For future work, we plan to extend the annotations for the Location Mention Disambiguation (LMD) task. We further plan to explore different transfer learning, domain adaptation, and active learning techniques to tackle the LMR task. 8https://creativecommons. org/licenses/by/4. 0/ legalcode
Limitations
There are a few shortcomings that we discuss below: Underrepresented fine-grained LMs: Although we had chosen a careful sampling method focused of event-centric informative dataset aiming to increase the likelihood of fine-grained LMs’ occurrence (Kitamoto and Sagara, 2012), we think the low frequency of fine-grained LMs in IDRISI-RA is a major limitation as it contains solely 25. 5% fine-gained LMs. Human errors: There are some human errors made during annotation due to the difficulty of the task. • Annotators sometimes fail in distinguishing between Location and Organization entities (e. g. , “Red Cross"). • Different location types could be used interchangeably for the same locations which forms a difficulty for annotators (refer to Section 5. 1). • Annotators highlight the locations that are mentioned as descriptions within the context of the tweet. We plan to overcome these errors as part of Location Mention Disambiguation (LMD) annotation that aim to remove ambiguity of geo/geo entities (as a sequel to the geo/non-geo LMR annotations). Temporary locations: Temporary facilities (i. e. , medical camps, shelters, etc. ) are constructed during emergencies to provide resources and support for the affected people. However, these facilities could be disassembled (e. q. , quarantine centers) once the emergency event is over. Additionally, the names of some locations could change during emergencies. For example, allocating a specific school as a shelter and giving it a new expressive name (e. g. , “main shelter”). Once the disaster event is over, the school will return to providing its original services. The difficulty of these temporary locations lies in their need for context when resolved. Although they are important for the affected people and response authorities, not all of them are labeled in IDRISI-RA. Generalizability: Due to the absence of public LMR datasets, we could not compare the generalizability of IDRISI-RA against existing LMR datasets. Hence, we study the generalizability within IDRISI-RA for domain and geographical aspects