dataesr/scientific-paragraphs-categorization huggingface dataset:
Crises such as natural disasters and public health emergencies generate vast amounts of text data, making it challenging to classify the information into relevant categories. Acquiring expert-labeled data for such scenarios can be difficult, leading to limited training datasets for text classification by fine-tuning BERT-like models. Unfortunately, traditional data augmentation techniques only slightly improve F1-scores. How can data augmentation be used to obtain better results in this applied domain? In this paper, using neural network explicability methods, we aim to highlight that fine-tuned BERT-like models on crisis corpora give too much importance to spatial information to make their predictions. This overfitting of spatial information limits their ability to generalize especially when the event which occurs in a place has evolved and changed since the training dataset has been built.
To reduce this bias, we propose GeoNLPlify, 1 a novel data augmentation technique that leverages spatial information to generate new labeled data for text classification related to crises. Our approach aims to address overfitting without necessitating modifications to the underlying model architecture, distinguishing it from other prevalent methods employed to combat overfitting. Our results show that GeoNLPlify significantly improves F1-scores, demonstrating the potential of the spatial information for data augmentation for crisis-related text classification tasks. In order to evaluate the contribution of our method, GeoNLPlify is applied to three public datasets (PADI-web, CrisisNLP and SST2) and compared with classical natural language processing data augmentations.
The problem we face is the limited labeled data available. In particular, for health crises, the rarity and nonsimilarity of events are important  [4]. Even by applying adaptation methods to the crisis domain  [5] , no satisfactory solution exists. At the same time, as NLP has been revolutionized by the rise of language models (LMs), this study aims to enhance the performance of these methods for crisis-related situations. These models, based on attention mechanisms  [6] , have given pride to very large datasets. Even if LMs are intended to be used in transfer learning downstream tasks using a smaller corpus, they still need a sizeable dataset  [7]. How can these major advancements be beneficial when working on a small corpus? Borrowed from the computer vision field, different data augmentation techniques have been developed in NLP  [8].
Indeed, for example, if its training dataset a model learned that Pakistan is associated with flood, it may misclassify other events occurring in Pakistan. Our hypothesis  [11]  is that BERT-like fine-tuned models overfit on spatial information in this context. Among all the methods to reduce overfitting, only data augmentation does not change the model architecture. Indeed, different regularization methods exist such as dropout (randomly deactivates some neurons) or L2 (reduces the weights of neurons when these are too high compared to the others). The model we use in this paper, RoBERTA, already includes these regularization mechanisms, leading us to focus on data augmentation. To do this, we aim to create variations of the original documents by replacing the locations mentioned in texts.
This helps to expand the model's exposure to different variations of geographical entity names, reducing the overfitting on pairs of places and labels encountered in the training dataset and increasing the model's generalizability to new examples. By employing this approach, the model is guided to redirect its attention towards the non-spatial words present in the document, thereby enhancing their significance in the overall attention mechanism. This approach can lead to an effective data augmentation with a strong quality of the artificially labeled data and reduce this geographical overfitting. Thus, we introduce GeoNLPlify, a new approach that integrates three data augmentation techniques based on the spatial information contained in texts. GeoNLPlify has a positive impact on the performance of LM classifiers.
GeoNLPlify detects location mentions with Spacy and then geocodes them using OpenStreetMap data. Finally, GeoNLPlify proposes to replace these places with others (either at the same spatial level or by zooming in or out) in order to create artificially labeled documents. To evaluate the GeoNLPlify impact on the spatial overfitting reduction, we apply it to three datasets. The first two are directly related to crises since the first, PADI-web  [12] , 2  deals with epidemiological crises in animal health and the second, crisisNLP  [13] ,  3  with natural disasters. The last one, SST2  [14] ,  4 does not focus on disasters at all but on sentiment analysis in film reviews. It, therefore, allows us to evaluate the contribution of GeoNLPlify when spatial overfitting is not observed. For all these datasets, we compare GeoNLPlify with two state-of-the-art NLP data augmentation provided by nlpaug  [15]  by substituting words with (i) WordNet (nlpaug_synonym)  [16]  and (ii) contextual embedding (nl-paug_contextual_embedding)  [17].
To sum up, this article proposes two main contributions: (i) We demonstrate that BERT-like models overfit on spatial information in the context of crises and (ii) Our method, GeoNPLlify, is effective to reduce this bias. In the following of this paper, we will provide an overview of related work in the field of data augmentation in Section 2. Our proposed methods, detailed in Section 3, will demonstrate that BERT-like models overfit on spatial information and the using of GeoNLPlify reduces this bias. Section 4 will describe, in details, the experimental framework and provide the results of the benchmark comparison of several data augmentations on three public datasets. In the Discussion Section 5, we will interpret the results to better understand why GeoNLPlify performs so well on crisis-related corpora, but also to identify if there are new potential sources of bias when using our method.
Related work The natural language processing (NLP) domain has been significantly improved by the emergence of language models (LMs). Several models have been trained and made available, such as BERT  [18]  and RoBERTa  [19]. The strength of these approaches, in addition to the quality of their deep learning architecture, relies on pre-training, which requires very large datasets. These models are nonspecific and can be specialized or fine-tuned to a particular domain, such as crisis management  [2] , which is of interest to us in this study. Unfortunately, this field struggles with a lack of data  [20] , also called a low data regime (< 10k training data points)  [9]. To overcome this issue, manually labeled datasets must be enlarged. The first way is to ask experts to manually label new data.
Method The crisis management field struggles with a lack of data  [4, 5] , so our main objective is to increase the training dataset since it is not enough to let BERT-like models generalize on new data. To highlight this issue, we introduce our training pipeline as illustrated by Fig. 1. The data used for fine-tuning RoBERTa  [19]  is PADI-web  [12] , a corpus of expert annotated documents related to animal health surveillance. Labels corresponding to a crisis stage are assigned to press media articles. For example, the article called "Bird flu cases confirmed at Nottingham's Colwick Country Park as public warned" (published by Nottingham Post)  5  has been annotated as an outbreak declaration, whereas "The decline in the supply of chicks penalizes the poultry sector" (published by El Watan)  6  received the label Consequences.
Indeed, outbreak declarations often report several cases. On the other hand, how does the geographical information "Nepal" (3rd contributor) help the model to classify the document as an "outbreak declaration"? Why does its prediction also rely on this word? Our hypothesis is because our fine-tuned RoBERTa encounters, in its training dataset, several documents containing "Nepal" and manually labeled by experts as "outbreak declaration". However, we are aware that explicability or interpretability methods in LM are controversial  [33] , so we do not use it as a full explanation of our classifier, but we consider the LIME result to be a powerful tool to gain intuition about our fine-tuned model. For each data point in the annotated corpus, we extract the three most salient words of each inference.
Let D = {(d 1 , y 1 ),. , (d n , y n )} be the annotated dataset of N samples. Given D, a multiclass classifier h : d → Y learns to attribute a crisis phase (Y : preparedness, outbreak, consequences) to a document d. We use ŷ to present the prediction of classifier h. Each d i is composed of a W = {w 1 ,. , w k } where k is the number of tokens in d i. Given ŷ, a LIME model g : (ŷ, w di ) → {(w 1 , s 1 ),. , (w k , s k )} computes saliency score s i for each token w i in W. For each d ⊂ D, we extract the three most salient tokens. For each salient token, we apply name entity recognition (NER), a NLP task that classifies tokens into well-known predefined categories, such as a person names, organizations, locations or GeoPolitical entities (GPE), which are cities, states or countries. We use spaCy, 7 which provides an easy-to-use LM implementation and is often used as a baseline for studies that aim to improve the NER task  [34].
GeoNLPlify GeoNLPlify is a set of three data augmentation methods that make variations of original annotated data using spatial information to increase the training corpus size. Indeed, as highlighted in the previous section, fine-tuned BERT-like models tend to overfit on tokens carrying spatial information when working with crisis-related datasets. By artificially creating new documents by making variations on geographical entities, GeoNLPlify aims to expose the model to a lot of new different locations, which makes it better able to generalize when it encounters, in inference mode, new places or when spatial information is missing. Its attention to spatial entities decreases in favor of the other words in the document. At the same 7 https://spacy. io/. time, our empirical intuition is that the spatial level has an influence on the classification (by spatial level, we mean city, state or country level); people provide a better report of the local situation during a crisis by being at the right spatial level  [35, 36].
For example, an outbreak declaration will focus on the city or state level where the case occurs, although the consequences will be reported at a country level, as illustrated by Fig. 4. This figure shows the spatial level distribution in the training dataset, by label, of GPE tokens that contribute the most to model predictions. To evaluate this assumption, we define three strategies. The three GeoNLPlify DA methods are illustrated in Fig. 5. The first one, Generalization, duplicates annotated documents that contain GPE tokens (at a city level) and replace them by their countries. For example, the title of this new article: "2 Cases of Powassan Virus Confirmed in New Jersey" will be duplicate to "2 Cases of Powassan Virus Confirmed in United-States. Equivalence, the second method, duplicates documents by making variations at the same level of GPE tokens.
The variant is selected randomly among "world cities" from Simple Maps. 8  For example, the document "2 Cases of Powassan Virus Confirmed in Philadelphia" is duplicated by "2 Cases of Powassan Virus Confirmed in Glasgow". Finally, the last method, Specialization, duplicates documents that contain GPE at a country level by a randomly chosen city. We use, once again, the database "world cities". For example, we duplicate the document "Kenya issues alert to foot and mouth disease outbreaks" by creating "Munich issues alert to foot and mouth disease outbreaks". Munich is randomly selected from the list of cities.
Experiments In this study, we aimed to highlight the spatial overfitting of BERT-like models on crisis-related corpora and to propose a new data augmentation to reduce this bias. To achieve this, we conducted a series of experiments using two crisis-related datasets and a third not dealing with this subject. The experiments process involves saliency maps and different RoBERTa fine-tuning on several data augmentation methods in order to compare them on these datasets. The results of our experiments show GeoNLPlify is successful in reducing spatial overfitting and allows fine-tuned models with the best F1-score.
CrisisNLP CrisisNLP is a human-annotated Twitter corpora of crisis-related messages  [13]  collected during 19 different crises between 2013 and 2015. Two kinds of annotation were performed: one by volunteers and the second by paid workers. For this benchmarking, we use both to obtain 11570 tweets with a label among 14 categories (Affected people, caution and advice, deaths reports, disease signs or symptoms, disease transmission, displaced people and evacuations, donation needs, offers or volunteering services, infrastructure and utilities damage, injured or dead people, missing trapped people of found people,  not related or irrelevant, other useful information, prevention and sympathy and emotional support). The distribution is, once again, very unbalanced, as illustrated by Fig. 7. What interests us in using CrisisNLP is that it is a reference dataset in crisis management.
SST2 The Stanford Sentiment Treebank (SST) consists of sentences from movie reviews and human annotations of their sentiment (positive/negative). SST-2 is part of the general language understanding evaluation benchmark  [14] , which is a collection of resources for training, evaluating and analyzing natural language understanding systems. 9  The training dataset contains more than 67K sentences. As this study focuses on tiny corpora, we artificially reduced the number of documents by random selection to 300 and 11000 to be closer to the PADI-web and CrisisNLP corpus sizes. In using SST, the objective is to analyze how GeoNLPlify performs on a dataset when the spatiality does not matter to better evaluate the generality of our method.
Experimental process description Two experiments have been conducted. The first one, the saliency map, aims at showing the overestimation of the role of spatial information granted by fine-tuned models on a non-augmented corpus related to crises. To set up this experimentation, a pre-trained RoBERTa has been fine-tuned on PADI-web data and LIME has been applied to each prediction of our model for the whole dataset. The second experiment aims to compare GeoNLPlify against classical NLP data augmentation on three public datasets. The technical aspects of both experiments are described in the two following sections.
Saliency map To gain insight into which type of tokens a pre-trained model (such as RoBERTa) relies on to classify documents, saliency maps are computed by the Python library LimeTextExplainer,  10  which implements LIME  [10]. The three most salient tokens are extracted for each document and categorized using name entity recognition (NER) with spaCy, as shown in Fig. 3. As described in Section 3, fine-tuned model on non-augmented corpus relies on its prediction on GeoPolitical Entities (GPE) tokens. This means that the model considers certain locations as class markers. For example, if in the training dataset, an outbreak event occurs in Indonesia, then the model may classify new documents with Indonesia as a new outbreak. However, the article may well report on the end of a crisis or general information.
Applying data augmentation From the original datasets, five augmented datasets are processed. The three first come from GeoNLPLify and the two last from nlpaug. GeoNLPlify data augmentation relies on a three-step pipeline: NER, geocoding and spatial variation. First introduced in 1996  [37] , NER aims to provide a category to tokens, such as organization, person or location. Since then, multiple methodologies have been used  [38]. To simplify the implementation, our pipeline uses a spaCy algorithm. 11 The second step focuses on tokens identified by NER as geographical entities (GPEs). Using OpenStreetMap (OSM) 12 data through the photon geocoder, 13 the pipeline retrieves a token's spatial information, such as its spatial level (i. e. , city/county/state/country). To reduce the number of queries for the geocoder, the results are stored in a cache.
14  Depending on the spatial data augmentation methods, the third step uses spatial-level information to create variation. The generalization approach replaces the GPE cities by their country, while the specialization retrieves a randomly selected city for each GPE country, and finally, the spatial equivalence replaces the GPE cities. The two last methods use the Simple Maps database. 15 We provide a Python library to easily re-use GeoNLPlify. The repository is available at https://github. com/remydecoupes/GeoNLPlify. To compare the GeoNLPlify data augmentation to classical NLP data augmentation approaches, the nlpaug python library  [15]  is used. This library provides several augmentations at different levels (i. e. , character, word and sentence) through multiple approaches (contextual embedding, synonym, back translation, random variation, etc.
). Two augmentations have been used for the comparison, both at the world level (such as GeoNLPlify): (i) nlpaug_synonym: based on synonyms (using WordNet  [16] ) and (ii) nlpaug_contextual_embedding based on contextual word embedding (using the BERT model  [18] ). Multiple studies used DA at the word level  [25, 17, 26]. It first randomly selects tokens from the document and then replaces the drawn tokens with one of its variant candidates proposed by WordNet or BERT (also randomly selected). The default parameters are used, i. e. , up to 30% of words are changed in documents. As shown in Fig. 8 , the different classes of the three datasets are not equally augmented. Indeed, while nlpaug can generate variation in any sentence, GeoNLPlify only amplifies documents that contain spatial information.
Training strategy Both GeoNLPlify & nlpaug augmentations and the original datasets use the same training pipeline. A cross-validation process is used with ten folds. The folds preserve the class imbalance. RoBERTa is then fine-tuned with 3 epochs on a server with a V100 NVIDIA and 315 GB RAM using the HuggingFace python library. 16 To make the results reproducible, all random seeds (shuffling data and the last fine-tuning layers initiating weights) are fixed  [39].
Evaluation protocol The evaluation metrics are computed on the 10-fold evaluation datasets without any augmentation (they are all removed to keep only original documents). The means of the evaluation metrics (F1-score, recall and precision) on k-folds are calculated (with their standard deviation) for each data augmentation method to be compared in the next section. We propose to compare the three GeoNPLify data augmentations with two other NLP data augmentation methods: nlpaug_synonym (by replacing words with Wordnet  [25] ) and nlpaug_contextual_embedding (by self-masking words and replacing them by inferring using an LM  [17] ). In addition, we add 4 combinations of the previous methods to further improve the scores. Among all the possible combinations, we propose the following: (i) combined_nlpaug: which applies nlpaug_synonym and nlpaug_contextual_embedding, (ii) combined_geonlplify_s_eq: for GeoNLPlify_specialization, (iii) combined_geonlplify_all: for all the three GeoNLPlify methods and GeoNLPlify_equivalence (which obtained the best results) and finally (iv) combined_geonlplify_nlpaug: for GeoNLPlify_specialization & GeoNLPlify_equivalence and nlpaug_synonym & nlpaug_contextual_embedding.
The benchmarking is performed on three different datasets. The first two are related to crises (PADIweb  [12]  and CrisisNLP  [13] ). The third one is dedicated to sentiment analysis applied to movie critics (SST2), and it is part of the general language understanding evaluation (GLUE)  [14]. SST2 was created to evaluate the text classification models. Since SST2 has very little spatial information, it is interesting to observe how GeoNLPify behaves with corpora for which spatiality matters very little.
PADI-web The five data augmentation methods alone and the four combinations are applied to the PADI-web dataset. The F1-score of each class is shown in Table  1. Concerning the benchmark of the DAs alone, the first observation is that all augmentation techniques achieve better results than without augmentation (called "None" in the figure). Second, two GeoNPLlify methods, "spatial equivalence" and "Specialization" obtain better results comparing the nlpaug "Synonym" except for the class "General Information". The fact that this class has little geographic information does not completely explain the poorer results that GeoNLPlify receives since we do not observe a decrease in results for the "Other" class. The best combination is "combined_geonlplify_nlpaug" which obtains also the best results for all the classes.
SST2 The results are shown in Tables  3  and 4. The first observation is that no augmentation method truly stands out from the original dataset (e. g. , without augmentation). The second observation is that the size of the dataset (300 reviews or 11000 reviews) greatly influences the results and their stability. Indeed, for the dataset truncated at 300 reviews, "nlpaug_synonym" is the best, while "geonlplify_specialization" comes first on the 11000 reviews dataset. The third observation is that GeoNLPlify still performs better than it would without augmentation for all the sizes, whereas nlpaug performance is completely degraded on the 11000 documents corpus (especially for the negative class). This may be because some reviews are very small. Often, these reviews do not contain spatial information, so GeoNLPlify is unable to create variations from them.
Interpretation of GeoNLPlify effectiveness As shown in the experiments Section 4, GeoNLPlify substantially improves a fine-tuned LM on text classification even on datasets for which the spatial information matters less, such as sentiment analysis on movie reviews (SST-2). However, GeoNLPlify is similar to other data augmentation approaches, i. e. , duplicating documents by making changes at the word level. How then, can we explain the better results achieved by GeoNLPlify compared to the classical techniques? Longpre et al. [9] , by an empirical study, stated that DA techniques help only when they provide linguistic patterns that are not seen during pretraining. Linguistic patterns are a set of grammar, syntax rules and semantics. According to  [40] , the BERT-like models store their syntax knowledge in their token representations (embedding) and not on their attention heads.
The closer the dots are, the more similar the documents are. We can observe that the GeoNLPlify and nlpaug_synonym have documents with representations that are the most distant. Since it can be difficult to visually analyze the extent brought by each DA in the figure, we propose to compute and compare the area of the smallest polygon containing all the 2D points for each DA. The sizes of these polygons enable to estimate and compare the semantic extent in order to quantify the linguistic patterns brought by each DA. Table  5  shows these results and we can see that the GeoNLPlify methods increase the semantic extent of the training dataset significantly. GeoNPLify artificially creates labeled data very similar to the original data for a human but not for an LM. For example, the cosine similarity (which is the common measure of the distance between two vectors) between the embedding of two countries, such as New Zealand and France is only 17% (for Germany and France it is 60%).