SlowGuess/NAACL_2022_OCR huggingface dataset:
Pei Chen<sup>1</sup> Haotian Xu<sup>1</sup> Cheng Zhang<sup>2</sup> Ruihong Huang<sup>1</sup>
$^{1}$  Department of Computer Science and Engineering, Texas A&M University  $^{2}$  Purdue University Northwest
{chenpei, hx105, huangrh}@tamu.edu
zhan4168@pnw.edu
General domain Named Entity Recognition (NER) datasets like CoNLL-2003 mostly annotate coarse-grained location entities such as a country or a city. But many applications require identifying fine-grained locations from texts and mapping them precisely to geographic sites, e.g., a crossroad, an apartment building, or a grocery store. In this paper, we introduce a new dataset HarveyNER with fine-grained locations annotated in tweets. This dataset presents unique challenges and characterizes many complex and long location mentions in informal descriptions. We built strong baseline models using Curriculum Learning and experimented with different heuristic curricula to better recognize difficult location mentions. Experimental results show that the simple curricula can improve the system's performance on hard cases and its overall performance, and outperform several other baseline systems. The dataset and the baseline models can be found at https://github.com/brickee/HarveyNER.
The Named Entity Recognition (NER) task aims to locate and classify textual phrases as entity mentions that belong to predefined entity categories. Location is one of the general entity categories and has been annotated in many NER datasets, including CoNLL-2003 (Tjong Kim Sang, 2002) and OntoNotes 5.0 (Pradhan et al., 2013). However, these datasets contain mostly coarse-grained entities such as a continent (e.g., Europe), a country (e.g., the U.S.), or a city (e.g., London).
Many downstream applications require identifying fine-grained location entities from texts, such as an apartment building (e.g., Bayou Oaks) or a specific store (e.g., the HEB on Montrose), in order to locate the geographic places on a map, which is vital to identify actionable information
Figure 1: An example of a disaster response system.
(Khanal and Caragea, 2021). For example, in Figure 1, a flood disaster happened in the Houston area and then someone tweeted the shortage of necessities in two locations. If a disaster response system can detect the disaster-related tweets, identify the two location mentions from the text, and link them to location entities on the map, necessary help can be directly delivered to the people living in disaster-affected places. Accurately identifying the fine-grained location mentions plays a critical role in such a system.
Several previous works have attempted to create crisis-related datasets with fine-grained location mentions, either automatically (Middleton et al., 2013) or by manual (Khanal et al., 2021) annotations. However, these prior datasets contain many incomplete descriptions of locations that otherwise can be precisely projected to a map with certain geo-coordinates. For example, "the corner of Richey St and W Harris Ave in Pasadena" is an intersection of two roads and we annotate it as a Point on the map, but previous work regard it as two Road mentions "Richey St" and "W Harris Ave in Pasadena", and such incomplete location mentions will affect uses in many applications. We introduce
Human Language Technologies, pages 3329 - 3339
HarveyNER, the first dataset that annotates such coordinate-oriented location mentions.
We use the Harvey disaster in Houston as an example to demonstrate how to annotate such location mentions. Specifically, we consider tweets about Hurricane Harvey affecting the Houston metropolitan area in 2017 and annotate mentions of locations that exist in this city. Compared with the location mentions in prior NER datasets, HarveyNER focuses on the location mentions that can link to specific sites on a map. We carefully constructed the annotation guidelines and trained annotators to obtain high-quality annotations. We also built strong baselines over the dataset for future reference.
The unique characteristics of HarveyNER present challenges for NER systems. First, many location entities in this dataset are long and complex to precisely describe a place. E.g., the above example of a Point entity contains up to 11 words, and it could be wrongly recognized as two road entities by a NER system. Second, as an instant social medium, tweets contain many informal contents, local conventions, and even grammatical errors, which create many out-of-vocabulary (OOV) words that cannot be found in pretrained word embedding such as Glove (Pennington et al., 2014) or BERT (Devlin et al., 2019).
To improve the performance on these hard location mentions and build strong baselines for the HarveyNER dataset, we adopt Curriculum Learning (CL) (Bengio et al., 2009) to better learn difficulty samples by ordering examples during training based on their difficulty. We design two heuristic curricula based on entity length and word complexity considering that many long and complex entities in HarveyNER are naturally difficult (as shown in Figure 3, the performance of baseline systems is worse on these hard cases). We further assume that the difficulty to learn may not only depend on the inherent difficulty of a type of case but also depend on how commonly seen or how well represented such cases are in the dataset. Therefore, we propose a novel curriculum with a difficulty scoring function that comprehensively considers the two heuristic difficulty metrics as instance frequencies. Empirical results show that all of the curricula can outperform several other baseline systems, and our novel curriculum performs the best.
We also find that different NER-based systems benefit from different curriculum scheduling strate
gies. In our experiments, the normal curriculum (training with easier samples first) is suitable for training the neural network-based model NCRF++ without pretrained language models, while the anticurriculum (training with harder samples first) facilitates fine-tuning of the pretrained language model BERT.
NER research has a long history and many NER datasets have been created with certain pre-defined entity categories. General domain datasets such as CoNLL-2003 (Tjong Kim Sang, 2002) and OntoNotes 5.0 (Pradhan et al., 2013) attend to certain common entity types including Location. Location mentions in these datasets are mostly coarse-grained, e.g., the U.S. (a country) or London (a city). Li and Sun (2014); Ji et al. (2016) focus on identifying fine-grained points-of-interest for location-based services, and their dataset is automatically constructed by mapping location inventory to tweets. Khanal and Caragea (2021); Khanal et al. (2021) try to identify crisis-related location mentions, but their dataset contains incomplete location mentions and is of limited use for a disaster response system. In contrast, our dataset HarveyNER emphasizes fine-grained locations that can map to coordinates on a map.
Recent approaches (Yang and Zhang, 2018; Li et al., 2020; Chen et al., 2021) use Neural Network models like BiLSTM-CNN-CRF (Ma and Hovy, 2016) and contextual embeddings like BERT (Devlin et al., 2019), and have greatly improved the NER performance. However, none of these approaches consider the difficulty of different NER cases in their model training. Bengio et al. (2009) pointed out that using a curriculum strategy enables the model to learn from easy examples to complex ones and leads to generalization improvement. Many Natural Language Processing tasks such as machine translation (Platanios et al., 2019; Liu et al., 2020; Zhang et al., 2021), natural language understanding (Xu et al., 2020), text generation (Liu et al., 2018, 2021) and dialogue systems (Su et al., 2021) benefit from such curriculum learning strategies. Considering that HarveyNER contains many long and complex location mentions, we design corresponding curricula to learn them.
Data Split	Train	Valid	Test	Total
# of Tweets	3,967	1,301	1,303	6,571
Tweets w/ Entity	1,087	366	353	1,806
Tweets w/o Entity	2,880	935	950	4,765
# of Entity Mentions	1,581	523	500	2,604
Point	591	206	202	999
Area	715	236	212	1,163
Road	158	51	57	266
River	117	30	29	176
3 The HarveyNER Dataset
Data Collection We used the Twitter PowerTrack API to retrieve the tweets posted during the time of peak disruption caused by Hurricane Harvey in the Houston area, specifically from 5:00 a.m. August 25 to 4:59 a.m. August 31, 2017. In total, we collected 1,121,363 tweets excluding retweets and replies.
Data Cleaning We applied several strategies to filter out irrelevant tweets. First, we only keep the tweets that are related to the Houston area, i.e., the geo-coordinates of the tweets or the authors' profile locations are within the bounding of Houston. Second, we applied our weakly-supervised event detection system (Yao et al., 2020) to identify tweets on disaster-related topics; these tweets are likely to be related to Hurricane Harvey during the specified period. We also manually filtered out remaining irrelevant tweets (such as non-English and repeated tweets) during the annotation process. In total, 6,571 tweets were selected for this study, as shown in Table 1.
3.2 Location Entity Annotation
Location Types HarveyNER focuses on the coordinate-oriented locations so we mainly annotate Point that can be precisely pinned to a map and Area that occupies a small polygon of a map. Considering that some disasters can affect line-like objects (e.g., a flood can affect the neighbors of a whole river), we also include Road and River types.
- Point: denote an exact location that a geo-coordinate can be assigned. E.g., a uniquely named building, intersections of roads or rivers;
- Area: denote geographical entities such as city
Table 1: Statistics of the HarveyNER Dataset.
kappa κ	A1 & A2	A1 & A3	A2 & A3	Average
All	85.64	82.17	83.12	83.64
Annotated	66.54	60.49	62.09	63.04
Table 2: Inter-Annotator Agreement (%) at token-level. All for all the tokens and Annotated for annotated tokens only. There are three annotators A1, A2 and A3.
subdivisions, neighborhoods, etc;
- Road: denote a road or a section of a road;
- River: denote a river or a section of a river.
Annotation Quality To train the annotators to well annotate the fine-grained location mentions, especially to distinguish the Point locations, we conduct rounds of initial annotation exercises and receptively update annotation guidelines to reduce ambiguity and subjectivity. The detailed guidelines can be found in Appendix A.1.
We trained three annotators and calculated their Inter-Annotator Agreement (IAA) based on 500 randomly selected tweets they all annotated. We pairwise calculate the Cohen's kappa  $(\kappa)$  scores based on the token-level annotations from each pair of annotators. As suggested by Brandsen et al. (2020), we report two scores: one calculated using all the token annotations and one only using the annotated tokens that exclude non-entity tokens. As shown in Table 2, we observe a high average  $\kappa$  score of  $83.64\%$  for all tokens and an average  $\kappa$  score of  $63.04\%$  for annotated tokens only. After that, the three annotators annotated the remaining tweets independently.
We randomly split the annotated tweets into training, validation, and test sets for experiments with a ratio of 6:2:2. Table 1 shows some basic data statistics. We can see that  $27.48\%$  of the tweets contain at least one location entity mention, while the remaining tweets do not mention any location. As for location types, Point and Area are two dominant entity types covering  $38.36\%$  and  $44.66\%$  of entity mentions respectively, while Road and River only make up  $10.22\%$  and  $6.76\%$  of entity mentions respectively.
Comparisons with CoNLL-2003 We compare HarveyNER with CoNLL-2003, a general NER dataset annotated with coarse-grained locations in news articles, and Table 3 shows the comparisons in several aspects.
First of all, entities in HarveyNER are longer
Datasets	HarveyNER	CoNLL-2003 (Loc-only)
Avg Entity Length (word)	2.68	1.15
Avg Entity Length (char)	13.91	7.24
Complex Entity Rate (%)	11.8	0.19
OOV Rate (%)	14.47	2.33
Avg Sent Length (word)	20.07	14.53
Avg Sent Length (char)	117.03	76.89
Avg Entity Count	0.40	0.51
– non-empty	1.44	1.38
Avg Entity Ratio (%)	5.33	7.23
– non-empty	19.39	19.43
Figure 2: Number of Location Mentions with Each Complexity Indicator Word.
on average at both word level and character level. We observed that many location mentions of the type Point and Area are especially long to precisely describe a site or area on a map. We manually analyzed long entities in the validation set and observed that many location mentions are complex noun phrases with a conjunction or a prepositional phrase attachment. We noted down two commonly seen conjunctions and seven commonly seen prepositions, and table 4 shows location examples for each. We calculate the percentage of complex entity mentions with one of these words, and HarveyNER has  $11.8\%$  of entity mentions fall in this category while CoNLL-2003 has few such entity mentions  $(0.19\%)$ . Figure 2 shows the number of tweets with each of the words.
Second, considering that the language used in tweets is informal and contains many abbreviations
Table 3: HarveyNER v.s. CoNLL-2003.
Indicators	Examples
"and"	the corner of Richey St and W Harris Ave in Pasadena (Point)
"&"	Beltway 8 & Tidwell (Point)
"at"	Brazos River at Richmond (River)
"@"	Copperfield Church @ 8350 hwy 6 north (Point)
"in"	Constellation Field in Sugar Land (Point)
"on"	Chimney Rock on I-10 East (Point)
"near"	IH 10 near Monmouth (Point)
"between"	249 between Cypresswood / Louetta (Point)
"of"	0.25-0.5 north of I-10 (Point)
Table 4: Examples of complex entities.
and even grammatical errors, we calculate the out-of-vocabulary (OOV) rates (in Table 3) for both datasets by counting words that are absent from the pretrained Glove $^{1}$  (Pennington et al., 2014) word lists. We can see that the HarveyNER has a much higher OOV rate than CoNLL-2003. The high OOV rate might degrade the performance of NER systems relying on pretrained word embeddings.
In addition, we compare the average sentence length between the two datasets. To our surprise, HarveyNER has overall longer sentences, based on both word counts and character counts. This is counter-intuitive since the tweet content is strictly constrained to be no more than 140 characters each. One possible reason is that short tweets are less likely to provide useful event information and have been filtered out by the event detection system (Yao et al., 2020) we used.
Lastly, we measure the density of annotated location entities in the two datasets by calculating the average number of location entity mentions per sentence and calculating the percentage of entity words out of all the words in a sentence. We also calculate these two measures for the subset of sentences that contain at least one location mention (non-empty sentences). The last section of Table 3 shows the results. We can see that the two datasets are similar over these annotation density measures.
4 Curriculum Arrangement
In consideration of the characteristic difficulties of HarveyNER, we employ curriculum arrangements to help learn these hard cases. We follow the curriculum designing approach introduced by Bengio et al. (2009), which mainly requires specifying two functions:
- Difficulty Scoring Function: Given an input sample  $\mathbf{x}_i$ , this function map it to a numerical
For fair comparison, we use glove.twitter.27B for HarveyNER and glove.6B for CoNLL-2003.
score,  $d(\pmb{x}_i) \in \mathbb{R}$ . The score is used to represent the difficulty level of the corresponding sample. Usually, the higher the score, the more difficult the sample is.
- Pacing Function: The pacing function  $p(t) \in (0,1]$  specifies the input training data size at time or step  $t$ . Normally we use  $p(t)$  the lowest difficulty-scored samples for training at time  $t$ , but in the anti-curriculum setting, we use  $p(t)$  the highest difficulty-scored samples. Given such a subset of the dataset containing the easiest or hardest ones, we sample training batches uniformly from it for training.
The curriculum learning procedure using the two functions is described in Algorithm 1.
Algorithm 1 Curriculum Learning with Scoring and Pacing Functions
- The training Data,  $\mathcal{D}^{\mathrm{train}} = \{\pmb{x}_i\}_{i=1}^N$ , including  $N$  samples;
- A model  $\mathcal{M}$  that takes batches of data for training at each step  $t$ ;
- A difficulty scoring function  $d$ ;
- A pacing function  $p(t)$ .
Output: A model  $\mathcal{M}^{\mathrm{trained}}$  trained with the curriculum.
1: Compute the difficulty score  $d(\pmb{x}_i)$  for each sample;
2: Sort  $\mathcal{D}^{\mathrm{train}}$  ascendingly or descendingly based on  $d(\pmb{x}_i)$  and obtain  $\mathcal{D}_{\mathrm{sort}}^{\mathrm{train}}$ ;
3: Initialize the pacing function  $p(0)$ ;
4: Generate the initial curriculum  $\mathcal{D}_0$  using the top  $p(0)$  samples in  $\mathcal{D}_{\mathrm{sorted}}^{\mathrm{train}}$ ;
5: for training epoch  $t = 1,2,\ldots$  do
6: Uniformly sample batches from the current curriculum  $\mathcal{D}_{t - 1}$  for model training;
7: Update the pacing function  $p(t)$  based on equation Eq. (6);
8: Generate the next curriculum  $\mathcal{D}_t$  using the top  $p(t)$  samples in  $\mathcal{D}_{\mathrm{sort}}^{\mathrm{train}}$ ;
4.1 Three Difficulty Scoring Functions
We first design two dataset-specific heuristic curricula, based on maximum entity length and entity complexity<sup>2</sup>, inspired by the dataset analysis in Section 3.3. Then, we introduce a new metric that integrates the two heuristic metrics.
Maximum Entity Length (Max): As mentioned previously, our HarveyNER dataset has longer entity mentions on average compared to CoNLL-2003, and this brings many long and difficult entities that are hard to identify. Intuitively, we can design a corresponding curriculum based on such an entity-level difficulty. Specifically, given
an input tweet sample  $\pmb{x}_i$  that contains  $n$  words,  $\pmb{x}_i = \{w_1, w_2, \dots, w_n\}$ , and  $k$  ( $k \geq 0$ ) entities,  $\{E_1, E_2, \dots, E_k\}$ .  $|E_j|$  represents the length of  $j$ -th entity, specifically, the number of words in the  $j$ -th entity. Now, we can assign each tweet sample a score using the maximum length<sup>3</sup> of its entities:
$$
d _ {\max } \left(\boldsymbol {x} _ {i}\right) = \max  \left(L _ {i}\right) \tag {1}
$$
where,  $L_{i} = \{|E_{1}|, |E_{2}|, \ldots |E_{k}|\}$ , the set of entity lengths for the i-th sample  $\boldsymbol{x}_{i}$ .
With this scoring function, we need to pay attention to the tweets with zero entity mention (72.52% of tweets in HarveyNER as shown in Table 1) since their difficulty scores will all be 0. In this case, the algorithm will provide all these tweets in one step to the curriculum, which will mislead the model to a local minimum and learn that no entity exists in the data. We propose a remedy to this issue by randomly feeding the empty tweet samples. Specifically, when we order our dataset by the difficulty scores, we randomly intersperse those no-entity tweet samples among the ordered samples that have entities.
Complex Entity Rate (Complex): Corresponding to the analysis of the complex entity rate in HarveyNER, we define another difficulty scoring function. Specifically, we define the complexity of an entity  $c(E)$  based on whether the entity contains one of the conjunction words or preposition words we identified and which word the entity contains. Heuristically, we assign a weight greater than 1 to these words, specifically, we assign a weight of 3 to each conjunction word and a weight of  $2^{4}$  to each preposition word to reflect our intuition that entities with conjunctions can be more difficult cases.
If an entity  $E$  contains more than one "complexity" indicator, we choose the one with the highest weight. For example, the entity example  $E$  "the corner of Richey St and WHarris Ave in Pasadena" contains the conjunction "and" and the preposition "in", we deem the complexity of this entity  $c(E)$  is 3 instead of 2. Then, one tweet sample  $x_{i}$  can have multiple entities with different complexities  $C_i = \{c(E_1), c(E_2), \ldots, c(E_k)\}$ , we assign the
3We also tried using the average entity length as the difficulty score in our experiment but the performance is not as good.
4We further lower the weight for the preposition "of" to 1 considering that this is a very general preposition and is often observed in regular location entities as well, e.g., "University of Houston".
2We tried using the OOV rate as the difficulty score in our experiment, but the performance is not as good.
maximum entity complexity value as the complexity value of the tweet sample, i.e.,
$$
d _ {\text {c o m p l e x}} \left(\boldsymbol {x} _ {i}\right) = \max  \left(C _ {i}\right) \tag {2}
$$
However, if none of the entities in a tweet sample contain complexity indicators, this scoring function will assign 0 as the difficulty score for the tweet sample. A similar issue has been discussed for the previous scoring function, and we use the same remedy as well and randomly interspersed these zero-scored samples among the other ordered samples.
Commonness of Difficulty (Commonness): We further propose a novel difficulty metric that considers both of the prior metrics as well as their support. We exploit the assumption that the difficulty to learn may not only depend on the inherent difficulty of a type of case but also depend on how commonly seen or how well represented such cases are in the dataset.
Specifically, to assign a difficulty score for a tweet sample, we first count the number of training samples that have the same difficulty score as the sample  $\boldsymbol{x}_i$  according to one of the prior two metrics and then divide it by the total number of tweet instances  $N$ . Then, we take the reciprocal of it and get  $f_{\mathrm{metric}}$ .
$$
f _ {\text {m e t r i c}} \left(\boldsymbol {x} _ {i}\right) = \frac {1}{\operatorname {c o u n t} \left(d _ {\text {m e t r i c}} \left(\boldsymbol {x} _ {i}\right)\right) / N} \tag {3}
$$
Where,  $d_{\mathrm{metric}}$  are the difficulty metrics  $d_{\mathrm{max}}$  or  $d_{\mathrm{complex}}$ . Hence, the larger support of a difficulty level based on one of the two prior heuristics, the lower the  $f_{\mathrm{metric}}$  value is. We then normalize this value to the range of [0, 1].
$$
f _ {\text {m e t r i c}} \left(\boldsymbol {x} _ {i}\right) = \frac {f _ {\text {m e t r i c}} \left(\boldsymbol {x} _ {i}\right) - \operatorname* {m i n} \left(f _ {\text {m e t r i c}}\right)}{\operatorname* {m a x} \left(f _ {\text {m e t r i c}}\right) - \operatorname* {m i n} \left(f _ {\text {m e t r i c}}\right)} \tag {4}
$$
Then, we integrate  $f_{\mathrm{max}}$  and  $f_{\mathrm{complex}}$ , and take the  $L2$ -norm to generate the final difficulty score.
$$
d _ {\text {c o m m o n}} \left(\boldsymbol {x} _ {i}\right) = \sqrt {f _ {\max } \left(\boldsymbol {x} _ {i}\right) ^ {2} + \left(\lambda f _ {\text {c o m p l e x}} \left(\boldsymbol {x} _ {i}\right)\right) ^ {2}} \tag {5}
$$
As a result, the more common a sample is concerning its length or complexity, the smaller the  $L2$ -norm value is, which indicates a lower difficulty based on the new metric. In addition, we add a hyperparameter  $\lambda$  to balance the influence of the two metrics.
Similar to the previous single difficulty-based curricula, the commonness difficulty score is zero when a tweet sample has no entity. We adopt the same remedy and randomly intersperse those no-entity tweet samples among the ordered ones that contain entities.
4.2 Pacing Function
We use the root-based pacing function introduced by Platanios et al. (2019) in all our experiments.
$$
p (t) = \sqrt {t \cdot \frac {1 - p (0) ^ {2}}{T} + p (0) ^ {2}} \tag {6}
$$
Here  $p(0)$  defines the proportion of samples we feed our model at the very beginning;  $T$  is the number of epochs that we apply curriculum learning to our model.
In our experiments, we use two state-of-the-art NER systems as base models and evaluate their performance on the HarveyNER dataset. Then, we test the effectiveness of the designed curricula by applying them to train the base models.