SlowGuess/NAACL_2022_OCR huggingface dataset:
$\mathbf{N}\mathbf{C}\mathbf{R}\mathbf{F} + +$  (Yang and Zhang, 2018) is an open-source Neural Sequence Labelling Toolkit. We use the BiLSTM-CNN-CRF structure as a base model. BERT (Devlin et al., 2019), a pretrained language model based on Transformer (Vaswani et al., 2017), has significantly improved many NLP tasks including NER. We fine-tune the base-uncased version of BERT with the BiLSTM-CRF structtrue for experiments.
5.2 Training Setup
For the NCRF++ model, we use the tweet-based version Glove as word embeddings and keep all the other hyper-parameters as default. For the BERT model, we set learning rate as 5e-5 and set batch size as 32. As for the  $\lambda$  hyperparameter in Eq. (5), we used grid search and set it 1 and 0.6 for the NCRF++ model and the BERT model respectively. We train all the NCRF++ models for 100 epochs and train all the BERT models for 50 epochs.
For fair comparisons, we keep all the training parameters the same when conducting curriculum learning. For the NCRF++ model, we use the normal curriculum setting and feed easier cases
Models	Entity Type in HarveyNER
Point	Area	Road	River	Micro-Average
NCRF++	71.43 / 72.26 / 71.85	66.00 / 61.68 / 63.77	77.39 / 77.93 / 77.66	61.40 / 44.56 / 51.64	68.69 / 65.16 / 66.88
+ Max	72.55 / 71.51 / 72.03	65.90 / 65.54 / 65.72	75.30 / 77.93 / 76.59	62.42 / 44.56 / 52.00	69.06 / 66.40 / 67.70
+ Complex	70.47 / 72.08 / 71.26	66.07 / 64.16 / 65.10	74.67 / 75.17 / 74.92	63.50 / 44.56 / 52.37	68.34 / 65.92 / 67.11
+ Commonness	71.40 / 72.64 / 72.02	68.27 / 65.84 / 67.03	77.23 / 77.24 / 77.24	66.68 / 45.96 / 54.42	70.09 / 67.12 / 68.57
BERT	71.55 / 73.11 / 72.32	62.04 / 72.87 / 67.02	76.42 / 82.07 / 79.15	62.11 / 55.09 / 58.39	66.62 / 71.48 / 68.97
+ Max	72.14 / 72.74 / 72.44	62.49 / 72.67 / 67.20	77.83 / 80.69 / 79.23	57.92 / 56.14 / 57.02	66.73 / 71.28 / 68.93
+ Complex	70.41 / 75.47 / 72.85	62.32 / 72.87 / 67.19	76.12 / 82.76 / 79.30	59.92 / 55.09 / 57.40	66.13 / 72.52 / 69.18
+ Commonness	72.98 / 73.87 / 73.42	62.53 / 71.98 / 66.92	79.20 / 78.62 / 78.91	63.55 / 60.00 / 61.72	67.66 / 71.80 / 69.67
Table 5: Evaluation on the test set, Precision / Recall / F1-Score (Percentages) $^5$ . Since we use the same pacing function, we use the scoring functions to name the curricula. Note that we apply the normal curriculum setting to the NCRF++ model and apply the anti-curriculum setting to the BERT model.
first, while for the BERT model, we use the anticurriculum setting (more explanations provided in Section 5.5). Note that we train all the experiments five times using different random seeds to alleviate random turbulence.
Table 5 shows the experimental results. We can see that the BERT base model outperforms the NCRF++ base model consistently across the four location categories on this dataset. Curriculum learning yields further performance gains for both the BERT model and the NCRF++ model, this is true for all the curricula paired with both models except the Max curriculum when applied to the BERT model, where the average performance almost stays the same. Among the three curricula, he Commonness curriculum achieves the best performance for both models.
Conducting curriculum learning have unequal impacts on the four location categories. When using the NCRF++ model, curriculum learning yields a small performance improvement on Point and clear improvements on Area and River, while using the BERT model, curriculum learning yields a relatively larger improvement on Point and a clear improvement on River as well.
Note that the best-performed BERT model only achieves a micro-average F1-score of  $69.67\%$  on this dataset, which is still much lower than recently published BERT-base performance on CoNLL-2003 (e.g.,  $92.4\%$  as reported in (Devlin et al., 2019)).) Meanwhile, we acknowledge that recognizing fine-grained locations is a trickier task than recognizing coarse-grained locations or many other general types of entities, even for humans as shown by the imperfect inter-annotator agreements.
5.4 How are the Difficult Samples Learned?
In order to understand if the models have better learned the difficult samples after applying curriculum learning, we divide the test set into "easy" and "hard" subsets based on either entity length or entity complexity and report experimental results on each subset. In this analysis, we only consider tweet samples in the test set that contains at least one entity mention since the groupings are determined by the characteristics of the entities. In the first grouping, we divide tweet samples into "short" and "long" groups depending on if the maximum entity length (number of words) in a tweet sample is greater than a threshold, four words in particular  $(<= 4\text{v.s.} > 4)$ . In the second grouping, we divide tweet samples into "simple" and "complex" groups by checking if a tweet sample contains a complex entity mention<sup>6</sup>.
Across all the experimental settings, we report the results on two subsets separately under each grouping (Figure 3). We can see that curriculum learning indeed yields noticeable improvements in identifying those hard cases, while the improvements vary when adopting different curricula. Meanwhile, curriculum learning also achieves mild improvements on identifying the remaining relatively easy cases.
5.5 Curriculum v.s Anti-curriculum
We find that applying different curriculum settings (normal curriculum that exposes the easiest examples first or anti-curriculum that exposes the most difficult examples first) results in a large performance difference between the NCRF++ model and the BERT model. As shown in Figure 4, for the NCRF++ model without pretrained language mod
${}^{6}$  A complex entity mention has one of the conjunctions or prepositions we identified.
5 All results are the average of 5 system runs.
Figure 3: Results on "easy" and "hard" subsets of the test data, F1-score (Percentage).
Figure 4: Curriculum v.s. Anti-curriculum, F1-score (Percentage).
els, the normal curriculum setting yields significantly better average F-1 scores across all the three curriculum scoring functions in comparison with the anti-curriculum setting. However, for the pretrained language model BERT, the results are the opposite; using anti-curriculum learning consistently yields better performance than using normal curriculum learning.
One possible explanation is that the volatile gradients resulting from using anti-curriculum learning can lead to better local minima for a well-pretrained model. Specifically, the anti-curriculum learning will feed those "hard" samples to the model first, and the gradients from those long-tailed hard cases will cause relatively large fluctuations compared to those from easy instances. BERT is a pretrained language model and the pretrained pa
rameters might constrain the model to some local regions. The fluctuations produced by the "hard" samples from the anti-curriculum learning can enable the BERT model to reach other better local minima regions.
We introduce a fine-grained location recognition dataset, HarveyNER, to enable many downstream applications such as building real-time disaster response systems. This dataset contains many long and complex location mentions that feature interesting internal syntactic and semantic structures and the state-of-the-art NER systems are unable to fully recognize these hard cases. Considering the clear characteristics of difficult cases in this dataset, we experimented with two heuristic curriculum learning strategies and a novel commonness-based curriculum strategy to better recognize the difficult location mentions. Empirical results demonstrate the effectiveness of the curricula, which serve as strong baseline results in this dataset. Future work may consider incorporating external knowledge or innovations on system architectures to better identify fine-grained location mentions.
We gratefully acknowledge support from National Science Foundation via the awards IIS-1909252 and IIS-1759537. In addition, this research is par
tially supported by 2021 PRISE Grant 02-243712. We would also like to thank the anonymous reviewers for their valuable feedback and input.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, Montreal, Quebec, Canada, June 14-18, 2009, volume 382 of ACM International Conference Proceeding Series, pages 41-48. ACM.
Alex Brandsen, Suzan Verberne, Milco Wansleeben, and Karsten Lambers. 2020. Creating a dataset for named entity recognition in the archaeology domain. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4573-4577, Marseille, France. European Language Resources Association.
Pei Chen, Haibo Ding, Jun Araki, and Ruihong Huang. 2021. Explicitly capturing relations between entity mentions via graph neural networks for domain-specific named entity recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 735-742, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zongcheng Ji, Aixin Sun, Gao Cong, and Jialong Han. 2016. Joint recognition and linking of fine-grained locations from tweets. In Proceedings of the 25th International Conference on World Wide Web, WWW 2016, Montreal, Canada, April 11 - 15, 2016, pages 1271-1281. ACM.
Sarthak Khanal and Doina Caragea. 2021. Multi-task learning to enable location mention identification in the early hours of a crisis event. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4051-4056.
Sarthak Khanal, Maria Traskowsky, and Doina Caragea. 2021. Identification of fine-grained location mentions in crisis tweets. arXiv preprint arXiv:2111.06334.
Chenliang Li and Aixin Sun. 2014. Fine-grained location extraction from tweets with temporal awareness. In The 37th International ACM SIGIR Conference on Research and Development in Information Retrieval,
SIGIR '14, Gold Coast, QLD, Australia - July 06 - 11, 2014, pages 43-52. ACM.
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020. A unified MRC framework for named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5849-5859, Online. Association for Computational Linguistics.
Cao Liu, Shizhu He, Kang Liu, and Jun Zhao. 2018. Curriculum learning for natural answer generation. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 4223-4229. ijcai.org.
Fenglin Liu, Shen Ge, and Xian Wu. 2021. Competence-based multimodal curriculum learning for medical report generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3001-3012, Online. Association for Computational Linguistics.
Xuebo Liu, Houtim Lai, Derek F. Wong, and Lidia S. Chao. 2020. Norm-based curriculum learning for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 427-436, Online. Association for Computational Linguistics.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074, Berlin, Germany. Association for Computational Linguistics.
Stuart E Middleton, Lee Middleton, and Stefano Modaferri. 2013. Real-time crisis mapping of natural disasters using social media. IEEE Intelligent Systems, 29(2):9-17.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom Mitchell. 2019. Competence-based curriculum learning for neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1162-1172, Minneapolis, Minnesota. Association for Computational Linguistics.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina,
Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143-152, Sofia, Bulgaria. Association for Computational Linguistics.
Yixuan Su, Deng Cai, Qingyu Zhou, Zibo Lin, Simon Baker, Yunbo Cao, Shuming Shi, Nigel Collier, and Yan Wang. 2021. Dialogue response selection with hierarchical curriculum learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1740-1751, Online. Association for Computational Linguistics.
Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. 2020. Curriculum learning for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6095-6104, Online. Association for Computational Linguistics.
Jie Yang and Yue Zhang. 2018. NCRF++: An open-source neural sequence labeling toolkit. In Proceedings of ACL 2018, System Demonstrations, pages 74-79, Melbourne, Australia. Association for Computational Linguistics.
Wenlin Yao, Cheng Zhang, Shiva Saravanan, Ruihong Huang, and Ali Mostafavi. 2020. Weakly-supervised fine-grained event recognition on social media texts for disaster management. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 532-539. AAAI Press.
Mingliang Zhang, Fandong Meng, Yunhai Tong, and Jie Zhou. 2021. Competence-based curriculum learning for multilingual machine translation. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 2481-2493, Punta Cana, Dominican Republic. Association for Computational Linguistics.
1. Location types can be "Area", "Point", "Road", and "River."
- "Area" refers to all the named entities of cities, neighborhoods, super neighborhoods, geographic divisions etc.
- "Point" refers to a location that is a building, a landmark, an intersection of two roads, an intersection of a river with a lake/reservoir/ocean, or a specific address.
- "Road" refers to a road/avenue/street or a section of a road/avenue/street when the tweet does not provide an exact location on that road.
- "River" refers to a river or a section of a river when the tweet does not imply there is an intersection between the river and other places.
- 2. A section of a road/river between two detailed/precise locations should be considered as a point. However, if the distance between the two points is very large, it might be considered as a stretch of a road/river.
- 3. A road passing through a small area can be designated as a point. A road intersecting a very large area cannot be a point and must be denoted as a stretch of a road. In some peculiar cases, the road takes a small detour and tangentially brushes off an area – in such specific cases, roads can be annotated as a point.
- 4. The following locations, Lake Houston, Barker Reservoir, and Addick's Reservoir, are annotated as areas due to their significant size while all other lakes/reservoirs are considered as points.
- 5. Ignore generic company/franchise names like HEB, Kroger etc. unless it is accompanied with a precise location, for example, HEB at Kirkwood Drive. However, non-franchised small businesses with only one unique location are considered as a point.
- 6. Ignore any locations in the Twitter username, like @HoustonABC. However, if the @ does not refer to a Twitter account name,
please recognize the location. For example, I am @ XXX High School, “XXX High School” will be considered as a point.
- 7. For abbreviations or vague location names, always look up the tweet's context (or even other tweets' context) to decide if it is a location or not. We will use search engine if it is necessary.
- Eg: Coke Ck; Here, "Ck" refers to a creek. This is understood when multiple such tweets point towards a creek.
- 8. Similarly, for names that can refer to different or multiple locations, like "Bellaire" can either refer to Bellaire St or the Bellaire area, we always look up the tweet's context to decide their location types.
- 9. We annotate the mentioned location as the complete set of phrases that describes the detail of the location including the core noun and all defining relative clauses. If a tweet mentioned the same location multiple times, they will be annotated as multiple location mentions.
- 10. Ignore the location that only contains "Houston", "Harris County", or "Texas"
- 11. Ignore any tweet outside Houston (like London, Dallas, etc) and all non-English tweets.
- 12. We keep the exact words in tweet context as the location name after extracting the entities.