SlowGuess/NAACL_2022_OCR huggingface dataset:
2021b. 3M-Transformers for Event Coding on Organized Crime Domain. IEEE International Conference on Data Science and Advanced Analytics (DSAA).
Ellie Pavlick, Heng Ji, Xiaoman Pan, and Chris Callison-Burch. 2016. The gun violence database: A new task and data set for nlp. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1018-1024.
Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In Proceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019), pages 58-65.
Alec Radford, Jeff Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Benjamin Radford. 2020a. Seeing the forest and the trees: Detection and cross-document coreference resolution of militarized interstate disputes. In Proceedings of the Workshop on Automated Extraction of Socio-political Events from News 2020, pages 35-41, Marseille, France. European Language Resources Association (ELRA).
Benjamin J Radford. 2020b. Multitask models for supervised protests detection in texts. arXiv preprint arXiv:2005.02954.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789.
Clionadh Raleigh, Andrew Linke, Håvard Hegre, and Joakim Karlsen. 2010. Introducing acled: an armed conflict location and event dataset: special data feature. Journal of peace research, 47(5):651-660.
Sayeded Salam, Patrick Brandty, Jennifer Holmesy, and Latifur Khan. 2018. Distributed framework for political event coding in real-time. In 2018 2nd European Conference on Electrical Engineering and Computer Science (EECS), pages 266-273.
Philip A. Schrodt. 2006. Twenty Years of the Kansas Event Data System Project. The Political Methodologist, 14(1):2-6.
Philip A. Schrodt. 2009. TABARI. Textual Analysis by Augmented Replacement Instructions.
Erick Skorupa Parolin, Latifur Khan, Javier Osorio, Vito D'Orazio, Patrick T. Brandt, and Jennifer Holmes. 2020. Hanke: Hierarchical attention networks for knowledge extraction in political science domain. In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pages 410-419.
Mohammad S Sorower. 2010. A literature survey on algorithms for multi-label learning. Oregon State University, Corvallis, 18:1-25.
START. 2019. The global terrorism database (gtd) [data file]. Retrieved from https://www.start.umd.edu/gtd.
Ralph Sundberg and Erik Melander. 2013. Introducing the ucdp georeferenced event dataset. Journal of Peace Research, 50(4):523-532.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Neural Information Processing Systems (NIPS), pages 5998-6008.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355.
Michael Ward, Andreas Beger, Josh Cutler, Matthew Dickenson, Cassy Dorff, and Ben Radford. 2013. Comparing GDELT and ICEWS Event Data.
John Wilkerson and Andreu Casas. 2017. Large-scale computerized text analysis in political science: Opportunities and challenges. Annual Review of Political Science, 20:529-544.
Mark D Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E Bourne, et al. 2016. The fair guiding principles for scientific data management and stewardship. Scientific data, 3.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system:
Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32.
Table 5 and Table 6 list the detailed sources in our Expert Domain Corpora and Main Stream Media Collection described in Section 3.2, respectively.
Type: Sources	Size (MB)
United Nations:
news.un.org, unodc.org, ohchr.org,
unhcr.org, Refworld.org	427
U.S. Department of State:
Annual Country Reports on Human
Rights Practices,
Annual Country Reports on Terrorism,
International Religious Freedom Reports,
Trafficking in Persons Report,
Foreign Relations of the United States*	1,027
Non-government organizations:
amnesty.org, hrw.org, rescue.org, phr.org,
thenewhumanitarian.org, satp.org, cfr.org	838
*We filter a subset after World War II (Sep 1945 to 1989)
Table 5: Sources in Expert Domain Corpora.
Region	Sources	Size (GB)
Asia	Aljazeera, CNA, IndianTimes, JapanTimes, SCMP, TheNewsIntl, Xinhua	2.0
Europe	BBC, DW, France24, Guardian, Reuters, RFI, TASS	3.7
US	ABC, AP, NBC, CNN, LATimes, NBC, NPR, NY Post, NYT, PBS, Politico, SFGATE, UPI, USA Today, US News, WASHPOST, WSJ	14.3
Others	AllAfrica, News24, EFE, TheConversation	0.8
Table 6: Sources in Mainstream Media Collection.
Filtering News Wires. We considered all the stories in EDC as relevant. However, for the general news in MMC, Gigaword, and PRT, we needed to
filter our specific domain of political conflict and violence based on the websites' metadata information such as URLs, subjects, and tags. For example, we collected the stories with the tags such as Conflicts, Violence, War, Politics, Defense, Crime, et al. We also defined a bag-of-words classifier to assess unlabeled stories' relevance to our domain. Therefore, we statistically summarized two lists of the most frequent keywords' regular expressions from relevant stories and irrelevant stories. There are 266 patterns in the relevant list and 246 in the not relevant list. For example, our relevant list contains patterns such as "activist", "protest", "counter. ?terrorism", and "jails? \b". Sports news use bellicose language similar to that of conflict stories with words such as attack, shoot, and defeat, thus presenting a classification challenge. The not relevant list contains frequent patterns such as "shot \w+ goal" to remove sports news. We compared the number of unique matching in the two lists and tuned the thresholds with the help of conflict experts. Finally, we filtered a small subset from MMC, Gigaword, and PRT in the conflict domain.
Filtering Wikipedia. We modified Wikiextractor (Attardi, 2015) to extract 18 GB size of documents with category labels from the Wikipedia dump<sup>5</sup> released on March 20, 2021. We used PetScan<sup>6</sup> to fetch pages of interest in the category hierarchy graph. We searched all the sub-categories within 0 to 4 depths under the union of five high-level topics: politics, activism, crime, government, and war. And we got 5 GB size of stories within 208,008 sub-categories from the query. Then, we summarized the top  $300+$  frequent keywords from our targeted categories to prune irrelevant or too far-away child nodes based on the sub-category labels. We also removed unrelated categories such as fictional characters, movies, video war games, and historical events or people before the 20th Century, et al.
Table 7 and Table 8 describe the detailed hyperparameters used in our pre-training and fine-tuning experiments, respectively. We implement our models using Huggingface API (Wolf et al., 2020).
<sup>5</sup>https://dumps.wikimedia.org
$^{6}$ https://petscan.wmflabs.org
Hyperparameter	SCR	Cont
Number of layers	12	12
Hidden Size	768	768
FFN inner hidden size	3072	3072
Attention heads	12	12
Mask percent	15	15
Learning Rate Decay	Linear	Linear
Warmup steps	10000	10000
Learning Rate LR	5e-4	5e-4
Adam ε	0.9	0.9
Adam β1	0.98	0.98
Adam β2	1e-6	1e-6
Attention Dropout	0.1	0.1
Dropout	0.1	0.1
Weight Decay	0.01	0.01
Train Steps	15,000	8,000
Vocabulary	ConfliVocab	BaseVocab
Uncased VOCab Size	30,000	30,552
Cased VOCab Size	30,000	28,996
Batch Size	2048	2048
Table 7: Hyperparameters for pre-training ConfliBERT using two strategies, pre-training from scratch (SCR) and continual pre-training (Cont). BaseVocab refers to the original BERT's vocabulary, while ConfliVocab refers to our domain-specific vocabulary.
Dataset - Tasks	max epochs	batch size	max seq-len	learning rate	drop-out
BBC News-BC	3	16	512	4e-5	0.1
20 News.-BC	3	16	512	4e-5	0.1
Gun V.-BC	10	8	512	5e-5	0.05
GLOCON-Sent BC	20	128	128	5e-5	0.05
GLOCON-Doc BC	5	8	512	5e-5	0.05
GTD-MCC	10	16	128	4e-5	0.1
SATP-BC	10	16	256	5e-5	0.05
SATP-Rel MLC	10	16	256	4e-5	0.1
SATP-All MLC	10	16	256	4e-5	0.1
InSight C.-MLC	5	16	512	4e-5	0.1
India P.-Sent MLC	10	16	128	4e-5	0.1
India P. - Doc MLC	10	16	512	4e-5	0.1
Event S.-TS MCC	10	192	150	5e-5	0.05
Event S.-BC	10	192	150	5e-5	0.05
CAMEO-PC MCC	40	32	128	5e-5	0.05
CAMEO-ST NER	60	32	128	5e-5	0.3
MUC4-NER	20	16	128	4e-5	0.1
Re3d-NER	25	16	128	4e-5	0.1
C Other detailed results
This section analyzes in a detailed manner the model's performance on certain datasets. Specifically, we analyze two rare cases where all ConfliBERT models outperform BERTs and where Cont models significantly outperform SCR models. Table 9 indicates how Cont significantly outperforms SCR in all performance metrics (p<0.05 for all metrics). Table 10 shows how Cont-cased beats all the other counterparts for classifying event status of
pieces of civil unrest. While there may be many factors, we postulate that some words in the original SCR-based vocabulary are accidentally good at tokenizing the out-of-domain text in Gun Violence, while that vocabulary is also good at classifying ongoing (OG) and future (FU) events.
Table 8: Hyperparameters for fine-tuning all the models in our evaluation experiments.
Tags	BERT uncased	cased	Confli.-Cont uncased	cased	Confli.-SCR uncased	cased
0-TRUE	85.50	86.53	91.21	91.39	87.53	87.23
1-FALSE	83.11	83.95	88.84	89.14	85.17	85.02
Micro F1	84.40	85.36	90.17	90.40	86.47	86.23
Macro F1	84.30	85.24	90.02	90.27	86.35	86.13
AUROC	90.13	91.19	94.63	95.45	92.54	92.87
AUPRC	88.30	89.86	94.95	95.76	92.03	92.07
Table 9: Gun Violence Binary Classification.
Tags	BERT uncased	cased	Confli.-Cont uncased	cased	Confli.-SCR uncased	cased
PA	88.38	87.28	89.27	89.53	88.92	88.63
OG	53.13	43.39	52.86	56.17	54.97	53.23
FU	70.45	70.77	77.82	79.40	73.76	70.94
Micro F1	79.47	77.49	81.28	82.29	80.48	79.87
Macro F1	70.65	67.15	73.32	75.03	72.55	70.94
MCC	56.67	51.45	60.40	62.71	59.74	57.40
Table 10: Event Status Temporal Status Classification.