SlowGuess/NAACL_2022_OCR huggingface dataset:
We used four V-100 GPUs with 32 GB memory to train each model. We used Adam optimizer (Kingma and Ba, 2015). The learning rate was warmed up over the first 10,000 steps to the peak value of 5e-4, and then linearly decayed. We pre-trained each SCR model for about 150K steps over the 7 billion word corpus. We followed Devlin et al. (2018) to train the model with a sequence length of 128 for  $80\%$  of the steps. Then, we trained the remaining  $20\%$  steps with a sequence of 512. The overall training time for each SCR model took about eight days. We trained Cont models the same as SCR models but in two fewer days because they were trained from intermediate checkpoints. See Appendix for more details.
4.2 Fine-Tuning Setup
Architecture. We followed the same architecture modification as BERT (Devlin et al., 2018) in the
downstream tasks. Our task mainly consists of classification and sequence labeling. The sequence labeling tasks predict the sequence of BIO tags for each token in the input sentence. The classification tasks require a sequence classification/regression head on top of the pooled output of BERT. We used cross-entropy loss for binary/multi-class classification. We used mean-square loss and set the discrimination thresholds as 0.5 in all the multi-label classification tasks.
Casing. Devlin et al. (2018) use the cased models for NER and the uncased models for all other tasks. However, other works report that uncased models perform slightly better than cased models in specific domains, even on NER tasks (Beltagy et al., 2019; Gu et al., 2021). Therefore, we evaluated both cased and uncased versions of all models.
Hyperparameters. Devlin et al. (2018) propose a hyperparameter tuning strategy relying on a grid-search on the ranges such as the number of training epochs  $\in \{3,4\}$ , and batch size  $\in \{16,32\}$ . However, this strategy for general domain benchmarks (e.g. GLUE (Wang et al., 2018)) has not been sufficiently justified in other datasets (Chalkidis et al., 2020). The optimal hyperparameters are highly dataset- and task-dependent in our tasks. For instance, the models may be underfitting after the suggested maximum of four epochs. Additionally, based on our observations from the conflict datasets (e.g., GTD, SATP, MUC-4, InSight Crime), ConfliBERT models converge to the best results faster than BERT. Therefore, to compare with BERT fairly, we used early stopping based upon the development dataset within a range of the maximum training epochs when all the models have achieved stable results. A more detailed description of other hyperparameters can be found in the Appendix. Finally, we repeated all the experiments ten times with different seeds.
5.1 Pre-training Results
We use perplexity (ppl) to measure how well the language models predict a masked token in an unseen test set. We sampled  $0.02\%$  of stories from each source during the data preparation, ending with an 8.62 MB held-out dataset representing our corpus's distribution. Table 4 shows the ppl of our models on the held-out dataset. We also list the values reported by the original models (Devlin et al.,
Dataset	Domain	Tasks	Train/dev/test	Metrics	BERT	Confli.-Cont	Confli.-SCR
uncased	cased	uncased	cased	uncased	cased
BBC	General	BC	1588/315/322	F1	97.24	96.38	97.9	96.95	98.08	98.13
20 News.	General	BC	9044/2270/7532	F1	80.30	79.58	80.4	80.51	81.05	80.37
Gun V.	Violence	BC	3387/423/423	macro F1	84.30	85.24	90.02	90.27	86.35	86.13
GLOCON	Protest	Sent BC	1549/193/193	macro F1	84.53	84.92	85.60	85.72	86.57	82.20
Doc BC	782/130/130	macro F1	88.97	84.61	89.76	89.97	91.13	88.27
GTD	Terrorism	MCC	2825/471/471	macro F1	83.55	82.05	81.97	83.23	83.82	83.16
SATP	Terrorism	BC	5956/744/745	F1	87.78	87.10	87.51	87.49	88.12	88.72
Rel MLC	1085/232/232	example F1	87.81	88.36	88.14	88.37	89.08	88.64
All MLC	4794/1192/1489	example F1	63.36	63.32	64.14	63.72	64.47	64.53
Insight C.	Crime	MLC	1002/332/319	example F1	68.57	67.83	69.09	69.15	68.68	69.47
India P.	Violence	Sent MLC	14943/3172/3276	example F1	64.89	64.54	63.03	63.40	67.27	66.22
Doc MLC	905/165/187	example F1	66.80	63.41	67.09	67.38	69.97	66.71
Event S.	Protest	TS MCC	1818/226/227	macro F1	70.65	67.15	73.32	75.03	72.55	70.94
BC	4010/500/501	F1	91.72	90.67	92.42	91.85	92.10	92.40
CAMEO	Politics	PC MCC	1348/224/225	macro F1	86.44	85.85	87.88	86.12	87.64	87.83
ST NER	1153/224/225	macro F1	72.29	72.25	74.00	74.45	74.35	72.87
MUC-4	Terrorism	NER	1300/200/200	macro F1	62.96	60.33	60.29	60.90	63.97	60.31
Re3d	Defence	NER	574/191/200	macro F1	63.44	62.46	64.40	66.20	66.40	64.23
Table 3: The datasets, tasks and summary results of our evaluation.
BERT uncased	Confli.-SCR uncased	cased	Confli.-Cont uncased	cased
ppl	3.99	3.14	3.14	3.40	2.93
Table 4: Perplexity on held-out training data by model.
2018). Low ppl scores indicate that our models have been sufficiently pre-trained and have better generalization on our corpora.
5.2 Fine-Tuning Results and Analysis
Table 3 reports the F1 scores for each task using the mean of 10 seeds. We have the below observations:
ConfliBERT's superiority over BERT. ConfliBERT provides additional improvement to the original BERT in our target domain. In Table 3, although the performance is task-, dataset- and casing- dependent, our models consistently report the best results (in bold). In Figure 1, we compare ConfliBERT SCR-uncased with the best results from both cased and uncased versions of BERT in each experiment. We use different colors to denote four p-value thresholds  $(p < 0.01$ ,  $p < 0.05$ ,  $p < 0.1$ , and  $p \geq 0.1)$  of statistical significance. SCR-uncased demonstrates superior performance across all the tasks, and the difference is statistically significant at  $p < 0.1$  in all but three. Specifically for GTD, we observed that SCR-uncased slightly beats the best BERT, but it still shows a significant
Figure 1: Significance test of SCR-uncased vs. the best of BERT models in each task, regardless of casing.
level of confidence, as depicted in Figure 1. We also observed that on InSight Crime, ConfliBERT achieves the best results in SCR-cased. Yet for SCR-uncased, the margin is not significant when compared with the best BERT in Figure 1. However, we conduct certain experiments on GTD and Insight Crime showing ConfliBERT's significant superiority when tackling limited training data in section 5.2.
Evaluating differences between the two pre
(a) GTD
(b) SATP-relevant
(c) Insight Crime
Figure 2: Performance vs. varying size of the training data.
training strategies, Cont and SCR, remains to be studied. Table 3 shows that SCR slightly beats Cont in most cases (13 out of 18 tasks), and SCR-uncased provides the most stable improvements over BERT among our four models. However, the performance is still dataset- and task-dependent. For example, Cont beats SCR significantly in Gun violence and Event Status. We present an in-depth analysis of these two cases in Appendix.
Effect of ConfliVocab One major difference between SCR and Cont is the use of in-domain vocabulary. Section 5.2 shows that both Cont and SCR outperform BERT significantly while SCR slightly beats Cont. We have discussed the substantial advantage of using ConfliVocab during the tokenization in Section 3.1. Besides the examples in Table 2, in ConfliVocab we also find terrorist groups and criminal organizations frequently mentioned in the reports of violence and crime. Examples include Boko Haram, Al Qaeda, Sinaloa Cartel, PCC, FARC, Mara, among others. On the other hand, the range of actor entities in the politics domain is much larger and sparser than terrorist and criminal organizations. Given that we have a more distinct in-domain vocabulary in the conflict domain, we expect a more significant benefit from ConfliVocab in the conflict domain instead of the general politics domain.
ConfliBERT requires less annotated data than BERT. ConfliBERT performs well with limited data in various conflict datasets. Figure 2 shows three groups of experiments on GTD, SATP, and Insight Crime, where we used varying training data sizes but the same valid and testing set as the original experiments respectively. We repeated each experiment with five seeds and plotted the average
metric scores.
Figure 2a shows that ConfliBERT beats BERT using limited size of GTD training data. Especially in the case of 1/32 size of GTD training data (88 examples), both SCR models still have  $69\%$  F1 scores, while BERT models drop to  $55\%$  F1 scores. In Figure 2b, we sampled various subsets of SATP-relevant, the SATP subset related to terrorist attacks. Results show that three of our models remain  $65\%$  to  $73\%$  F1 scores when using only 1/32 size of the training data (34 examples), while BERT drops to only  $44\%$  F1 scores. Finally, we also observe that both ConfliBERT SCR models significantly beat Cont and BERT models with a large margin on Insight Crime in Figure 2c.
These results show large improvements when using ConfliBERT with limited training data. Given the resources required to annotate data in conflict research, this is a particularly encouraging finding. These experiments also show that ConfliBERT outperforms BERT on GTD, SATP, and Insight Crime, strengthening the results in Figure 1.
This paper presents the development, application, evaluation, and further exploration of ConfliBERT, a pre-trained language model for political conflict and violence. The development of ConfliBERT rests on an unprecedented effort on three fronts. First, we collect and curate a large domain-specific corpus to support the pre-training process. Second, we conduct a comprehensive evaluation across several datasets and various NLP tasks of distinct nature and varying degrees of complexity.
The results show that ConfliBERT consistently outperforms BERT in the conflict and political violence domain. Furthermore, the biggest improve
ments are with limited training data, which conflict researchers often have due to the high costs of data annotation. In this way, ConfliBERT constitutes a valuable development that will contribute to a broad community of researchers in political science and policy sectors interested in tracking, analyzing, and predicting political violence and conflict on a global scale.
Due to limited time and computational resources, we did not conduct more experiments to explore various hyperparameters that could affect finetuning results, such as vocabulary size and pretraining epochs, to name a few. Future work should analyze how to optimize ConfliBERT, expand ConfliBERT to multi-lingual settings, and apply ConfliBERT to more challenging tasks such as understanding, inference, question answering, uncertainty qualification (Hu et al., 2021; Hu and Khan, 2021), and few-/ zero-shot tasks to speed up the study of NLP application for the political science community.
7 Ethical Impact
Our research considers several measures to mitigate concerns of bias in machine learning: (i) we implement standard social science practices to select corpora and training data (Barberá et al., 2021); (ii) for the pre-training stage, we gather a corpus with unprecedented global coverage to reduce regional biases; (iii) we move beyond the biases introduced from dictionary-based methods by using machine learning, as suggested by Wilkerson and Casas (2017); (iv) finally, we use multiple coders for the training data. However, copyright issues prevent us from sharing the raw data and hinder FAIR data principles (Wilkinson et al., 2016).
The broader goal of producing accurate and valid conflict data is to prevent or mitigate harm. These types of data provide a more objective means to understand and study conflict and armed violence. Our effort is an attempt to produce higher-quality data resources to serve this purpose.
The research reported herein was supported in part by NSF awards DMS-1737978, DGE-2039542, OAC-1828467, OAC-1931541, and DGE-1906630, ONR awards N00014-17-1-2995 and N00014-20-1-2738, Army Research Office Contract No. W911NF2110032 and IBM faculty award (Research).
Open Event Data Alliance. 2015. Petrarch: Python engine for text resolution and related coding hierarchy. http://www.github.com/openeventdata/petrarch.
Emily Alsentzer, John Murphy, William Boag, WeiHung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72-78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elghohary, Sergey Feldman, Vu Ha, et al. 2018. Construction of the literature graph in semantic scholar. NAACL HLT 2018, pages 84-91.
Giuseppe Attardi. 2015. Wikiextractor. https://github.com/attardi/wikiextractor.
Benjamin E Bagozzi, Daniel Berliner, and Ryan M Welch. 2021. The diversity of repression: Measuring state repressive repertoires with events data. Journal of Peace Research, 58(5):1126-1136.
Pablo Barbera, Amber E Boydstun, Suzanne Linn, Ryan McMahon, and Jonathan Nagler. 2021. Automated text classification of news articles: A practical guide. Political Analysis, 29(1):19-42.
Andreas Beger, Cassy L Dorff, and Michael D Ward. 2016. Irregular leadership changes in 2014: Forecasts using ensemble, split-population duration models. International Journal of Forecasting, 32(1):98-111.
John Beieler. 2016. Generating politically-relevant event data. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 37-42.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615-3620.
Brian Blankenship. 2020. Promises under Pressure: Statements of Reassurance in US Alliances. International Studies Quarterly, 64(4):1017-1030.
Doug Bond, Joe Bond, Churl Oh, Craig J. Jenkins, and Charles L. Taylor. 2003. Integrated Data for Events Analysis (IDEA): An Event Typology for Automated Events Data Development. Journal of Peace Research, 40(6):733-745.
Elizabeth Boschee, Jennifer Lautenschlager, Sean O'Brien, Steve Shellman, James Starz, and Michael Ward. 2016. ICEWS Coded Event Data.
Patrick T Brandt, Vito D'Orazio, Latifur Khan, Yi-Fan Li, Javier Osorio, and Marcus Sianan. 2022. Conflict forecasting with event data and spatio-temporal graph convolutional networks. International Interactions, pages 1-23.
Berfu Buyukoz, Ali Hürriyetoglu, and Arzucan Özgür. 2020. Analyzing ELMo and DistilBERT on sociopolitical news classification. In Proceedings of the Workshop on Automated Extraction of Socio-political Events from News 2020, pages 9-18, Marseille, France. European Language Resources Association (ELRA).
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898-2904. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pretraining text encoders as discriminators rather than generators. In ICLR.
Matthew J Connelly, Raymond Hicks, Robert Jervis, Arthur Spirling, and Clara H Suong. 2021. Diplomatic documents data for international relations: the freedom of information archive database. Conflict Management and Peace Science, 38(6):762-781.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
DSTL. 2018. Relationship and entity extraction evaluation dataset. https://github.com/dstl/re3d/. Accessed: 2021-07-01.
X. Du and Claire Cardie. 2020. Document-level event role filler extraction using multi-granularity contextualized encoding. In ACL.
Deborah J Gerner, Philip A Schrodt, Omür Yilmaz, and Rajaa Abu-Jabr. 2002. Conflict and mediation event observations (cameo): A new event data framework for the analysis of foreign policy interactions. International Studies Association, New Orleans.
Goran Glavas, Federico Nanni, and Simone Paolo Ponzetto. 2017. Cross-lingual classification of topics in political texts. In Proceedings of the Second Workshop on NLP and Computational Social Science, pages 42–46, Vancouver, Canada. Association for Computational Linguistics.
Derek Greene and Pádraig Cunningham. 2006. Practical solutions to the problem of diagonal dominance in kernel document clustering. In Proc. 23rd International Conference on Machine learning (ICML'06), pages 377-384. ACM Press.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1-23.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of ACL.
Andrew Halterman, Katherine Keith, Sheikh Sarwar, and Brendan O'Connor. 2021. Corpus-level evaluation for event qa: The indiapoliceevents corpus covering the 2002 gujarat violence. In Findings of the Association for Computational Linguistics: ACLIJCNLP 2021, pages 4240-4253.
Andrew Halterman and Benjamin J. Radford. 2021. Few-shot upsampling for protest size detection. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 3713-3720, Online. Association for Computational Linguistics.
Alex Hanna. 2017. Mpeds: Automating the generation of protest event data. Available at https://osf.io/preprints/socarxiv/xuqmv (2020/05/22). Unpublished Manuscript.
Yibo Hu and Latifur Khan. 2021. Uncertainty-aware reliable text classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 628-636.
Yibo Hu, Yuzhe Ou, Xujiang Zhao, Jin-Hee Cho, and Feng Chen. 2021. Multidimensional Uncertainty-Aware Evidential Neural Networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7815-7822.
Ruihong Huang, Ignacio Cases, Dan Jurafsky, Cleo Condoravdi, and Ellen Riloff. 2016. Distinguishing past, on-going, and future events: The EventStatus corpus. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 44-54, Austin, Texas. Association for Computational Linguistics.
Ali Hürriyetoglu, Hristo Tanev, Vanni Zavarella, Jakub Piskorski, Reyyan Yeniterzi, Deniz Yuret, and Aline Villavicencio. 2021. Challenges and applications of automated extraction of socio-political events from text (case 2021): Workshop and shared task report. In Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Sociopolitical Events from Text (CASE 2021), pages 1-9.
Ali Hürriyetoglu, Erdem Yürek, Deniz Yüret, Çağrı Yoltar, Burak Gürel, Fürat Durusan, and Osman Mutlu. 2019. A task set proposal for automatic protest information collection across multiple countries. In European Conference on Information Retrieval, pages 316-323. Springer.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimiciii, a freely accessible critical care database. Scientific data, 3(1):1-9.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. In NeurIPS.
Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Proceedings of the Twelfth International Conference on Machine Learning, pages 331-339.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics.
Patrick Lewis, Myle Ott, Jingfei Du, and Veselin Stoyanov. 2020. Pretrained language models for biomedical and clinical tasks: Understanding and extending the state-of-the-art. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 146-157. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969-4983, Online. Association for Computational Linguistics.
J. Lu and Joydeep Roy. 2017. Universal Petrarch: Language-agnostic political event coding using universal dependencies. Available at https://github.com/openeventdata/UniversalPetrarch (2020/05/22).
Roseanne W McManus. 2017. Statements of Resolve: Achieving Coercive Credibility in International Conflict. Cambridge University Press.
MUC-4. 1992. Fourth message understanding conference (muc-4). In Fourth Message Understanding Conference (MUC-4): Proceedings of a Conference Held in McLean, Virginia, June 16-18, 1992.
Clayton Norris, Philip Schrodt, and John Beieler. 2017. Petrarch2: Another event coding program. Journal of Open Source Software, 2(9):133.
Sean P. O'Brien. 2010. Crisis Early Warning and Decision Support: Contemporary Approaches and Thoughts on Future Research. International Studies Review, 12(1):87-104.
Fredrik Olsson, Magnus Sahlgren, Fehmi ben Abdesslem, Ariel Ekgren, and Kristine Eck. 2020. Text categorization for conflict event annotation. In Proceedings of the Workshop on Automated Extraction of Socio-political Events from News 2020, pages 19-25, Marseille, France. European Language Resources Association (ELRA).
Faik Kerem Örs, Süveyda Yeniterzi, and Reyyan Yeniterzi. 2020. Event clustering within news articles. In Proceedings of the Workshop on Automated Extraction of Socio-political Events from News 2020, pages 63-68, Marseille, France. European Language Resources Association (ELRA).
Javier Osorio and Alejandro Reyes. 2017. Supervised Event Coding From Text Written in Spanish: Introducing Eventus ID. Social Science Computer Review, 35(3):406-416.
Javier Osorio, Alejandro Reyes, Alejandro Beltrán, and Atal Ahmadzai. 2020. Supervised event coding from text written in Arabic: Introducing hadath. In Proceedings of the Workshop on Automated Extraction of Socio-political Events from News 2020, pages 49-56, Marseille, France. European Language Resources Association (ELRA).
Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. English gigaword fifth edition ldc2011t07. Linguistic Data Consortium, Philadelphia.
Erick Skorupa Parolin, Mohammadsaleh Hosseini, Yibo Hu, Latifur Khan, Javier Osorio, Patrick T Brandt, and Vito D'Orazio. 2022. Multi-CoPED: A Multilingual Multi-Task Approach for Coding Political Event Data on Conflict and Mediation Domain. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society.
Erick Skorupa Parolin, Yibo Hu, Latifur Khan, Javier Osorio, Patrick T Brandt, and Vito D'Orazio. 2021a. CoMe-KE: A New Transformers Based Approach for Knowledge Extraction in Conflict and Mediation Domain. In 2021 IEEE International Conference on Big Data (Big Data), pages 1449–1459. IEEE.
Erick Skorupa Parolin, Latifur Khan, Javier Osorio, Patrick Brandt, Vito D'Orazio, and Jennifer Holmes.