SlowGuess/NAACL_2022_OCR huggingface dataset:
Yibo Hu†, MohammadSaleh Hosseini†, Erick Skorupa Parolin†, Javier Osorio§, Latifur Khan†, Patrick T. Brandt†, Vito J. D'Orazio†
†The University of Texas at Dallas, §The University of Arizona
{yibo.hu,seyyedmohammadsaleh.hosseini,erick.skorupaparolin, lkhan,pbrandt,dorazio}@utdallas.edu,josorio1@email.arizona.edu
Analyzing conflicts and political violence around the world is a persistent challenge in the political science and policy communities due in large part to the vast volumes of specialized text needed to monitor conflict and violence on a global scale. To help advance research in political science, we introduce ConfliBERT, a domain-specific pre-trained language model for conflict and political violence. We first gather a large domain-specific text corpus for language modeling from various sources. We then build ConfliBERT using two approaches: pre-training from scratch and continual pretraining. To evaluate ConfliBERT, we collect 12 datasets and implement 18 tasks to assess the models' practical application in conflict research. Finally, we evaluate several versions of ConfliBERT in multiple experiments. Results consistently show that ConfliBERT outperforms BERT when analyzing political violence and conflict. Our code is publicly available. $^{1}$
The study of political violence is a central concern of conflict scholars and security analysts in the academic and policy communities. For decades, scholars and governments have devoted incalculable resources to monitoring, understanding, and predicting the dynamics of social unrest, political violence, and armed conflict worldwide. Conflict research is a sub-field of political science that analyzes a broad scope of interactions between government agents, their challengers, and the civilian population, including material and verbal conflict and cooperation. Conflict research covers protest, riots, repression, insurgency, civil war, terrorism, human rights, genocide, criminal violence, forced displacement, conventional and unconventional warfare, nuclear deterrence, peacekeeping, diplomatic disputes and cooperation, among others.
Traditionally, researchers used manual coding to track conflict processes worldwide (Raleigh et al., 2010). Unfortunately, the high costs and slow pace of domain experts conducting these tasks make it extraordinarily difficult and costly to monitor highly complex and rapidly changing conflicts in an ever-growing volume of information available on a global scale. Furthermore, these efforts tend to focus on quantifying particular types of conflict events between specific kinds of actors (Sundberg and Melander, 2013).
Initial efforts to address these challenges motivated political scientists to develop automated systems to classify or extract structured event data from news articles (Bond et al., 2003; Boschee et al., 2016; O'Brien, 2010; Osorio and Reyes, 2017; Schrodt, 2006, 2009; Alliance, 2015; Norris et al., 2017; Lu and Roy, 2017; Ward et al., 2013). These systems capture a broader range of event types, including different conflict and cooperation events, between a larger number of political actors. They can also extract volumes of data that are orders of magnitude greater than manual coding efforts. Automated event data such as the Integrated Crisis Early Warning System have been used for conflict forecasting and other kinds of research in political science (Bagozzi et al., 2021; Beger et al., 2016; Brandt et al., 2022).
However, these existing systems rely on dated pattern matching techniques and large dictionaries, which often yield low-accuracy results and are too costly to maintain. Recent efforts by political scientists employ traditional machine learning (Hanna, 2017; Osorio et al., 2020) and deep learning (Beieler, 2016; Radford, 2020b; Glavaš et al., 2017; Skorupa Parolin et al., 2020) to analyze political conflict and violence. Standard supervised learning requires labeled data, which are expensive to obtain due to the expertise required for quality annotation. This led conflict scholars to seek alternative solutions based on the latest developments
https://github.com/eventdata/ConfliBERT
Human Language Technologies, pages 5469 - 5482
in natural language processing (NLP).
Recent progress in NLP has been driven by pre-trained transformer language models (Vaswani et al., 2017; Radford et al., 2019; Devlin et al., 2018; Yang et al., 2019). Self-supervision using large-scale unlabeled text can significantly alleviate the annotation bottleneck using transfer learning. The training parallelization of transformers also improves their efficiency on large datasets. As a result, the use of powerful computational devices and the advantage of transformer structures make large-scale language models' pre-training possible. Furthermore, the introduction of extensive benchmarks (Wang et al., 2018, 2019; Rajpurkar et al., 2018; Lai et al., 2017) validates the significant improvement of pre-trained language models on various downstream tasks.
While many language models are built on general domain corpora, such as Wikipedia, BookCorpus (Zhu et al., 2015), and WebText (Radford et al., 2019), recent works show that pre-training on domain-specific corpora can boost downstream performance on those domains (Lee et al., 2019; Gururangan et al., 2020). Domain-specific work in bio-medicine focuses not only on developing pre-trained models (Lee et al., 2019; Beltagy et al., 2019; Alsentzer et al., 2019; Lewis et al., 2020; Gu et al., 2021) but also on proposing domain-relevant evaluation benchmarks (Peng et al., 2019; Gu et al., 2021). Pre-training models also have advanced research in other domains such as academic papers (Beltagy et al., 2019) and legal studies (Chalkidis et al., 2020). Despite some efforts to apply transformers-based approaches in political science (Buyukoz et al., 2020; Olsson et al., 2020; Ors et al., 2020; Radford, 2020a; Halterman and Radford, 2021; Hürriyetoglu et al., 2021; Parolin et al., 2021a, 2022), we are unaware of any studies that develop and evaluate domain-specific pretrained language models for political science or conflict research.
By combining the expertise of conflict scholars and computer scientists, we developed ConfliBERT, a pre-trained language model designed for conflict and political violence. ConfliBERT improves downstream tasks for conflict research while significantly alleviating the annotation bottleneck. We expect it to support a broad community of academic and policy researchers, enabling the analysis of conflict processes using a domain-specific NLP tool that yields accurate and valid
results at minimum operational cost. Our paper provides the following key contributions: (1) We curate a large domain-specific corpus for language modeling in the domain of political violence, conflict, cooperation, and diplomacy. (2) Based on our domain-specific corpora, we devise a pre-trained language model, ConfliBERT, and make it available to the general public, which directly benefits the political science and policy communities. (3) To evaluate our model in practical applications, we collect 12 datasets and conduct 18 tasks relevant to conflict research. We are the first to carry out such a comprehensive evaluation of language models for conflict studies. (4) We evaluate different versions of ConfliBERT and show it outperforms models trained on generic domains. We also perform in-depth analyses of different tasks to investigate the factors affecting the performance.
Recent pre-trained transformer language models, such as the Bidirectional Encoder Representation from Transformers (BERT) (Devlin et al., 2018), follow a two-steps framework: (1) pre-train on a large unlabeled corpus; and (2) fine-tune on task-specific labeled data. These models learn semantics during the pre-training step and require smaller labeled data to significantly improve their performance on downstream tasks. The fine-tuning step requires minor network modifications to create state-of-the-art models for various tasks.
Technically, BERT uses the multi-layer, multi-head self-attention mechanism, which provides substantial advantages for language modeling, such as allowing parallel GPU computation and capturing long-range dependencies. This allows to efficiently pre-train large language models on large corpora using powerful devices.
Another key element of BERT-like models is the design of self-supervision tasks. Self-supervision refers to generating labels for unlabelled data and using them to train a model in a supervised manner. BERT uses two self-supervision tasks during pre-training. On the one hand, masked language model (MLM) is a fill-in-the-blank task based on randomly masking a token and then using the surrounding words to predict the word hidden behind the mask. On the other hand, next sentence prediction (NSP) determines whether one sentence follows another one in the same document.
Recent works also propose variants of self
supervision tasks. For example, Joshi et al. (2020) mask out contiguous sequences of tokens to improve span representations. Clark et al. (2020) use replaced token detection, where the model distinguishes real input tokens from plausible but synthetically generated replacements. However, Liu et al. (2019) prove that MLM is competitive with other recently proposed training objectives with more data and improved training strategies.
Finally, most BERT-like models focus on a generic domain, such as Wikipedia, BookCorpus (Zhu et al., 2015) and WebText (Radford et al., 2019). However, BERT without domain adaptation tends to underperform in target domains with distinct characteristics such as specialized vocabulary, language style, and specific semantics. Our domain includes political violence, armed conflict, international cooperation, and diplomacy—all of which have these characteristics. This performance gap is the primary motivation for developing domain-specific language models.
Specifically, the language of political actors involves strategic and complex semantics. Policy positions that show support for one actor while also threatening another are sometimes embedded in simple statements. For example, "NATO will not tolerate this aggression" mixes a negation, a conditional, and the action of potential interest. Signals are highly context-dependent, adapted for a target audience, and vary in strength depending on the specific actor sending the signal (McManus, 2017; Blankenship, 2020). Compared to a generic language model, we expect ours to incorporate important contextual information and learn more about the strategic ways that political actors convey information. This context and the related political biases in it are exactly a need that ConfliBERT aims to fulfill. Political conflict and violence text gains from this domain knowledge: one political actor's definition of "rebels" is another's "freedom fighters".
Table 1 summarizes various recent domain-specific BERT models, including our model, ConfliBERT. These models mainly differ in their corpora and pre-training strategies, including: (1) continuing pre-training (Cont); and (2) pre-training from scratch (SCR). In the next section, we elaborate on the strategies and our method of developing ConfliBERT in the domain of political conflict and violence.
Model	Method	Corpora and Text Size
BERT	-	Wiki+Books, 3.3B words/16 GB
BioBERT	Cont	PubMed, 4.5B words
SciBERT	SCR	BIO+CS papers, 3.2B words
BlueBERT	Cont	PubMed+MIMIC, 4.5B words
PubBERT	SCR	PubMed, 3.1B words/21 GB
LegalBERT	Both	legislation, court cases, 12 GB
ConfliBERT	Both	organization/government reports, news, 7B words/34 GB
Table 1: Summary of selected BERT models in general and specific domains.
As described in Section 2, MLM-based BERT achieves competitive performance among other transformer models with different self-supervision tasks. Besides, BERT has been validated in various domains (Lee et al., 2019; Beltagy et al., 2019; Peng et al., 2019; Chalkidis et al., 2020; Gu et al., 2021) shown in Table 1. Therefore, we develop our domain-specific model based on BERT. The key components of developing and validating our model, ConfliBERT, include pre-training strategies, corpora, and evaluation tasks.
3.1 Domain-specific Pretraining
We explore both strategies (SCR and Cont) of adapting BERT to the political conflict and violence domain. A Cont model initializes with BERT's checkpoint and vocabulary, and trains for additional steps on a domain-specific corpus. Since BERT has already been pre-trained about one million steps on the generic domain, Cont usually requires fewer steps than training a new model from scratch. For example, Lee et al. (2019) report that continual pre-training of BERT on a biomedical dataset for 470K steps yields comparable performance to pre-training for one million steps.
On the other hand, when pre-training BERT from scratch (SCR) on the domain-specific corpora, we generate a new vocabulary from the target domain instead of using the original BERT's vocabulary. Various papers (Beltagy et al., 2019; Gu et al., 2021) argue that SCR generates substantial gains over Cont for domains with sizeable unlabeled text.
We refer to the original BERT vocabulary as BaseVocab and our domain vocabulary as ConfliVocab. We generated both cased and uncased versions of ConfliVocab on our training corpus using the Wordpiece algorithm (Wu et al., 2016). We set the ConfliVocab size to 30,000 words to
Words	BERT	ConfliBERT
Daesh	Da-esh	Daesh
extremists	ex-tre-mist-s	extremists
FARC	FA-RC	FARC
IED	I-ED	IED
indiscriminately	in-dis-c-rim-inate-ly	indiscriminately
manhunt	man-hun-t	manhunt
mutilation	m-uti-lation	mutilation
paramilitaries	para-mi-lit-aries	paramilitaries
perpetrator	per-pet-rator	perpetrator
punitive	pu-ni-tive	punitive
racketeering	rack-ete-ering	racketeering
separatists	se-par-ati-sts	separatists
subversive	sub-vers-ive	subversive
undemocratic	und-em-oc-ratic	undemocratic
xenophobic	x-eno-phobic	xenophobic
Table 2: Examples of common terms in conflict domain.
match that of BaseVocab. The resulting token overlap between BaseVocab and ConfliVocab is  $58.3\%$ , which indicates a considerable difference  $(41.7\%)$  in high-frequency words between the general and conflict-specific corpora.
In particular, we find a substantial advantage of using ConfliVocab during the tokenization. Table 2 shows examples of conflict-related terms that exclusively appear in ConfliVocab. For example, the term "separatists" is not included in BaseVocab, and BERT erroneously splits it into four sub-words ["se", "#par", "#ati", "#sts"]. This fragmentation may hinder learning in downstream tasks. We will validate the advantage of ConfliVocab in the downstream tasks in our experiments section.
3.2 Corpora
The first step to develop ConfliBERT is to build a domain-specific corpus for pre-training. As illustrated in Table 1, there exist large-scale publicly available biomedical datasets, such as PubMed and MIMIC (Johnson et al., 2016). SciBERT (Beltagy et al., 2019) is built from a large corpus of academic papers (Ammar et al., 2018; Lo et al., 2020). History Lab<sup>2</sup> provides many government documents, but lacks the breadth we need for the politics and conflict domain (Connelly et al., 2021). Thus we curated a domain corpus that consists of 33.7 GB of clean, plain text in the BERT required format. We bin the sources into five categories below and provide more details in Appendix.
Expert Domain Corpora (EDC). We curated 2,293 MB of plain text from multiple professional sources relevant to conflict and diplomacy.
The sources include United Nations' websites and databases, international humanitarian nongovernmental organizations, think tanks, and government sources such as the Foreign Relations of the United States. These are examples of objective records of government and diplomatic activity from non-partisan observers.
Mainstream Media Collection (MMC). We crawled 35 worldwide news agencies reporting in English and with coverage from 1966 to 2021. We pre-processed and filtered 20 GB of stories using metadata such as document tags for War and Politics. These cover a period during and after the Cold War with global coverage that focuses on primarily state-based conflict.
Gigaword. This corpus provides a distinct coverage of seven international English newswires from 1994 to 2010 (Parker et al., 2011). We removed the overlapping stories (which also existed in MMC) and filtered an 8,818 MB domain-specific subset.
Phoenix Real-Time (PRT). PRT is a developing event dataset crawled from more than 400 news agencies worldwide from October 2017 (Salam et al., 2018). It contains many news agencies in areas other than Europe and the U.S., thus improving the scope of our coverage. We removed the duplicated news agencies (which also existed in MMC and Gigaword) and filtered a 2,425 MB relevant subset. This allows the capture of post-Cold War actors, the Global War on Terrorism Service Medal (GWOT), and more recent events.
Wikipedia. Wikipedia has a different language style for describing political events and can enrich the diversity of our corpus. Based on its category labels, we curated 2,845 MB of relevant articles from an 18 GB size of the Wikipedia dump released on March 20, 2021.
3.3 Evaluation Tasks
The introduction of comprehensive benchmarks accelerated the development of pre-trained language models in the general NLP domain (Wang et al., 2018, 2019; Rajpurkar et al., 2018; Lai et al., 2017) and biomedical applications (Peng et al., 2019; Gu et al., 2021). However, few comprehensive benchmarks exist for evaluating language models in the political conflict and violence domain. The focus of political science professionals is different from that of general NLP researchers. For example, they
$^{2}$ http://history-lab.org
are more interested in classifying, tracking, and predicting conflict events from the text.
To conduct a comprehensive evaluation of ConfliBERT, we collected a broad range of NLP tasks related to political conflict and violence from both publicly available and our newly-annotated datasets. Table 3 shows the datasets and their corresponding tasks. Some datasets may contain subsets and are related to various tasks. The table also lists the number of examples in the training, development, and test datasets as well as the evaluation metrics used for each task. In particular, we use F1 scores as performance metrics for binary classification tasks. We use example-based F1 metrics for multi-label classification tasks (Sorower, 2010). For all the other tasks, we rely on Macro F1 to assess the model's performance. Next, we describe the datasets and their tasks.
Binary classification (BC). We collected BBC News (Greene and Cunningham, 2006) and 20 Newsgroups (Lang, 1995) for identifying political news, a subset from Gun Violence Database (Pavlick et al., 2016) for finding articles related to gun violence. We also used the samples from Global Contention Politics Dataset (GLOCON) (Hurriyetoglu et al., 2019) to conduct one sentence-level and one document-level classification task to predict whether the story is related to protests. These BC tasks are essential for political scientists as a first step to classify and filter documents containing political and conflict events from large-scale news wires.
Multi-class classification (MCC). GTD refers to Global Terrorism Database which collects terrorist incidents from 1970 onward (START, 2019). We sampled a subset with description text longer than 40 words and single labels to classify 9 types of attacks such as bombing/explosion, armed assault, and hostage-taking.
India Police Events (Halterman et al., 2021) consists of sentences from English-language Times of India articles about police activity events in Gujarat during March 2002 (a relevant period due to widespread Hindu-Muslim violence). The labels are available for both document and sentence levels and consider five categories of police activity: kill, arrest, fail to act, force, and any action.
Event Status includes English news articles about civil unrest events annotated with temporal tags (Huang et al., 2016). Following the original
setting, we conduct a temporal status classification (TS MCC) to detect the primary temporal distinctions among past, ongoing, and future. Besides, we also build a BC task of predicting if the story contains civil unrest events.
Multi-label classification (MLC). SATP stands for South Asia Terrorism Portal<sup>3</sup> from which we manually annotated a sample of 7,445 narratives between 2011 and 2019. We focus on incidents initiated by terrorist organizations.  $23.6\%$  of the sample are relevant stories classified into one or more categories: armed assault, bombing/explosion, kidnapping, and others. The rest samples are irrelevant (stories not about terrorism attacks such as arrests or armed clashes). Based upon this, we built three tasks. The first is a BC task to find relevant stories. The second is an MLC task to predict attack types on the relevant subset (Rel MLC). The third is the same as the second but conducted on the more imbalanced full dataset (All MLC).
InSight Crime (Parolin et al., 2021b) contains annotated stories about organized criminal activity in Latin America and the Caribbean from InSight Crime.4 We applied an MLC task to predict multiple crime categories expressed in the stories, such as drug trafficking, corruption, and law enforcement.
Sequence Labeling or Named Entity Recognition (NER). MUC-4 consists of documents reporting terrorism events, annotated with entities such as Perpetrator Individuals, Perpetrator Organizations, Physical targets, Victims, and Weapons (MUC-4, 1992). We split the dataset following Du and Cardie (2020).
Re3d stands for Relationship and Entity Extraction Evaluation Dataset (DSTL, 2018), comprising task-specific documents focused on the topic of the conflict in Syria and Iraq. The data contains annotations in span format with their corresponding entity types: Organization, Weapon, Military platform, Person, among others.
CAMEO (Conflict and Mediation Event Observations) is the industry standard for event extraction in political science (Gerner et al., 2002). An event classification, known as pentacode, consists of five event types: 0-Make a Statement, 1-Verbal Cooperation, 2-Material Cooperation, 3-Verbal Conflict, and 4-Material Conflict, and spans
<sup>3</sup>https://satp.org
4https://insightcrime.org
of texts containing sources (who conducted the action) and targets (to whom the action was conducted). We formulated two tasks for CAMEO event extraction on our newly-annotated dataset: sources and targets labeling (ST NER), and penta-code classification (PC MCC).
4.1 Pre-training Setup
We implemented ConfliBERT using two methods, Cont and SCR. Each approach has an uncased and a cased version. The architecture is the same as BERT-Base with 12 layers, 768 hidden units, 12 attention heads, and 110M parameters in total. Specifically, for our Cont models, we ran additional pre-training steps of the released checkpoints of BERT-Base models on our domain-specific corpus. The vocabulary is the same as the original BERT's vocabulary. For our SCR models, we use an indomain vocabulary, ConfliVocab (See Section 3.1 for more details).
We discarded the next sentence prediction (NSP) task. We found that the predicted NSP accuracy quickly reached  $90\%$  in the middle of our training, which indicated that NSP might be less challenging for the model to learn in our domain. However, learning NSP simultaneously affected the speed of optimization of masked language models (MLM) loss. Following many recent works discarding NSP (Lample and Conneau, 2019; Yang et al., 2019; Joshi et al., 2020; Liu et al., 2019) and our observation, we optimized MLM only.