Leveraging Codebook Knowledge with NLI and ChatGPT for Zero-Shot Political Relation Classification (2024)

Abstract: Is it possible accurately classify political relations within evolving event ontologies without extensive annotations? This study investigates zero-shot learning methods that use expert knowledge from existing annotation codebook, and evaluates the performance of advanced ChatGPT (GPT-3.5/4) and a natural language inference (NLI)-based model called ZSP. ChatGPT uses codebook’s labeled summaries as prompts, whereas ZSP breaks down the classification task into context, event mode, and class disambiguation to refine taskspecific hypotheses. This decomposition enhances interpretability, efficiency, and adaptability to schema changes. The experiments reveal ChatGPT’s strengths and limitations, and crucially show ZSP’s outperformance of dictionary-based methods and its competitive edge over some supervised models. These findings affirm the value of ZSP for validating event records and advancing ontology development. Our study underscores the efficacy of leveraging transfer learning and existing domain expertise to enhance research efficiency and scalability. The code is publicly available1.

heading: 1 Introduction
text: Event coding is a crucial task in political violence research for both academic and policy communities. It transforms unstructured text from news articles into structured event data, represented as source-action-target triplets, achieved through entity extraction and relation classification. It provides a structured record of interactions among political actors and serves as input for monitoring, understanding, and forecasting political conflicts and mediation processes worldwide (Schrodt and Gerner, 1996; Schrodt et al., 2003, 2004; Schrodt, 1997, 2006a, 2011; Shellman and Stewart, 2007; Shearer, 2007; Brandt et al., 2011, 2013, 2014).
1https://github.com/snowood1/Zero-Shot-PLOVER
However, manually coding events from extensive datasets is labor-intensive.
To streamline this, experts have developed event ontologies and knowledge bases (McClelland, 1978; Azar, 1980; Gerner et al., 2002; Bond et al., 2003; Schrodt, 2006b; Boschee et al., 2016; Lu and Roy, 2017; Osorio and Beltrán, 2020; Osorio et al., 2019). Yet, traditional pattern-matching models based on static dictionaries suffer from inflexibility, low recall, and high maintenance costs. Recent advancements in deep learning and pretrained language models (PLMs) offer promising supervised learning solutions (Glavaš et al., 2017; Büyüköz et al., 2020; Olsson et al., 2020; Örs et al., 2020; Parolin et al., 2020, 2021, 2022b; Hu et al., 2022a). Yet, their reliance on extensively annotated datasets introduces significant challenges, especially for in-depth and subnational studies requiring nuanced categorization and non-exclusive labeling within political event ontologies. Moreover, labeled datasets lack flexibility and may require frequent relabeling as ontologies evolve. Thus, much of the current PLM-based research in event coding targets broad, coarse-grained categorizations, often constrained by limited evaluation sets. Costs, time, and effort associated with developing training data have foiled the large-scale adoption and ready deployment of PLMs by government security agencies, researchers, and practitioners in need of monitoring and understanding rapidly-changing conflict processes around the world.
In light of these challenges, we pose the following questions: (1) Is it possible to leverage existing expert knowledge to enhance the efficiency of event coding without extensive annotation of new data? (2) Is it possible to create an interpretable and adaptable system that easily accommodates ontology or schema changes?
To tackle these questions, our paper focuses on relation classification, a key aspect of event coding. The goal is to categorize events in a source-
583
target pair following a predefined event ontology PLOVER (Open Event Data Alliance, 2018) without external labeled data. We achieved this by combining the transferred semantic knowledge of PLMs with expertise derived from annotation codebooks. The codebook, as depicted in Figure 2, contains label descriptions and guidelines for resolving confusing labels. To unlock this knowledge, we explore two zero-shot methods: the emerging ChatGPT (GPT-3.5/4), and our proposed NLI-based model called ZSP (Zero-Shot fine-grained relation classification model for PLOVER ontology).
While GPT-4 showcases notable improvements over GPT-3.5, it still exhibits instability in finegrained tasks, promising further enhancement. Conversely, ZSP, despite being built upon a smaller model, offers substantial advantages. It leverages easily constructed hypotheses from the codebook and employs a tree-query framework to capture nuanced semantics and mode distinctions within a focused set of hypotheses at each level. Additionally, ZSP’s adaptability allows straightforward updates by modifying the hypothesis table or class disambiguation rules to align with evolving ontologies. This approach proves more cost-effective than maintaining extensive dictionaries or re-labeling datasets for event record validation.
In sum, the untapped potential of GPT-4 and the success of ZSP encourage experts to reevaluate the value of existing knowledge bases and inspire innovative uses of this knowledge to expedite research within the political science community.
Obama said he won't provide military aid to Israel. Source: Obama-USAGOV Target: Israel-ISR Action: 1222-Reject request for military aid Root.: 12- REJECT Quad.: 3. V-Conf.
Other event modes -------------------- Root. ----- Quad. - Obama halted military aid to Israel. SANCTION 4. M-Conf. Obama provided military aid to Israel. AID 2. M-Coop. Obama agreed to provide aid to Israel. AGREE 1. V-Coop.
Figure 1: Event coding illustration: How event modes affect Rootcode and Quadcode labeling for sentences involving identical entities.

heading: 2 Preliminaries

heading: 2.1 Event Coding, Ontology, and Mode
text: The ontology of the event coding system defines how to code the actors (McClelland, 1978; Azar, 1980; Jones et al., 1996; Bond et al., 2003; Raleigh et al., 2010; Mitamura and Hovy, 2015; Boschee et al., 2016). One prominent schema is CAMEO (Gerner et al., 2002), which incorporates knowledge from the codebook2, action-pattern dictionaries, and actor dictionaries. It categorizes political interactions into 200+ fine-grained 4-digit codes (01XX–20XX). These are then aggregated into 20 more frequently utilized Rootcodes (01–20), and further into 4 high-level Quadcodes: 1-Verbal Cooperation, 2-Material Cooperation, 3-Verbal Conflict, and 4-Material Conflict. Later, the PLOVER scheme (Open Event Data Alliance, 2018) simplifies CAMEO by removing 4-digit codes, reducing Rootcodes to 16, and enhancing semantic clarity. Rootcode and Quadcode overviews of CAMEO/PLOVER are presented in Table 1, with a codebook snippet in Figure 2. Appendix Table 17 and the codebook present the Rootcode in detail.
Figure 1 illustrates an event coding scenario where the interaction between the source (Obama representing the USA government) and the target (Israel, coded as ISR) is classified using lowerlevel 4-digit codes as well as higher-level Rootcodes and Quadcodes. However, the classification is highly sensitive to subtle variations in what we term event mode—whether Obama has provided, plans to provide, has stopped, or intends to stop military aid—resulting in significant adjustments to Rootcodes and Quadcodes for identical entities.
PLOVER’s codebook3 suggested the concept of event mode through auxiliary modes—historical, future, hypothetical, or negated—to flexibly depict
2https://eventdata.parusanalytics.com/cameo. dir/CAMEO.Manual.1.1b3.pdf
3https://github.com/openeventdata/PLOVER/blob/ master/PLOVER_MANUAL.pdf
event statuses and assist in labeling. However, it did so without providing strict definitions or implementations. Inspired by this, we refine these event modes and integrate them into our NLI system to enhance classification accuracy. This refinement process and its impact on enhancing event coding precision are detailed in Section 3.2 and further elaborated in Appendix A.
The shift from the dictionary-based CAMEO to the more semantically friendly PLOVER aligns with the domain’s broader trend. Our focus on PLOVER is the result of careful consideration and validation with domain experts.

heading: 2.2 Related Work
text: Relation or event extraction has been studied across various domains (Hendrickx et al., 2019; Zhang et al., 2017; Han et al., 2018; Du and Cardie, 2020; Luan et al., 2018; Riedel et al., 2010; Fincke et al., 2022), with some studies partially overlapping in entities or categorizations relevant to political science (Doddington et al., 2004; Ebner et al., 2020; Li et al., 2021). However, our work distinguishes itself by considering event modes, a dimension not fully explored in existing works.
Our work also relates to zero-shot learning across various schemes (Huang et al., 2018; Obamuyide and Vlachos, 2018; Yin et al., 2019; Meng et al., 2020; Geng et al., 2021; Lyu et al., 2021; Sainz et al., 2021), especially socio-political event classification (Hürriyetoğlu et al., 2021; Radford, 2021; Barker et al., 2021; Haneczok et al., 2021; Halterman and Radford, 2021). However, many works focus on sentence-level classification rather than relations between multiple entity pairs. The others with complex templates cannot be adapted to our political ontology easily. Thus, we design our framework to efficiently integrate with the existing knowledge base.
Finally, recent large language models (LLMs) (Brown et al., 2020; Ouyang et al., 2022; OpenAI, 2022, 2023) have greatly advanced zero-shot learning in reasoning and text generation (Hu et al., 2022b; Halim et al., 2023; Jin et al., 2024). However, the application of ChatGPT for zero-shot event extraction remains underexplored and lags behind advanced supervised methods (Yuan et al., 2023; Cai and O’Connor, 2023; Li et al., 2023; Gao et al., 2023; Aiyappa et al., 2023). We will evaluate ChatGPT on PLOVER as part of our investigation.

heading: 3 Approach
text: We start by discussing the discovery of NLI as a potential solution and the construction process of the NLI-based ZSP framework, followed by the deployment of ChatGPT.

heading: 3.1 Limitations of NLI for Event Coding
text: NLI measures how likely a premise entails a hypothesis (Bowman et al., 2015; Williams et al., 2017). Initially, we explores the feasibility of using NLI to assign PLOVER codes by selecting the most probable entailed hypothesis from a set of candidates. We designed a tiny experiment with only 18 hypotheses derived from the Rootcode names4.
Table 2 illustrates three example hypotheses, where <S> and <T> denote the source (Indonesian students) and the target (President Suharto’s government), respectively. The labeled premise “THREATEN 3” indicates the intention to initiate protests. Notably, NLI accurately recognizes AID as contradictory and identifies REQUEST and PROTEST as entailments. Moreover, the tiny NLI model with only 18 hypotheses surpasses dictionary-based methods that rely on 81k verb patterns, with a remarkable macro F1 increase of 17.1% for Quadcode classification, thus comfirming the NLI potential as a valuable solution.
However, upon closer examination, we find that NLI models measuring semantic entailment may not directly suit our classification task, as the best entailed hypotheses do not always match our desired labels. The adaptation raises two key issues:
First, NLI disregards event mode. In Table 2, the premise labeled as THREATEN stands for a hypothetical, verbal protest event. NLI partially captures the event’s context (PROTEST) but fails to consider its mode. To address this, we incorporate mode information to enhance candidate precision.
Second, event category labels lack mutual exclusivity in semantics. In Table 2, the premise correctly entails both PROTEST and REQUEST with high scores from the semantic aspect. However, in CAMEO/PLOVER’s single-label schema, the context “demonstrate to demand reforms” aligns with PROTEST, a Material Conflict, rather than REQUEST, a Verbal Conflict. An easy solution is to prioritize “protest” over “request” when encountering “protest to request”, following the codebook’s disambiguation rules illustrated in Figure 2.
4See more details about this “Tiny model” experiment in Section 4.4, and Rootcode names in Table 1).
In summary, we identify three key dimensions to ensure accurate predictions: Context, Mode, and Class Disambiguation. Firstly, we narrow down predictions to the top candidates, PROTEST and REQUEST. Secondly, we incorporate mode information and identify the event as future, verbal, or hypothetical. Lastly, we apply the class disambiguation rule, giving precedence to PROTEST over REQUEST. By combining these dimensions, we achieve the final correct answer THREATEN. These findings motivate our NLI-based framework in Figure 2. Next, we provide detailed explanations for each component.

heading: 3.2 Enabling NLI to Classify Event Mode
text: NLI’s inability to accurately determine the event mode often leads to misclassification. In Table 3, we present a sentence with reversed labels compared to another in Table 2. Although labeled as AGREE for Verbal Cooperation, indicating a willingness to mitigate dissent, it incorrectly scores high (76.4%) for the hypothesis “protested against”. Semantically, this isn’t entirely wrong, as “agreed to ease protests” implies prior protests, but it suggests cooperation rather than the implied conflict of the hypothesis.
To address this, we introduce mode-aware hypotheses that incorporate four event modess, adapted from PLOVER’s guidelines: Past (P) for historical events or events that initiated or are ongoing, Future (F) for future, verbal, or hypothetical events, Contradict_Past (CP) and Contradict_Future (CF) for their respective contradictions. See examples in Table 3.
Labels for each mode are directly adopted from the codebook, requiring no new definitions. For example, PROTEST’s CF is labeled as AGREE, mirroring “03- EXPRESS INTENT TO COOPERATE” Rootcode, specifically item “0352- Express intent to ease popular dissent.” Likewise, the CP events like “reduced protests against” are labeled as YIELD, following CAMEO code 0833, which signifies yielding to demands.
To facilitate this nuanced classification approach, we have assembled a mode mapping table (Table 7) that clarifies mode transitions and associated label modifications. This streamlined process not only simplifies the task of classification but also significantly reduces the complexity of navigating the codebook. For an in-depth exploration of these event modes, please see Appendix A.
The mode-aware hypotheses in Table 3 enable NLI to accurately dismiss the Past hypotheses and correctly score the CF mode highest, underscoring its capability to discern semantic subtleties. Additionally, NLI’s semantic generalization avoids the need for exact word matching, unlike the complex dictionary-based methods. For example, similar hypotheses such as “increase” and “launch more protests” receive similar scores, and phrases like “promised to reduce” are considered similar to “will reduce”.
Our mode-aware NLI system leverages these insights, using the codebook to guide hypothesis construction. This approach enables easy conversion of present-tense label description into different modes or tenses, even for non-experts. The codebook also provides contrasting label examples, like “YIELD: ease protests” and “AGREE: agree to ease
protests” for the CP and CF modes for PROTEST, significantly streamlining the engineering process. We only need to ensure hypotheses, especially in Past mode, clearly reflect event trends. For instance, in Table 3, “protested against” was rephrased to “increased/launched more protests against” for enhanced clarity. Likewise, “imposed bans” can be modified to “increased/imposed more bans”.

heading: 3.3 Class Disambiguation
text: To address the issue of class ambiguity and overlaps in CAMEO/PLOVER, experts have documented instructions and annotation rules in the codebook. Annotators frequently consult the codebook when faced with ambiguous cases. In contrast, we can integrate this information into our machine to reduce manual annotation and effectively handling boundary cases. Note that incorporating excessive rules goes against our goal of designing a simple and adaptable system. It can lead to overfitting and inflexibility, similar to the limitations found in traditional dictionary-based methods. Therefore, we’ve chosen to include only the most frequent rules explicitly outlined in the codebook, considering this step as supplementary to our system.
One notable rule, referred to as the Conflict Override, is summarized from the codebook. This rule gives priority to labels in Material Conflict over Verbal Conflict, as depicted in Figure 2. If the top predictions include candidate labels in Material Conflict, the labels in Verbal Conflict will be overridden. For example, we label “protest to request” as material PROTEST other than verbal
REQUEST, as explained in Section 3.1. Similarly, we label “convict and arrest” as material COERCE other than verbal ACCUSE, considering the more severe actions involved. These rules can be easily customized and expanded by users to accommodate changes in the schema or ontology. Additional discussions are provided in Appendix B and C.

heading: 3.4 Tree-Query NLI Framework
text: We enhance precision and efficiency by integrating mode-aware NLI with class disambiguation into a tree-query framework, as shown in Figure 2. This contrasts with the “flat-query” method, where hypotheses are arranged at a single level, by organizing them hierarchically.
At Level 1 Context, we compare 76 Past hypotheses (≈ 5 hypotheses per Rootcode) to classify the premise’s context. Using a customized threshold, such as selecting the top-3 candidates with scores higher than the maximum score minus 0.1, we narrow down the most probable candidates. In the example, this filtering yields two candidates related to REQUEST and PROTEST.
At Level 2 Mode, we compare the hypotheses in other modes for the selected candidates to determine their mode. We focus on two types of modes in the experiments: Past and Future. For instance, PROTEST leads to two branches - the existing PROTEST and a new THREATEN (PROTEST+future). However, for certain Rootcodes like REQUEST, querying their Future variants is unnecessary since the labels remain the same from Past to Future (details in Table 7 and Ap-
pendix B). This reduces the number of Future hypotheses in Level 2 to 58, and only a subset requires querying per premise. In Figure 2, we collect all necessary scores for Level 2 analysis by querying just the new THREATEN hypothesis once.
At Level 3 Class Disambiguation, we apply specific rules, including the Conflict Override, to eliminate REQUEST since PROTEST already exists among the top predictions from Level 2.
ZSP is interpretable, flexible, efficient, and precise. First, we split the complicated, ambiguous classification into a simple tree framework that both computer science and political science researchers can easily understand. Second, experts can quickly update ZSP by revising the hypothesis table or class disambiguation rules according to a evolving ontology, which is much cheaper than maintaining large dictionaries or relabeling a dataset (detailed in Appendix C). Third, it improves efficiency. For instance, we only query 76 times in Level 1 + one time in Level 2 without comparing all 134 hypotheses in Figure 2. Finally, NLI scores within ZSP accurately capture nuanced entailment relations within the limited scope of compared hypotheses at each level. This minimizes potential errors that can arise from mixed hypotheses in different contexts and modes. We will validate this in experiments.

heading: 3.5 ChatGPT
text: Besides our proposed NLI-based ZSP model, we explored the zero-shot performance of LLMs on this task. We focused on two versions of ChatGPT: GPT-3.5 and GPT-4. We used the OpenAI API and designed prompts that incorporate task descriptions and pre-defined label sets, building upon insights from previous research (Wei et al., 2023; Li et al., 2023). The label descriptions were summarized and refined from the PLOVER codebook’s comprehensive Rootcode descriptions. Further insights are available through exemplified input output instances in Appendix H.

heading: 4 Experiments

heading: 4.1 Datasets
text: Since there were limited datasets with fine-grained annotation, we built a Rootcode-level PLV dataset from the CAMEO codebook and a balanced coarsegrained-labeled dataset (Parolin et al., 2022a), resulting in 1050 training examples and 1033 testing examples. We built three classification tasks with
varying degrees of complexity: Binary (cooperation vs. conflict), Quadcode, and Rootcode.
Besides the political science dataset PLV, we also explored how event ontology knowledge benefits and generalizes in other NLP datasets. Thus, we built a binary A/W dataset from ACE (Doddington et al., 2004) and WikiEvents (Li et al., 2021), which contain many conflict-related subjects that overlap the political ontologies. A/W consists of 802 training examples and 805 testing examples. See more details in Appendix D.

heading: 4.2 Setup
text: Regarding our proposed ZSP, we incorporated a finetuned NLI model5 into our tree-query system. For ChatGPT, we used OpenAI’s Chat completions API to access GPT-3.5 and GPT-4. To assess the practical usefulness of these zero-shot models, we compared them with notable baselines, including Universal PETRARCH (UP) (Lu and Roy, 2017), a widely-used dictionary-based CAMEO event coder. We measured UP’s ideal performance on relation classification by considering incomplete triplets, as detailed in Appendix E.
Additionally, we examine the performance of various supervised learning models, including masking language models (MLM) like BERT-baseuncased (Devlin et al., 2018) and ConfliBERT-scruncased (CBERT) (Hu et al., 2022a). Notably, CBERT reports greater effectiveness in the political science domain. We also use text generation models, namely BART (Lewis et al., 2020) and T5 (Raffel et al., 2020), to generate original label texts for this classification task. We trained these supervised models on either the entire training set or sampled subsets of varying sizes using a single V-100 GPU with default hyperparameters. Subsequently, we evaluated them on the complete testing dataset. We ran each scenario with five different seeds and reported average results for reliability.

heading: 4.3 Results and Analysis
text: We summarized the performance of dictionarybased and zero-shot models, as well as the supervised learning models trained on the entire training datasets, in Table 4. Additionally, in Figure 3, we compared ZSP with supervised learning models trained on varying limited datasets. UP and ChatGPT were excluded from the analysis due to their significant performance gap compared to the other
5https://huggingface.co/roberta-large-mnli
models, to maintain focus and relevance.
Supervised learning. Among supervised models, CBERT emerged as the top performer, surpassing BERT with less data required. BART closely trailed. It outperformed T5 by exhibiting less overfitting on small, imbalanced labeled datasets.
ZSP. ZSP consistently outperformed UP and ChatGPT, and it achieved competitive results with supervised learning models in most tasks (Figures 3a, 3c, 3d). Notably, in these scenarios, ZSP matched BERT and T5, while the stronger models CBERT and BART still required 25%-50% of the training data to achieve a slight performance gap (less than 4.3%) over ZSP. The only exception was a notable 6.7% performance gap observed between CBERT and ZSP on PLV-Quadcode (Figure 3b). This difference can be attributed to the dataset’s balanced and coarser-grained nature, which favors supervised learning.
However, supervised models experience a significant performance decline in more challenging fine-grained Rootcode classification (Figure 3c), emphasizing the need for sufficient and balanced annotation. Actually, our experience across multiple projects to develop event coding datasets with
approximately 1,000 examples typically extends beyond several months. Creating evaluation datasets like PLV and AW, or even relabeling existing ones, proves to be far more time-consuming. In contrast, designing NLI prompts from the codebook for ZSP takes just a few days, greatly reducing annotation efforts and demonstrating clear advantages in real-world applications. Furthermore, ZSP’s lack of a training phase significantly cuts down on GPU resource needs, enhancing its adaptability and enabling efficient inference on both CPUs and GPUs. This efficiency starkly contrasts with supervised models, which rely heavily on costly GPU resources for training.
We further analyzed ZSP’s confusion matrix for Rootcode classification (see Figure 7 in Appendix F). The results reveal high ZSP accuracy by correctly classifying most Rootcodes, yet there are some misclassifications, particularly for AGREE, SUPPORT, AID, and YIELD labels. These labels have subtle semantic differences, with AGREE representing a future, verbal, or hypothetical version of the other three categories. For instance, consider the sentence labeled as diplomatic SUPPORT “... <S> had approved an agreement with <T> ...”, ZSP produces conflicting predictions, with a score of 96.9% for the hypothesis “SUPPORT: approved an agreement” and 97.0% for “AGREE: agreed to sign an agreement”. This discrepancy arises due to the fine distinction between these two labels, which even human annotators may find challenging.
ChatGPT. We observed notable differences in the performance of GPT-3.5 and the latest GPT-4 models. Specifically, GPT-3.5 exhibited inconsistent results. Despite excelling in binary tasks, it struggles with more specific labels and even performs worse than UP in Rootcode classification. These challenges align with previous research in similar tasks (Yuan et al., 2023; Cai and O’Connor,
2023; Li et al., 2023; Gao et al., 2023). One ongoing challenge is generating formatted results and avoiding random labels outside the predefined set. To address this, we found that instructing GPT-3.5 to output digits (01-15) instead of text labels (AGREE - ASSAULT) partially alleviates these challenges and improves recall scores.
Another difficulty lies in effectively incorporating complex task descriptions and predefined label information into GPT-3.5. While our ZSP model can utilize class disambiguation rules easily, GPT3.5 struggles to retain large amounts of information and may forget relevant details after just one round of chatting. This limitation necessitates the repetitive input of essential information in every interaction, which reduces efficiency.
Furthermore, balancing the preservation of necessary information and the compression of prompts to accommodate actual questions proves challenging. Continuous refinement of the prompts does not consistently improve performance, and it is counterintuitive that longer label descriptions with more disambiguation instructions result in performance decline. The quest for an optimal prompt design remains an open question for future research.
However, GPT-4 stands out as a significant improvement over GPT-3.5. It effectively reduces formatting errors, with few occasional issues lingering. The most significant enhancement is its ability to comprehend and process longer input tokens, allowing for better use of input information and finer class distinctions. Interestingly, class disambiguation notes were found to be effective for GPT-4 but not for GPT-3.5, further distinguishing the two models. The success of GPT-4 highlights the vast potential of LLMs. While extensive API queries can be costly, and precision may be slightly lower than ZSP, GPT-4’s effectiveness with fewer prompts and superior generalization are notable advantages for future applications.

heading: 4.4 Ablation Study
text: We conducted an ablation study to address the following questions on ZSP: (1) Is a tree-query approach superior to a flat-query approach, which compares all hypotheses at single level simultaneously? (2) Does having more hypotheses guarantee better performance?
Table 5 displays the results of other zero-shot models, UP, GPT-3.5/4, and two variants of our ZSP models across multiple tasks. For the Flat-
query approaches, the Tiny model uses 18 hypotheses derived from the Rootcode names (See Table 1). The Full model incorporates a complete list of 222 label descriptions from the codebook. The Tree-query approach consists of our ZSP model at different levels: l1, l2, and l3.
The observation that the Tiny model with 18 hypotheses outperforms UP with 81k inflexible patterns, confirms the effectiveness of generalized PLM features. Furthermore, Tiny surpasses GPT3.5, highlighting the unreliability of GPT-3.5 and emphasizing the significance of expert knowledge in achieving superior results.
Despite the Tiny model’s limited capacity to handle nuanced cases, adding more unorganized hypotheses does not consistently improve performance. The Full model’s improper mixing and comparison of hypotheses for verbal and material events at different levels result in arbitrary NLI scores, leading to poor performance on PLV and inferior results compared to the Tiny model on A/W.
In contrast, the tree-query models outperform all flat-query models by a large margin at Level 1. Adding additional levels brings stable improvements, primarily for Quadcode and Rootcode. The tree-query framework effectively delimits the scope of candidate hypotheses and offers precise NLI scores that capture semantic differences. This ensures a more controllable and accurate result.

heading: 5 Conclusion
text: Future event coding tools should prioritize ease of interpretation and flexibility, making them more practical than annotating new datasets for blackbox supervised models. Therefore, we explored the potential of zero-shot relation classification using ChatGPT (GPT-3.5/4) and introduced our ZSP model. While GPT-3.5 struggled with fine-grained
classification, GPT-4 showed promise in mitigating instability issues. Our ZSP offers an even more cheap, precise, and adaptable solution. The key is structuring the complex problem into an interpretable, three-level tree framework, integrating mode-aware NLI, and incorporating class disambiguation rules from the codebooks. Overall, our study highlights the value of integrating transferred knowledge with expert linguistic insights to streamline the process of verifying event records for the political science community.

heading: 6 Limitations
text: Balancing generalization and specificity is a common challenge across many methods. ZSP was developed to address the complexities of annotation codebooks and the inefficiencies in training annotators. This led to streamlined annotation, such as labeling an event as PROTEST instead of DEMAND for a protest related to rights, aided by the Conflict Override rule which simplifies complex annotation notes into machine-understandable rules. To maintain a balance between complexity and adaptability, we included only the most frequently used rules from the codebook.
Our approach’s broader applicability, particularly in political science and related fields where codebooks are traditionally used to train annotators (Raleigh et al., 2010; Pavlick et al., 2016; Parolin et al., 2021), underscores the practicality of our method in streamlining data labeling. We are optimistic that combining established codebooks or knowledgebases with language models can extend to other domain-specific data. For instance, in legal studies, this approach can enhance the classification of legal documents by leveraging codified laws and regulations. In healthcare, it can aid in categorizing patient records and medical literature using clinical codebooks. In media studies and communication, it can support media content analysis by categorizing articles, broadcasts, and social media posts using thematic or sentiment-related codebooks.
However, challenges persist in zero-shot models when classifying semantically non-mutually exclusive fine-grained labels due to the intensive hypothesis engineering required. We addressed these challenges through the codebook’s attainable expertise, but ZSP may struggle with tasks lacking accessible domain knowledge bases or those with overly nuanced and ambiguous labels. For
example, classifying subcategories of ASSAULT (crime vs. attack vs. kidnap) or distinguishing peace protests from riots may require as many hypotheses as keywords (Barker et al., 2021; Radford, 2021). For such tasks, hybrid methods such as integrating ZSP or ChatGPT with few-shot learning, pattern-matching, or in-context learning could effectively address tasks of varying complexity, reducing human efforts. Future work will focus on exploring these hybrid methods.
Our exploration of ChatGPT models highlights the trade-off between generalizability and precision. While ChatGPT can adapt to any task with its chat-style format, it may sacrifice precision compared to NLI models. Due to time constraints and cost considerations, we did not investigate multi-turn interactions to enhance ChatGPT’s precision, leaving this for future research. Nonetheless, both zero-shot models show promising potential to surpass traditional dictionary-based methods and annotation-driven supervised learning.
In selecting comparative methods, we included the most pertinent and recent ones that meet the specific needs of our task. Political event coding differs significantly from NLP event extraction tasks, such as those using the popular ACE dataset, in the availability of directly comparable studies. The prevailing methodologies in ACE event extraction are predominantly supervised and don’t easily align with our unique ontology. Our investigation focuses on exploring the effectiveness of zero-shot models in event coding rather than achieving the highest accuracy through supervised approaches. Adapting existing zero-shot methods, designed with ACE contexts in mind, to our distinct requirements presents distinct challenges. Moving forward, we plan to expand our baseline comparisons in future studies.
While we initially considered expanding experiments to include other ontologies, we chose to focus on CAMEO/PLOVER, deferring broader explorations to future studies. This decision was influenced by practical constraints such as time and API costs, as well as a desire to pioneer within less-explored research domains. Unlike many widely-studied ontologies that rely on manuallycreated NLI systems and lack mode considerations, CAMEO/PLOVER presents unique challenges and opportunities. Its integration of mode features and codebooks makes it an ideal candidate for exploring PLMs in complex areas like political event
coding. By converting the complex expertise embedded in the codebooks into practical applications, we transcend the limits of conventional zero-shot modeling and showcase how PLMs like NLI and ChatGPT can be adapted to specialized domains.

heading: 7 Ethics Statement
text: The broad goal of producing accurate event data is to objectively measure and understand processes of political conflict and mediation around the world in order to prevent or mitigate their harm. We aim to produce a simple, flexible tool to serve this purpose. In particular, the zero-shot approach of this study largely reduces the costs, effort, and time to produce highly-quality event data on conflict, thus helping international and domestic government agencies, as well as researchers and practitioners to track, analyze, and mitigate the causes and effects of political violence. The project relied exclusively on news-story-like text as second-hand accounts of conflict events, but did not involve human research subjects.

heading: Acknowledgments
text: The research reported herein was supported in part by NIST Award # 60NANB23D007, NSF awards DMS-1737978, DGE-2039542, OAC-1828467, OAC-1931541, OAC-2311142, and DGE-1906630, ONR awards N00014-17-1-2995 and N00014-201-2738, and the National Center for Transportation Cybersecurity and Resiliency (TraCR).

heading: A Mode Design and Mapping
text: PLOVER suggests auxiliary modes to indicate whether a reported event is historical, futureoriented, hypothetical, or negated, as shown in Table 6. Some event types can theoretically combine with an auxiliary mode, such as AGREE becoming SUPPORT + future or THREATEN becoming ASSAULT + hypothetical. However, PLOVER’s guidance lacks concrete implementation for annotators, merely assuming that “the coding engine will be able to resolve these and put that information in the context.”
Moreover, while there are overlaps between PLOVER’s auxiliary modes and the field of linguistic modality in NLP (Palmer, 2001; Saurí and Pustejovsky, 2009; Rudinger et al., 2018; Pyatkin et al., 2021), notable differences exist. For instance, Pyatkin et al. (2021) explore modes like event plausibility, which partially echoes aspects of political actors’ intentions and event factuality in PLOVER. However, these explorations, though relevant, lack the precision and simplicity needed for direct application in PLOVER’s context. Our focus, therefore, is on a simplified, practical, and task-specific mode framework for PLOVER.
Our proposed mode for PLOVER only consider four types: Past (P), Future (F), Contradict_Past (CP), Contradict_Future (CF). These modes were derived from our examination of the CAMEO/PLOVER ontology and PLOVER’s auxiliary modes from the PLOVER codebook. Within this framework, we make a clear distinction between verbal, future or hypothetical events (Future) and historical or ongoing events (Past). And considering contradiction, we arrived at a simple 2x2 matrix with four modes outlined in Table 7. The table simplifies event coding and aids in accurately assigning Rootcode and Quadcode when an event’s mode changes.
P F CP CF
AGREE 1 AGREE 1 REJECT 3 REJECT 3CONSULT SUPPORT
COOPERATE 2 AGREE 1 SANCTION 4 REJECT 3AID
or ongoing events, often presented in past tense but not restricted to it. Future includes verbal, hypothetical or future events. We consolidate hypothetical and future auxiliary modes in Table 6 because their similar nature in transitions between material and verbal events. For instance, THREATEN (Verbal Conflict, e.g., threatening to attack) can be considered either hypothetical or future ASSAULT (Material Conflict). Contradict_Past and Contradict_Future encompass events contradicting Past or Future occurrences, respectively. As illustrated in Table 3, CF and CP may include words with contradictory meanings, not necessarily containing negation words like “do not.” Here, NLI’s ability to identify negation allows us to focus on positive hypotheses with contradictory meanings, aligning with PLOVER’s guideline to exclude negated events from datasets. Moreover, the codebook already provides mirrored hypotheses, eliminating the need for manual construction. For example, “YIELD: reduced protest against” is the CP of “PROTEST: protested against.”
An additional observation in Table 7 is that verbal actions remain classified as verbal regardless of mode. In contrast, material actions are categorized differently based on their contradictory forms. For instance, the contradiction or negation of AGREE (e.g., “didn’t agree to help”) is always REJECT, Verbal Conflict. However, for material actions (e.g., “provided aid to”), its CP form (e.g., “stopped providing aid to”) is SANCTION, Material Conflict, but its CF form (e.g., “would stop aid to”) is REJECT, Verbal Conflict.
In sum, our task-specific mode concept aligns with PLOVER’s auxiliary modes but enhances
PLOVER’s functionality, providing a practical, clear, and unambiguous approach to event coding.

heading: B ZSP’s Hypotheses and Class Disambiguation Rules
text: Table 16 shows the mode-aware hypotheses used in our experiments. We selected a subset of label descriptions in different Rootcode and Quadcode from the CAMEO codebook and converted these sentences to Past and Future modes. Some of them do not need Future variants as their labels from Past to Future remain the same, following Table 7.
Crafting disambiguation rules is a smaller part of our work compared to developing broad event modes. Event modes have resolved many cases, with custom rules providing fine-tuning. Our primary goal is to maintain a simple and generalizable model. Therefore, we only add commonly encountered rules like Conflict Override, which is prevalent in the CAMEO codebook and affects the coding for all conflict events.
Peace Override. As the second frequent case, classes related to “forces” vary according to actions and entities. For example, sending peacekeeping forces/workers/observers indicates cooperation, while sending forces to attack/occupy stands for conflict. Thus, we add hypotheses with “peace forces” distinct from normal “forces”, as shown in Table 8. Predictions with “peace forces” have higher priority. I.e., we override “forces” if the top predictions contain “peace forces” because the latter one is more specified and infrequent. This simple rule ensures high recall for general forces and high precision for peace forces.
Consult Penalty. Another common issue found in CAMEO/PLOVER is the overly general CONSULT class (e.g., consult/talk/meet/visit). Many actions (e.g., sending forces, attacks, and investigations) entail that the source visited the target. Likewise, an accusation or threat indicates that the source talked or met with the target. One simple
solution is to deduct the Consult Penalty, denoted as c (e.g., 2%), which penalizes the predicted entailment scores for the Rootcode “CONSULT”.
We analyze the impact of c at every level in Table 9, with (+c) indicating results with the penalty and (−c) showing results without it. The effect of c is evident, with an average increase of 1.6% in macro F1 for all the tasks. For deeper levels, c ensures the accuracy of Level 1 predictions to avoid error propagation. These findings confirm the importance of preventing overly general and ambiguous hypotheses. Incorporating c provides a simple solution to alleviate manual efforts in curating alternative hypotheses.
Besides the three main disambiguation rules, users can easily add or tailor less-important rules for their specific study purposes, as discussed in Appendix C.

heading: C Flexibility
text: The ZSP framework is notably flexible, easily accommodating changes in ontology or schema. Experts can swiftly update the ZSP method by modifying the hypothesis table or adjusting the class disambiguation rules to align with an evolving ontology. For example, if political scientists reclassify "arresting someone" from ACCUSE to COERCE (Table 7), they need only update the hypothesis label for “<S> arrested person of <T>”. Similarly, introducing sub-categories within YIELD involves simple updates to the hypothesis labels.
Disambiguation rules, such as the Conflict Override rule, which prioritizes PROTEST over REQUEST in certain contexts, can also be refined. Transitioning to a multi-label approach is straightforward by eliminating the Conflict Override rule
to acknowledge both PROTEST and REQUEST as valid labels.
When should users write their own disambiguation rules? The need for custom rules depends on specific user requirements and the balance between manual effort and system precision and recall. Custom rules can be particularly beneficial for fine-grained analysis. For example, the CAMEO codebook includes similar classes “COERCE- Impose blockade” and “PROTEST- Obstruct passage/ blockade”, as shown in Table 10. The key difference is whether the source is armed forces or protestors.
For researchers focusing on in-depth civil protest studies6, distinguishing between codes 144 (civilian protests) and 191 (military blockades) is crucial for accurate classification. Thus, they can define a simple rule, Blockade Override, without additional cost: remove the hypothesis “COERCEImpose blockade” if the top predictions contain PROTEST, indicating that the source is more likely protestors rather than armed forces. This adaptability showcases the model’s flexibility and customizability in complex political scenarios.
While ChatGPT can be generalized to any task with their chat-style format, they may sacrifice precision compared with the ZSP model. Yet, both zero-shot models show promising applicability to surpass traditional dictionary-based methods or annotation-driven supervised learning methods.

heading: D Building PLV and A/W Datasets
text: We extended existing resources to build our datasets, which is more efficient and effective than
6https://github.com/emerging-welfare/ glocongold.
creating a new dataset from scratch. Table 11 summarizes the two datasets’ detailed train and test split statistics. PLV is constructed from two resources. First, we outlined 335 examples (unique source-target pairs) with PLOVER Rootcode from the CAMEO codebook, and the PLOVER repository. Then we preprocessed a coarse-grainedlabeled dataset from CoPED (Parolin et al., 2022a) and manually extended its Quadcode labels to 15 Rootcode in the new PLOVER schema. The major modification can be seen in Table 1. However, given that the current PLOVER codebook is in development, we leave YIELD without splitting it to CONCEDE and RETREAT. Finally, Figure 5 visualizes our final dataset’s label distribution.
We built the A/W dataset from the ACE and WikiEvents datasets. First, the repository of (Li et al., 2021) provides templates for each event subtype of their ontology, enabling us to convert between different ontologies. For example, Figure 4 shows two frequent event types defined in the ontology. In both instances, argument 1 is equivalent to the source/actor, while argument 2 represents the target/recipient entities. Besides, the event type attack and arrest can be approximately mapped to ASSAULT and COERCE in PLOVER, respectively, as shown in Table 12.
Therefore, we built labeled source-target pairs from ACE and WikiEvents. We extracted major sentences that contained the labeled entities from each long document in WikiEvents. We also removed entities that only consist of pronouns. Finally, we got 1258 valid sentences with 1687 labeled Source-Target pairs. To prevent label leaking, we split the dataset by document IDs, ensuring dis-
tinct name entities for training and testing. Figure 6 shows the distribution of the original event types and the mapped binary class.
The nuanced differences between the two domains necessitate that event types be only “approximately” mapped to PLOVER Rootcodes. And extensive manual verification is needed to ensure accuracy. This complexity is rooted in the distinct focuses of NLP, which emphasizes predicates or topic-centric events, and Political Science, which concentrates on event status or mode. For instance, examples in Tables 2 (planned protest) and Table 3 (agreement to suspend protests) are both categorized as Conflict.Demonstrate in A/W, but in PLOVER, they are distinctly classified as THREATEN (verbal conflict) and AGREE (verbal cooperation), respectively. The binary labels even switch from conflict to cooperation in the second case. Thus, manual checking remains crucial even at the binary level.
The annotation process was carried out by two authors, achieving a Kappa score of 0.76, with discrepancies resolved through discussion.

heading: E UP Experiment Setup
text: Universal Petrarch (UP) is a popular dictionarybased event coder (Lu and Roy, 2017). We adapted UP into our task of relation classification with gold source and target, i.e., source-target-action triplets. We found that UP is too strict and often results in
incomplete or empty triplets. Thus, we reported the best possible result by the following methods. First, we used UP for each sentence to extract all possible events. Then we ranked the extracted triplets by the number of matched entities with gold sources and targets to decide the event code. We also counted the valid event code when there were no matched entities but only matched trigger action verbs. Even so, there are still 10% and 27% invalid event code results on PLV and A/W datasets, respectively. Finally, we mapped its output four-digit code to PLOVER Rootcode and Quadcode (similar to Figure 1).

heading: F ZSP’s Detailed Results Analysis
text: We examined the confusion matrix for ZSP on Binary (Figure 8), Quadcode (Figure 9), and Rootcode (Figure 7) classifications. The results show that ZSP perfectly classifies most contexts, with a slight degradation in differentiating mode (verbal vs. material).
In-depth class reports for PLV on Quadcode (Table 14) and Rootcode (Table 15) reveal that ZSP outperforms UP in nearly all metrics, except in the precision of the Verb-Conflict class (85.5%). However, UP’s lower recall impacts its overall F1 score, showcasing the superiority of PLM’s generalized knowledge over rigid pattern-matching approaches. Additionally, we noticed a performance trade-off when using overrides from Level 2 to Level 3. For instance, recall improves in Material-Conflict but decreases in Verbal-Conflict. Nevertheless, Level 3 significantly enhances overall F1 scores.
Further, we expand on the ablation study (Section 4.4), emphasizing why the tree-query approach, with fewer hypotheses, surpasses the “Full” model, which utilizes 222 flat hypotheses. Figure 10 illustrates the confusion matrix for the Full model’s Rootcode classification. A comparison be-
nary code. Figure 9: Confusion matrix
for ZSP on PLV Quadcode.
tween this matrix (Figure 10) and the default ZSP model using tree-query (Figure 7) reveals significant differences. The variable nature of NLI scores is a key factor in these differences. The tree-query model’s focused approach on controlled hypothesis groups with consistent entities and predicates, but varying modes, leads to more accurate hypothesis identification. In contrast, the Full model’s flat amalgamation of diverse hypotheses results in unpredictable outcomes and struggles with accurate mode classification, evident in frequent misclassifications between categories such as AGREE vs. SUPPORT, YIELD vs. AGREE, and REJECT vs. SANCTION or ASSAULT.

heading: G NLI Model Selection
text: We selected RoBERTa-Large-MNLI7 for its extensive usage in NLI research, with comparable alternatives like BART-Large-MNLI8 also showing favorable results. Employing smaller-sized base
7https://huggingface.co/roberta-large-mnli 8https://huggingface.co/facebook/bart-large-mnli
models for zero-shot tasks is less common, primarily due to the significant drop in performance. Consequently, there are limited models specifically designed and widely accepted for zero-shot classification tasks.
From an efficiency perspective, employing large models for zero-shot tasks proves efficient as they are only required during the inference phase. Conversely, training supervise large models can be relatively expensive. Besides, one of our chosen baselines, CBERT (Hu et al., 2022a), only has a base version. Therefore, we conducted supervised experiments using base models while reserving large models exclusively for zero-shot tasks. This approach ensures a relatively fair and meaningful comparison between the two model types.
However, we also considered the possibility that a more rigorous comparison could have strengthened our hypotheses, particularly in demonstrating the effectiveness of smaller base models for handling fine-grained tasks in zero-shot scenarios. To explore this, we conducted experiments using an existing RoBERTa base model9. The results are
9https://huggingface.co/cross-encoder/nli-roberta-base
presented in Table 13, offering valuable additional insights alongside the findings presented in Table 4. While we observed that base models can effectively classify context or topics, they encountered challenges in distinguishing nuanced differences in mode. This distinction can lead to a drop in performance compared to larger models.

heading: H ChatGPT Experiment Setup
text: Table 17 exemplifies inputs for relation classification tasks. Our task is characterized by challenging fine-grained classification that demands a substantial amount of input information. Due to token limitations and API costs, inputting one example at a time with a lengthy prompt is inefficient and costly. Instead, we used a long prompt followed by a list of input sentences to stay within the maximum token limits and obtain a list of predicted labels. More specifically, the inputs comprise the task and label description, a sentence list (usually limited to less than 50 sentences due to word constraints), and the task requirements. The anticipated output from the model is the predicted labels. Despite our repeated emphasis on ChatGPT generating only predefined labels, certain issues remain. To mitigate these, we use numerical codes (01-15) instead of text labels (AGREE - ASSAULT), reducing ChatGPT’s generation of labels outside the predefined set. Additionally, we’ve noticed that ChatGPT tends to forget the task description and predefined label information, necessitating their input each time. Finally, refining the task and label description doesn’t yield improved results. This
underscores the complexity of the task, involving semantically non-mutually exclusive fine-grained labels, which proves challenging for ChatGPT.

heading: Future Work
text: We selected a subset of label descriptions in different Rootcode and Quadcode from the CAMEO codebook and converted these sentences to Past and Future modes. Some of them do not need Future variants as their labels from Past to Future remain the same, following Table 7. Crafting disambiguation rules is a smaller part of our work compared to developing broad event modes. Event modes have resolved many cases, with custom rules providing fine-tuning. Our primary goal is to maintain a simple and generalizable model. Therefore, we only add commonly encountered rules like Conflict Override, which is prevalent in the CAMEO codebook and affects the coding for all conflict events. Peace Override. As the second frequent case, classes related to “forces” vary according to actions and entities. For example, sending peacekeeping forces/workers/observers indicates cooperation, while sending forces to attack/occupy stands for conflict. Thus, we add hypotheses with “peace forces” distinct from normal “forces”, as shown in Table 8. Predictions with “peace forces” have higher priority. I. e. , we override “forces” if the top predictions contain “peace forces” because the latter one is more specified and infrequent. This simple rule ensures high recall for general forces and high precision for peace forces. Consult Penalty. Another common issue found in CAMEO/PLOVER is the overly general CONSULT class (e. g. , consult/talk/meet/visit). Many actions (e. g. , sending forces, attacks, and investigations) entail that the source visited the target. Likewise, an accusation or threat indicates that the source talked or met with the target. One simple
solution is to deduct the Consult Penalty, denoted as c (e. g. , 2%), which penalizes the predicted entailment scores for the Rootcode “CONSULT”. We analyze the impact of c at every level in Table 9, with (+c) indicating results with the penalty and (−c) showing results without it. The effect of c is evident, with an average increase of 1. 6% in macro F1 for all the tasks. For deeper levels, c ensures the accuracy of Level 1 predictions to avoid error propagation. These findings confirm the importance of preventing overly general and ambiguous hypotheses. Incorporating c provides a simple solution to alleviate manual efforts in curating alternative hypotheses. Besides the three main disambiguation rules, users can easily add or tailor less-important rules for their specific study purposes, as discussed in Appendix C