CODE4STRUCT: Code Generation for Few-Shot Event Structure Prediction
Year: 2023

Abstract:
Large Language Model (LLM) trained on a mixture of text and code has demonstrated impressive capability in translating natural language (NL) into structured code. We observe that semantic structures can be conveniently translated into code and propose CODE4STRUCT to leverage such text-tostructure translation capability to tackle structured prediction tasks. As a case study, we formulate Event Argument Extraction (EAE) as converting text into event-argument structures that can be represented as a class object using code. This alignment between structures and code enables us to take advantage of Programming Language (PL) features such as inheritance1 and type annotation2 to introduce external knowledge or add constraints. We show that, with sufficient in-context examples, formulating EAE as a code generation problem is advantageous over using variants of text-based prompts. Despite only using 20 training event instances for each event type, CODE4STRUCT is comparable to supervised models trained on 4,202 instances and outperforms current stateof-the-art (SOTA) trained on 20-shot data by 29.5% absolute F1. By leveraging the inheritance feature of PL, CODE4STRUCT can use 10-shot training data from a sibling event type to predict arguments for zero-resource event types and outperforms the zero-shot baseline by 12% absolute F1. 3

heading: 1 Introduction
text: Large Language Model (LLM) trained on massive corpora of code mixed with natural language (NL) comments and docstrings4 (e.g., Chen et al. 2021,
1Inheritance is a way to create a hierarchy of classes in PL. A child class can base upon another class, retaining similar implementation.
2Developers use type annotations to indicate the data types of variables and input/outputs of functions.
3All code and resources are publicly available at https: //github.com/xingyaoww/code4struct.
4Text used to document a specific segment of code.
Nijkamp et al. 2022) has demonstrated the ability to translate natural language instructions into structured code. We ask if this conversion between language and code can serve as a bridge to build a connection between language and semantic structure, which is the goal of many structured prediction tasks (e.g., semantic parsing, information extraction) in Natural Language Processing (NLP). In particular, the target structure (e.g., event-argument graph in Figure 1) can be mapped to code more straightforwardly compared to natural language, which often requires careful prompt engineering (Hsu et al. 2022, Li et al. 2021, Table 2). In addition, code written in programming languages has an inherent advantage in representing complex and
3640
interdependent structures (Miller, 1981; Sebrechts and Gross, 1985) with features such as inheritance and type annotation.
As a case study, we showcase our proposed CODE4STRUCT on the Event Argument Extraction (EAE) task, which aims to extract event structures from unstructured text. EAE is the ideal testbed for our method due to the close alignment between EAE and PL as shown in Table 1. In CODE4STRUCT (Figure 1), we first translate the entity and event type ontology into Python class definitions. Conditioned on the relevant class definitions and the input sentence, we prompt an LLM to generate an instantiation of the event class, from which we can extract the predicted arguments.
By leveraging the alignment between PL and NLP problems, CODE4STRUCT enjoys various advantages as shown in Table 1. Using PL features like type annotation and argument default value, we can naturally enforce argument constraints for output structures. This allows CODE4STRUCT to handle multiple or zero argument fillers for the same argument role by annotating the expected type (i.e., expect a list of entities) and setting the default value for each argument (i.e., an empty list without any entity by default). Furthermore, we can naturally utilize the event hierarchy by leveraging inheritance. Inheritance allows a child event class (e.g., Transport) to reuse most components of its parent class (e.g., Movement) while preserving its unique properties. We demonstrate that hierarchical event types allow zero-resource event types to use annotated training examples from their high-
resource sibling types (§4.6). We outline our contributions as follows:
• We propose CODE4STRUCT to tackle structured prediction problems in NLP using code generation. As a case study, we use CODE4STRUCT for Event Argument Extraction (EAE).
• We perform extensive experiments contrasting the performance of code-based prompt and two variants of text prompt on different LLMs and show that code prompt is generally advantageous over text prompt when sufficient in-context examples are provided (§4.2).
• We demonstrate that 20-shot CODE4STRUCT rivals fully-supervised methods trained on 4,202 instances. CODE4STRUCT outperforms a SOTA approach by 29.5% absolute F1 gain when 20-shot data are given to both. 0- shot CODE4STRUCT can even outperform the SOTA on both 20 and 50 shots (§4.5).
• We show that integrating the event ontology hierarchy by class inheritance can improve prediction. Compared to the zero-shot baseline, we see 12% F1 gains for zero-resource event types when using 10-shot examples from their sibling event types (§4.6).

heading: 2 Code Generation Prompt Construction
text: In Event Argument Extraction (EAE) task, a model is provided with an event ontology and the tar-
get text to extract from. Similarly, we prompt an LLM with the ontology that consists of definitions of event types and argument roles, and input sentences to generate code that instantiates the given event type. We breakdown the input prompt into three components: (1) ontology code representation which consists of Python class definitions for entity types and an event type (§2.1); (2) optional k-shot in-context learning examples for the event type defined in (1) (§2.3); (3) task prompt for completion (§2.2). We show a breakdown of the full prompt in Figure 2.

heading: 2.1 Ontology Code Representation
text: To represent the event ontology as code, we concatenate the base class definition, entity class definitions, and event class definitions.
Base Class Definition We define base type Entity and Event to be inherited by other classes.
Entity Class Definition We use entity type definitions from the Automatic Content Extraction (ACE) program5. We construct Python classes that inherit from Entity and use the entity type as the class name (e.g., class GPE(Entity)). We add a natural language description as a docstring of the defined class for each entity type.
5https://www.ldc.upenn.edu/ collaborations/past-projects/ace

heading: 2.1.1 Event Class Definition
text: We define the event class using the name of the event type (e.g., class Transport). As ACE defines its event types in a hierarchical ontology, mimicking class definitions in Object-Oriented PL, we inherit the event class definition from its parent (e.g., class Transport(Movement)) or root event type if the event class does not has a parent (e.g., class Movement(Event)). An example of hierarchical event definition can be found in Figure A.9.
We define the argument roles (e.g., destination of Transport) as input arguments of the constructor __init__6. We specify the type of each argument role using Python type annotation, a commonly used PL feature: For example, agent: List[GPE | ORG | PER] means that the agent argument accepts a list of entities which could be either of type GPE (Geo-Political Entity), ORG (Organization), or PER (Person). We assign each input argument (e.g., agent) to a class member variable of the same name.
We include event description templates into the docstring of the class definition. The event description templates are modified from Li et al. (2021) by replacing each role with their corresponding member variable (e.g., self.agent).

heading: 2.2 Task Prompt
text: The task prompt consists of a docstring describing the task and incomplete event instantiation code for
6A constructor is a special function that initializes an instance of a class.
completion. An example of a task prompt can be found in Figure 2. The text-based docstring contains a task instruction and an input sentence. We mark the ground truth trigger words for the input text by surrounding them with **. We choose to use ** as it is used to set text to bold in Markdown (a markup language for creating formatted text), which is commonly found in code bases and web data on which our LLM is trained. The incomplete code prompt assigns a partial instantiation of an event class to a variable to trigger the model for completion, for example, transport_event = Transport(.
We observed that LLM tends to generate additional sentences paired with extracted arguments if no stopping constraint is applied. To focus on the given EAE task, we stop the code generation whenever any of the following patterns is generated by the model: """, class, print, or #.

heading: 2.3 In-context Learning
text: Optionally, we can include in-context learning examples, which are task prompts (§2.2) paired with completed event instantiations using ground-truth arguments (see Figure 2 for a specific example). For k-shot learning, we concatenate k such examples together. Given a task prompt, we deterministically gather k learning examples by collecting training instances with the same event type, following the order of occurrences in the training set.

heading: 3 Why Represent Event Structure in PL?
text: A wide range of NLP tasks have benefited from LLM (Brown et al., 2020; Hoffmann et al., 2022; Chowdhery et al., 2022) trained on web-scale language corpora. To effectively use LLM trained on language for EAE, one of the biggest challenges is to specify the desired output, namely event structures in our case, using natural language.
There is a tradeoff between the effort put into defining the output or designing the prompt (e.g., Text2Event in Table 2) and the benefit from pre-
training in natural language (e.g., DEGREE and BART-Gen in Table 2). Text2Event (Lu et al., 2021) resides at one end of the spectrum with a concise but unnatural output format. As a result, this formulation under-utilizes the pretraining power of the model and does not work in low-resource settings as shown in Table 4. Towards the other end, Hsu et al. (2022); Li et al. (2021) design manual templates for the model to fill in. We also design two variants of language prompt as shown in Figure A.5 and A.6 miciking our code prompt and BART-Gen style prompt for comparison. Note that these natural language prompts are much more verbose and, as shown in §4.2, usually result in sub-optimal performance with sufficient in-context examples.
Essentially, this tradeoff is a result of the mismatch between the pretraining corpora and task output formats. Instead of using LLM trained on only unstructured text, we turn to LLM trained with a mixture of text and code, where the text is often aligned in semantics with the accompanying code. Such Code-LLMs have the ability to convert text into corresponding code as demonstrated by (Chen et al., 2021; Nijkamp et al., 2022). Then we can map the desired output event structure into code in a straightforward manner and leverage the full pretraining power of these models. PLs like Python offer features (e.g., class, docstrings, type annotations, inheritance) that have a significant presence in the pre-training corpus of Code-LLM due to frequent usage. CODE4STRUCT leverages these features to succinctly describe event structures, which makes it better aligned with Code-LLM. By leveraging LLM’s learned knowledge from diverse pre-training domains, CODE4STRUCT can work well in open-domain, achieving non-trivial zeroshot performance given unseen event types (§4.5). CODE4STRUCT is also data-efficient as exemplified by reaching comparable performance to fullysupervised methods with much fewer annotated examples (20 per event type) (§4.5).

heading: 4 Experiments

heading: 4.1 Experiment Setup
text: LLM We use CODEX code-davinci-002 (Chen et al., 2021), a GPT-3 (Brown et al., 2020) model finetuned on code, which supports up to 8k input tokens. We compare its performance with InstructGPT (Ouyang et al., 2022) text-davinci-002 and its improved version text-davinci-003, both support up to 4k input tokens. We access these LLMs through OpenAI API7.
Hyperparameters We prompt LLM to generate code that instantiates an event using sampling temperature t = 0 (i.e., greedy decoding). We set the max number of new tokens for each generation to 128, which fits all code outputs for the test set.
Evaluation Tasks We use ground truth event type and gold-standard trigger words to perform Event Argument Extraction.
Dataset We evaluate our performance of EAE on the English subset of Automatic Content Extraction 2005 dataset (ACE05-E)8 (Doddington et al., 2004). We follow Wadden et al. (2019); Lin et al. (2020) for dataset processing. ACE05-E has hierarchical event types with 8 parent types and 33 child types. Among all child types, roughly half of the event types (14 out of 33) in ACE05-E have less than 50 event instances in the training set. We show statistics for each event type in Table A.4.
Evaluation metrics We use Argument F1score following prior work (Ji and Grishman, 2008; Li et al., 2021; Hsu et al., 2022): We consider an argument to be correctly identified when the head word span of predicted text9 matches that of the human-annotated text (denoted as Arg-I); We consider an argument to be correctly classified if the role (e.g., agent) of a correctly identified argument matches that of the human annotation (denoted as Arg-C).

heading: 4.2 Comparison with Text Prompt
text: To compare our code-based prompt with text-based prompts, we design two variants of text prompt:
7https://openai.com/api/ 8https://www.ldc.upenn.edu/
collaborations/past-projects/ace 9We find the span of predicted text in the given sentence, then use spacy library to find its head word.
T (1) mimicking our code prompt (i.e., code imitation, Figure A.5) and T (2) following BARTGen style prompt (Li et al., 2021) (Figure A.6) which resembles natural language more compared to T (1). Both text prompts have similar components as our code-based prompt in Figure 2. Text prompts rely on natural language to define the requirement and format of the desired output, while the code prompt utilizes PL syntax. We compare the F1 score difference between the code prompt (§2) and two variants of text prompts (i.e., ∆
(i) C−T = F1code − F1 (i) text, i ∈ {1, 2}) on different LLMs in Table 3. We include exact performance numbers of text prompts in Table A.3. We summarize our findings as follows:
• Code prompt outperforms both text prompts on Arg-C F1 (i.e., ∆(i)C−T > 0) for two text prompt variants and all LLMs except text-davinci-003 when sufficient incontext examples are given (i.e., k ≥ 5).
• For *-davinci-002 LLMs, there are more significant performance gains from using a code prompt (i.e., increasing ∆(i)C−T for all i) when the number of in-context examples k increases (for k ≥ 5).
• There is no clear trend on Arg-I F1 to differentiate code and text prompts, except for text-davinci-003, which exhibits similar behavior that code prompt performs better with larger k.
• Text prompt T (2) (BART-Gen style), which resembles natural language more, performs poorly under low-shot (k ≤ 1), primarily due to the LLM being unable to produce the desired structure output described using language in T (2), causing the low-shot code-text performance gap ∆(2)C−T to be larger compared to T (1). These low-shot performance differences between T (1) and T (2) further signify the need to prompt engineering for languagebased prompts to work well in a low-shot setting.

heading: 4.3 Comparison with different LLM
text: We measure the performance of the same CODE4STRUCT code prompt across different foundational LLMs in §4.1. LLM performance comparison can be found in Figure 3. text-davinci-002 is an InstructGPT
(Ouyang et al., 2022) model finetuned with human demonstrations based on code-davinci-002, yet these two LLMs perform similarly in Arg-C F1. Although having a similar code prompt Arg-C performance, text-davinci-002 generally has a larger ∆(i)C−T compared to code-davinci-002 of the same k in Table 3 (e.g., +3.6 vs. +8.0, +2.4 vs. +5.8 on 20-shot for both text prompt variants), suggesting the degradation of text prompt performance after finetuning with human demonstrations. text-davinci-003, which uses reinforcement learning (RL) with reward models to align with human preference10 (Ouyang et al., 2022), outperforms other LLMs for k > 5. In Table 3, text-davinci-003 obtains superior Arg-C F1 performance (60.4% vs. 56.5% on 20-shot) compared to text-davinci-002. This suggests RL with reward models effectively improves EAE performance (i.e., Arg-C) on code prompt.
Interestingly, text-davinci-003 has a very different ∆(i)C−T pattern for text prompt T
(2) compared to T (1). Like text-davinci-002, in Table 3, Arg-C ∆(1)C−T for text prompt T
(1) has an increasing trend with a similar magnitude (e.g., +7.8 vs. +8.0 on 20-shot). That is, in both LLMs, the code prompt is always better than text prompt T (1) with k ≥ 5. However, for text prompt T (2) which is more similar to natural sentences, the gap ∆
(2) C−T exhibits a vastly different pattern compared to other models: code prompt performs on par or even slightly worse than T (2) for k ≥ 5. We also notice that for zero-shot prediction, T (2) on text-davinci-003 performs better compared to other LLMs. This indicates that aligning LLM with RL and reward models helps improve LLM’s ability to follow zero-shot language instructions.
Even though code prompt still performs superior to both text prompt variants on 002 LLMs, results from text-davinci-003 suggest a better-
10https://beta.openai.com/docs/ model-index-for-researchers
aligned language model can perform equally well on a natural text prompt T (2) when sufficient incontext examples are provided.
01 5 10 20 30 50
50
52
54
56
58
60
62
64
Arg-I F1
01 5 10 20 30 50 35
40
45
50
55
60 Arg-C F1
Number of In-context Examples (k-shot)
code-davinci-002 text-davinci-002 text-davinci-003
Figure 3: CODE4STRUCT performance (in F1%) with different k. We observe improvements with diminishing returns when we increase the number of incontext examples. Exact performance numbers can be found in Table A.3 (code prompt). We stop at k = 20 for text-davinci and k = 50 for code-davinci-002 as including more examples would exceed the input length limitation imposed by corresponding LLM.

heading: 4.4 Comparison with different k
text: We examine the performance of code prompts with varying numbers of examples in Figure 3. We observe that F1 scores for all metrics generally increase with diminishing returns when providing more in-context learning examples. The initial in-context example (k = 1) brings the largest absolute performance gain (+11.8, +10.2, +6.9 Arg-C F1 for three LLMs). For k ≥ 20 on code-davinci-002, the Arg-I and Arg-C performance plateaus or even slightly degrade, as not all event types have enough in-context examples to benefit from increasing k (i.e., only 19 out of 33 event types have more than 50 examples for in-context learning). To further investigate why the performance plateaus, we analyze how the sentence variability (or diversity) of in-context examples influences Arg-C performance in §A.4; We find that
Arg-C performance is positively correlated with the variability of in-context examples which plateaus as k increases, hinting that in-context learning performance may eventually plateau with increasing k due to little variability gains from the additional data.

heading: 4.5 Comparison with Supervised Models
text: Baselines Unlike prior methods trained on the entire training set, CODE4STRUCT learns from up to 50 examples (i.e., 39 examples per event type on average, roughly 1% among all training instances) to predict arguments for each test event type. To ensure a fair comparison, for each event type t in the test set, we train a Text2Event model (Lu et al., 2021) and a DEGREE model (SOTA, Hsu et al. (2022)) on 20-shot and 50-shot in-context examples CODE4STRUCT used while providing gold-standard trigger words. We evaluate both models trained on event type t on a partition of the test set that only contains instances of event type t. We then aggregate F1 scores (micro F1) across all 31 event types on the test set and report them in Table 4. Following Hsu et al. (2022), we also compare with classification-based (DyGIE++ Wadden et al. (2019), BERT_QA Du and Cardie (2020), OneIE Lin et al. (2020)) or generationbased (TANL (Paolini et al., 2021), BART-Gen Li et al. (2021), DEGREE Hsu et al. (2022)) models trained on the full training set.
Results We report the performance of CODE4STRUCT using LLMs (§4.1) in comparison with prior work in Table 4. We report the performance of supervised models using the full dataset from Hsu et al. (2022). Note that 50-shot results for text-davinci are not available as the 50-shot input prompt will exceed LLM’s input token length limitation, hence we use code-davinci-002 for 50-shot comparison.
In the few-shot setting, 20-shot CODE4STRUCT using text-davinci-003 can surpass DEGREE (Hsu et al., 2022), the current state-ofthe-art, by a large margin (+29.5% Arg-C F1). Our zero-shot CODE4STRUCT using the bestperforming text-davinci-003 model can already achieve higher Arg-I and Arg-C performance than the 20-shot and 50-shot DEGREE. Despite only learning from 20 examples, 20-shot CODE4STRUCT achieves comparable performance with other fully-supervised models trained on 100% of the training data (4,202 instances).

heading: 4.6 Event Type Hierarchy Improves Zero-resource EAE
text: In this section, we show that CODE4STRUCT, when provided with hierarchical event definitions and few-shot training instances Des from a sibling event type es ∈ Siblings(e) under the same parent event type, can improve performance for child event type e as good as if training instances De from the same event type e were used. This allows zero-resource event types without annotated data to exploit the event type hierarchy and benefit from their high-resource siblings. We include an example task prompt with sibling examples in Figure A.11 and report our results in Table 5.
Setup We split the child types for each parent type into training and testing types by selecting the high-resource child type with the largest amount of training instances to be the training type and have the rest be testing types. The train-test split for ACE types can be found in Table A.5. Under the same parent event type, we use data instances from the training type (i.e., a sibling of testing types) as in-context examples to predict arguments for each testing type. We include event class definition (Figure 2) for parent event type (e.g., Transaction), child training (sibling) event type (e.g., Transfer_Money), and child testing event type (e.g., Transfer_Ownership). We show an example of event definition with sibling type in Figure A.10. The few-shot performance when using data from a sibling type Des is denoted with (sibling type) in Table 5. To demonstrate the
effectiveness of using data from sibling event types, we compare it with using training instances from the testing event type itself De (denoted as (same type)) and from a random non-sibling event type (denoted as (non-sibling type)).
Results We observe that CODE4STRUCT, when prompted with training examples from sibling type, performs on par with the prompt that uses training examples from the testing type itself on 1-shot and 10-shot. The substantial performance gain (+9% Arg-C F1 on 1-shot, +12% Arg-C F1 on 10- shot, compared with 0-shot) contributed by siblingtype training examples demonstrate the potential of applying CODE4STRUCT to zero-resource event types with no training data by exploiting their hierarchical relationship with other high-resource event types. Surprisingly, similar to the observation made by Min et al. (2022), using in-context examples from a random non-sibling type also benefits CODE4STRUCT performance, albeit not as helpful as sibling examples under 10-shot.

heading: 5 Related Work
text: Code-LLM for Structured Task Sun et al. (2019); Singh et al. (2022) focus on procedural tasks that aim to control situated agents in an embodied environment by representing the procedure plan in code. Madaan et al. (2022) uses CodeLLM to generate a structured commonsense reasoning graph represented in code, which is similar in spirit to our work but in a different task. Gao et al. (2022) tackles math and symbolic reasoning tasks by decomposing the natural language problem into runnable steps using Code-LLM and delegating
solution calculation to a PL interpreter. We leverage PL features (e.g., inheritance, type annotation) to introduce extra information and constraints for structured prediction, which is largely overlooked by prior work.
Event Extraction Li et al. (2013); Nguyen et al. (2016); Yang and Mitchell (2016); Wadden et al. (2019); Lin et al. (2020) use classification models and mitigate error propagation from pipeline models by leveraging global features to predict event triggers and arguments jointly. Recent work such as Liu et al. (2020) formulates event extraction as a reading comprehension problem and Li et al. (2021); Huang et al. (2021); Paolini et al. (2021); Hsu et al. (2022) converts event extraction to a text generation task to better exploit label semantics from pretrained language models. The most similar work to ours is Text2Event (Lu et al., 2021), which uses controlled generation to generate structures in a manually specified linearized format directly, hindering the model in leveraging pre-trained NL knowledge. On the other hand, our approach CODE4STRUCT directly generates structure in PL instead of using a manually designed format to fully exploit LLM’s knowledge of PL.

heading: 6 Conclusions and Future Work
text: We propose CODE4STRUCT for structured prediction tasks in NLP by leveraging LLMs trained on language and code. As a case study, we use CODE4STRUCT to extract event arguments from natural language sentences through code generation. We show that, with sufficient in-context examples, formulating EAE as a code generation problem is advantageous over using text-based prompts. Our proposed CODE4STRUCT rivals fully-supervised models trained on 4,202 data instances only using 20-shot. It also outperforms a SOTA model by 29.5% absolute F1 when both are given the same 20-shot data. Furthermore, benefitting from hierarchical event definitions, CODE4STRUCT can predict arguments for zeroresource event types only using 10-shot training instances from its sibling event type and outperforms 0-shot baseline by 12% absolute F1 score. Going forward, we plan to expand CODE4STRUCT to a broader range of more complex structured prediction tasks (e.g., relation prediction, schema matching). We would further explore the executable nature of PL to improve LLM’s ability for structured prediction.
Limitations
In this work, our approach assumes event triggers and argument templates (i.e., ontology) are given. This limits our approach’s applicability, as it requires an event detection system to produce event triggers and event types before LLMs can be prompted to generate event arguments.
We only explore hierarchical events with only 2 levels from the ACE05-E ontology and data, which has limited coverage of real-world complex event hierarchy. Similar to prior event argument extraction work, our approach relies on a human-curated hierarchical ontology. We leave automatically discover hierarchical ontology for future work.
Despite LLMs performing well on EAE with few-shot data, compared to existing supervised approaches, their inference is relatively slow and costly11 since the LLMs we used are generally more than 100x larger in the number of parameters. Prior work (Zhao et al., 2021; Lu et al., 2022) has demonstrated a strong relationship between performance and in-context demonstrations; however, for ease of comparison to supervised baselines, we use the same set of examples from the training set for in-context learning. We expect better selecting (Liu et al., 2021) and ordering (Lu et al., 2022) incontext examples can benefit CODE4STRUCT performance, which we leave for future work.
Ethical Considerations
Since event argument extraction only requires predicting arguments from the given text, the risk of generating toxic languages is relatively low as long as the given test is not toxic. This is because the prediction can be grounded in the input sentence, eliminating potential toxic tokens that did not appear in the original sentence. However, discrimination and bias are possible, as observed in the foundational LLMs we used (Brown et al., 2020; Chen et al., 2021; Ouyang et al., 2022), which we refer to Brown et al. (2020) for detailed discussion.

heading: Acknowledgement
text: We thank the anonymous reviewers for their helpful suggestions and comments. This research is based upon work supported by U.S. DARPA KAIROS
11We perform most of our experiments on code-davinci-002 which is in free public beta at the time of the experiment. For text-davinci models, around 700 USD was used to access its API to perform relevant experiments in this paper.
Program No. FA8750-19-2-1004, U.S. DARPA AIDA Program No. FA8750-18-2-0014 and U.S. DARPA ITM Program No. FA8650-23-C-7316. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

heading: A Appendix
text: A.1 Qualitative Analysis
We show examples of 0-shot and 50-shot CODE4STRUCT argument extraction result in Figure A.1. CODE4STRUCT can leverage implicit commonsense knowledge in LLM to infer arguments not presented in the text. In the first 0- shot example, the model inferred the place of Welch’s retirement is in the United States. This is a reasonable guess since Welch, in this example, is the former CEO of General Electric (GE), whose headquarter is in the United States. In the second 0-shot example, our model inferred that the Justice:Fine event should take place in a court, which matches our commonsense knowledge. Interestingly, we observe that increasing the number of in-context examples from 0-shot to 50- shot inhibits LLM from generating arguments (i.e., making LLMs more conservative), including these inferred arguments and a correctly predicted argument (i.e., SEC) in 0-shot predictions.
A.2 Prompt Component Analysis
In this section, we present an empirical analysis of other prompt component candidates. We compare different prompt components in Table A.1 using code-davinci-002 and following the same hyper-parameters described in §4.1.
• Event Keywords We augment event-related keywords into the docstring of event definition for CODE4STRUCT (illustrated in Figure A.8). We follow the same keywords used by Li et al. (2021).
• AMR Zhang and Ji (2021) have demonstrated the effectiveness of utilizing Abstract Meaning Representation (AMR) (Banarescu et al., 2013) for information extraction. We experiment with AMR-augmented prompts. We use armlib 12 to predict AMR, and append the AMR structure after the NL sentence in the task prompt §2.2 (see Figure A.7 for an example).
Prompts that include event keywords and AMR all perform slightly better than CODE4STRUCT under the zero-shot setting on all metrics (Table A.1).
12https://github.com/bjascob/amrlib, parse_xfm_bart_large v0.1.0
A.3 Ablation Study In Table A.2, we ablate different prompt components described in §2, including event trigger marking, event description in natural language, type annotation, and hierarchical ontology. We perform this ablation study using code-davinci-002.
Event Trigger Marking We find that removing event trigger marking consistently degrades per-
formance on all metrics over varying numbers of in-context examples.
Event Description Event descriptions generally provide a small F1 gain under the few-shot setting. However, removing event descriptions improves CODE4STRUCT’s zero-shot performance on argument identification. 0-shot Arg-I precision is relatively unchanged after removing event descriptions
(37.4 vs. 37.2). We argue that removing event descriptions loosens entity-related constraints and allows LLM to identify more relevant entities. This is supported by the improvement of 0-shot Arg-I recall (78.7 to 81.8) after description removal, which mainly accounts for the increase in 0-shot Arg-I F1. Despite being helpful in argument identification by boosting 0-shot Arg-I recall, we do not see the benefit of removing descriptions in few-shot Arg-C, where it performs consistently worse compared to CODE4STRUCT.
Type Annotation Type annotation is more helpful when more in-context examples are provided (k ≥ 20). Under a low-shot setting, the F1 difference resulting from type annotation removal is small and inconsistent across different shots. Prompts with type annotation consistently outperforms prompts without it when sufficient in-context examples are provided (k ≥ 20). We hypothesize that type annotations help disambiguate entity types accepted for each argument, and such disambiguation ability is only needed when the number of entity instances that appeared in in-context examples passes a certain threshold (e.g., k ≥ 20).
Hierarchical Event Definition Providing hierarchical event definition (i.e., the parent class definition of a given child event class) benefits CODE4STRUCT performance in high-shot setting (k ≥ 20). Prompts without parent class definition perform on par with CODE4STRUCT under k < 20.
A.4 In-context Example Variability Analysis
1 5 10 20 50 Number of in-context examples (k-shot)
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
Av er
ag e
eu cli
de an
d ist
an ce
fr om
c en
tro id
5-shot: 0.75
20-shot: 0.84 50-shot: 0.86
Variability of in-context examples across different event types
std mean
Figure A.2: The variability of k-shot in-context examples (i.e., average euclidean distance from centroid example) across different event types increases with diminishing returns when k increases.
To investigate why the Arg-C performance plateaus with an increasing number of in-context examples k as shown in Figure 3, we analyze the sentence variability of a fixed set of incontext examples (§2.3). We consider the set of k-shot in-context examples for each event type e as a cluster De where |De| ≤ k and use sentence-transformer13 to embed all the input sentences from De into a cluster of vectors Ve.
We use the average euclidean distance from the centroid example similar to (Halkidi et al., 2001) to measure the variability of in-context examples for each event type e:
Variability(e) = 1 |Ve| ∑
v∈Ve d(v, v̄)
where d(·, ·) is the euclidean distance between two vectors and v̄ = 1|Ve| ∑ v∈Ve v is the centroid example of the cluster Ve. We calculate the mean Variability(e) across all e for k ∈ {1, · · · , 50}. In Figure A.2, similar to Arg-C performance in Figure 3, we find the mean Variability(e) across all e increases with diminishing returns with increasing k. Furthermore, we find that, in Figure A.3, Arg-C F1 performance is positively correlated with the mean Variability(e) across all e. This suggests the lack of in-context example variability improvement could be one of the reasons Arg-C F1 plateaus, even when more examples are given.
13all-mpnet-base-v2 model
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7: First section after the conclusion
3 A2. Did you discuss any potential risks of your work? Section 8: Second section after the conclusion
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1 (Introduction)

heading: 7 A4. Have you used AI writing assistants when working on this paper?
text: Left blank.
B 3 Did you use or create scientific artifacts? Section 4
3 B1. Did you cite the creators of artifacts you used? Section 4
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In the appendix
C 3 Did you run computational experiments? Section 4

heading: 7 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? We prompt OpenAI hosted language model; we include the cost in the limitation section.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4 and the Ablation study in the appendix
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4 and Appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We leave automatically discover hierarchical ontology for future work. Despite LLMs performing well on EAE with few-shot data, compared to existing supervised approaches, their inference is relatively slow and costly11 since the LLMs we used are generally more than 100x larger in the number of parameters. Prior work (Zhao et al. , 2021; Lu et al. , 2022) has demonstrated a strong relationship between performance and in-context demonstrations; however, for ease of comparison to supervised baselines, we use the same set of examples from the training set for in-context learning. We expect better selecting (Liu et al. , 2021) and ordering (Lu et al. , 2022) incontext examples can benefit CODE4STRUCT performance, which we leave for future work. Ethical Considerations
Since event argument extraction only requires predicting arguments from the given text, the risk of generating toxic languages is relatively low as long as the given test is not toxic. This is because the prediction can be grounded in the input sentence, eliminating potential toxic tokens that did not appear in the original sentence. However, discrimination and bias are possible, as observed in the foundational LLMs we used (Brown et al. , 2020; Chen et al. , 2021; Ouyang et al. , 2022), which we refer to Brown et al