AMPERE: AMR-Aware Prefix for Generation-Based Event Argument Extraction Model
Year: 2023

Abstract:
Event argument extraction (EAE) identifies event arguments and their specific roles for a given event. Recent advancement in generationbased EAE models has shown great performance and generalizability over classificationbased models. However, existing generationbased EAE models mostly focus on problem reformulation and prompt design, without incorporating additional information that has been shown to be effective for classification-based models, such as the abstract meaning representation (AMR) of the input passages. Incorporating such information into generation-based models is challenging due to the heterogeneous nature of the natural language form prevalently used in generation-based models and the structured form of AMRs. In this work, we study strategies to incorporate AMR into generationbased EAE models. We propose AMPERE, which generates AMR-aware prefixes for every layer of the generation model. Thus, the prefix introduces AMR information to the generationbased EAE model and then improves the generation. We also introduce an adjusted copy mechanism to AMPERE to help overcome potential noises brought by the AMR graph. Comprehensive experiments and analyses on ACE2005 and ERE datasets show that AMPERE can get 4%− 10% absolute F1 score improvements with reduced training data and it is in general powerful across different training sizes.

heading: 1 Introduction
text: Event argument extraction (EAE) aims to recognize event arguments and their roles in an event. For example, in Figure 1, EAE models need to extract districts, u.s. supreme court, and washington and the corresponding roles — Plaintiff, Adjudicator, and Place for the Justice:Appeal event with trigger appeal. EAE has long been a challenging task in NLP, especially when training data is limited (Wang et al., 2019; Ma et al., 2022). It is
∗The authors contribute equally.
an important task for various downstream applications (Zhang et al., 2020; Berant et al., 2014; Hogenboom et al., 2016; Wen et al., 2021; Wu et al., 2022).
Recently, generation-based EAE models (Hsu et al., 2022a; Lu et al., 2021; Li et al., 2021; Paolini et al., 2021; Parekh et al., 2022) are proposed and have shown great generalizability and competitive performance compared to traditional classificationbased methods (Chen et al., 2015; Ma et al., 2020; Hsu et al., 2022b; Fincke et al., 2022). However, existing generation-based EAE models mostly focus on problem reformulation and prompt design without incorporating auxiliary syntactic and semantic information that is shown to be effective in classification-based methods (Huang et al., 2016; Xu and Huang, 2022; Huang et al., 2018; Ahmad et al., 2021; Veyseh et al., 2020).
In this work, we explore how to incorporate auxiliary structured information into generation-based EAE models. We focus on abstract meaning representation (AMR) (Banarescu et al., 2013), which extracts rich semantic information from the input sentence. As the Figure 1’s example shows, AMR graph summarizes the semantic structure of the input passage, and many of its nodes and edges share strong similarities with the event structures. For example, the trigger word appeal can be mapped to the node “appeal-01”, and the subject who appeals can be found using edge “ARG0”. Hence, the AMR graph could provide important clues for models to figure out event arguments, resulting in performance improvements (Zhang and Ji, 2021) and better generalizability (Huang et al., 2018) for classification-based methods. However, it is unclear how to best integrate AMR into generationbased methods. The heterogeneous nature between the AMR graph and the natural language prompts1 in the generation-based EAE models causes the
1For example, event type description and target generation templates.
10976
difficulty of the model design. To overcome the challenge, we propose AMPERE (AMr-aware Prefix for generation-based Event aRgument Extraction), which encodes AMR graph into prefix (Li and Liang, 2021) to regulate the generation-based EAE models. Specifically, an additional AMR encoder is used to encode the input AMR graph into dense vectors. Then, these vectors will be disassembled and distributed to every Transformer layer in generation-based EAE models as the prefix. These generated prefixes are transformed into additional key and value matrices to influence the attention calculation, hence, guiding the generation.
We also introduce an adjusted copy mechanism for AMPERE to overcome potential noises brought by the AMR graph. Specifically, as we can observe in Figure 1, AMR parsers will include additional normalization (turning washington into Washington) and word disambiguation (using appeal-01 rather than appeal) to create AMR graphs. Such normalization could impact the generation to produce some words that are not in the original input, especially when the training data is limited. Hence, we apply a copy mechanism (See et al., 2017) and add an additional regularization loss term to en-
courage copying from the input passage. We conduct experiments on ACE 2005 (Doddington et al., 2004) and ERE (Song et al., 2015) datasets using different ratios of training data. Our results show that AMPERE outperforms several prior EAE works in both datasets. Under lowresource settings that use only 5% or 10% of training data, we can get 4%− 10% absolute F1-scores of improvement, and our method is in general powerful across different training sizes and different datasets. We also present a comprehensive study of different ways to incorporate AMR information into a generation-based EAE model. We will show that AMPERE is the best way among the various methods we explored. Our code can be found at https://github.com/PlusLabNLP/AMPERE.

heading: 2 Method
text: AMPERE uses DEGREE (Hsu et al., 2022a) as the base generation-based EAE model 2 (Section 2.1), and augments it with AMR-aware prefixes, as shown in Figure 1. To generate the AMR-aware prefixes, we first use a pre-trained AMR parser to obtain the AMR graph of the input sentence (Sec-
2We use the EAE version of DEGREE (Hsu et al., 2022a).
tion 2.2). Then, the graph is transformed into dense vectors through graph linearization and an AMR encoder. Then, these dense vectors will be disassembled and distributed to each layer of our base generation-based EAE model so the generation is guided by the AMR information (Section 2.3). Finally, we introduce the training loss for AMPERE and our adjusted copy mechanism that can help AMPERE overcome additional noise brought from AMR graphs (Section 2.4).

heading: 2.1 Generation-Based EAE Model
text: Despite our AMR-aware prefix being agnostic to the used generation-based EAE model, we select DEGREE (Hsu et al., 2022a) as our base model because of its great generalizability and performance. Here, we provide a brief overview of the model.
Given a passage and an event trigger, DEGREE first prepares the prompt, which includes an event type description (a sentence describing the trigger word), and an event-type-specific template, as shown in Figure 1. Then, given the passage and the prompt, DEGREE summarizes the event in the passage following the format of the EAE template, so that final predictions can be decoded easily by comparing the template and the output text. Take the case in Figure 1 as an example, by comparing “districts in washington appealed the adjudication from u.s. supreme court.” with the template “somebody in somewhere appealed the adjudication from some adjudicator.”, we can know that the “districts” is the argument of role “Plaintiff”. This is because the corresponding placeholder “somebody” of the role “Plaintiff” has been replaced by “districts” in the model’s prediction.

heading: 2.2 AMR Parsing
text: The first step of our method is to prepare the AMR graph of the input passage. We consider SPRING (Bevilacqua et al., 2021), a BART-based AMR parser trained on AMR 3.0 annotation, 3 to be our AMR parser. As illustrated by Figure 1, the AMR parser encodes the input sentence into an AMR graph, which is a directed graph where each node represents a semantic concept (e.g., “give01”, “appeal-01”) and each edge describe the categorical semantic relationship between two concepts (e.g., ARG0, location) (Banarescu et al., 2013).
3https://catalog.ldc.upenn.edu/LDC2020T02

heading: 2.3 AMR-Aware Prefix Generation
text: Our next step is to embed the information into prefixes (Li and Liang, 2021) for our generation-based EAE model. To encode the AMR graph, we follow Konstas et al. (2017) to adopt a depth-first-search algorithm to linearize the AMR graph into a sequence, as shown in the example in Figure 1. Then, an AMR encoder is adapted to encode the representation of the sequence. One of the advantages of our method is the flexibility to use models with different characteristics to our generation-based EAE model to encode AMR. Here, we consider two AMR encoders to form different versions of AMPERE:
• AMPERE (AMRBART): We consider using the encoder part of the current state-of-the-art AMRto-text model — AMRBART (Bai et al., 2022) that pre-trained on AMR 3.0 data. 4 The model is based on BART-large and its vocabulary is enlarged by adding all relations and semantic concepts in AMR as additional tokens. Employing the model as our AMR encoder enables AMPERE to leverage knowledge from other tasks.
• AMPERE (RoBERTa): RoBERTa-large (Liu et al., 2019b) is also considered as our AMR encoder as pre-trained masked language models are typical choices to perform encoding tasks. In order to make RoBERTa better interpret the AMR sequence, we follow Bai et al. (2022) to add all relations in AMR (e.g. ARG0, ARG1) as special tokens. However, since the model is not pre-trained on abundant AMR-to-text data, we do not include semantic concepts (e.g. concepts end with -01) as extra tokens. 5
After getting the representation of the linearized sequence, we then prepare l learnable vectors as queries and an attention layer, where l is a hyperparameter that controls the length of the used prefixes. These queries will compute attention with the representations of the linearized AMR sequence, then, we will obtain a set of compressed dense vector P. This P will be transformed into the prefixes (Li and Liang, 2021) that we will inject into our generation-based EAE model.
To be more specific, we first disassemble P into L pieces, where L is the number of lay-
4https://github.com/goodbai-nlp/AMRBART 5If adding semantic concepts as extra tokens, RoBERTa will loss the ability to grasp its partial semantic meaning from its surface form, such as understanding that “appeal-01” is related to “appeal”.
ers in the base generation-based EAE model, i.e., P = {P 1, P 2, ...PL}. Then, in the n-th layer of the EAE model, the prefix is separated into two matrices, standing for the addition key and value matrices: Pn = {Kn, V n}, where Kn & V n are the addition key and value matrices, and they can be further written as Kn = {kn1 , ...,knl } and V n = {vn1 , ...,vnl }. k∗ and v∗ are vectors with the same hidden dimension in the Transformer layer. These additional key and value matrices will be concatenated with the original key and value matrices in the attention block. Therefore, when calculating dot-product attention, the query at each position will be influenced by these AMR-aware prefixes. The reason of generating layer-wise queries and keys is to exert stronger control. We generate layer-wise key-value pairs as each layer may embed different information. These keys influence the model’s weighting of representations towards corresponding generated values. Empirical studies on layer-wise versus single-layer control can be found in Liu et al. (2022b).
It is worth noting that Li and Liang (2021)’s prefix tuning technique uses a fixed set of prefixes disregarding the change of input sentence, AMPERE will generate a different set of prefixes when the input passage varies. And the variation reflects the different AMR graph’s presentation.
We can inject prefixes into the encoder selfattention blocks, decoder cross-attention blocks, or decoder self-attention blocks in our generationbased EAE model. Based on our preliminary experiments, we observe that using prefix in encoder self-attention blocks and decoder cross-attention blocks works best in AMPERE.

heading: 2.4 Adjusted Copy Mechanism
text: We follow DEGREE’s setting to use BART-large (Lewis et al., 2020) as the pretrained generative model, and the training objective of our generation-based EAE model is to maximize the conditional probability of generating a groundtruth token given the previously generated ones and the input context in the encoder x1, x2, ..xm:
Loss = − log( ∑
i
P (yi|y<i, x1, ..., xm)), (1)
where yi is the output of the decoder at step i. In DEGREE’s setting, the probability of predicting an token t fully relies on the generative model. Although this setting is more similar to how
BART-large is pre-trained and thus better leverages the power of pre-training, the loose constraints on the final prediction could generate hallucinated texts (Ji et al., 2022) or outputs not following the template. Such an issue could be enlarged if less training data is used and more input noise is presented, such as when incorporating AMR graphs.
To enhance the control, one commonly-used technique is to apply copy mechanism (See et al., 2017) to generation-based event models (Huang et al., 2022, 2021). , i.e.,
P (yi = t|y<i, x1, .., xm) = wigenPgen(yi = t|y<i, x1, ..., xm))+
(1− wigen)( m∑
j=0
P icopy(j|y<i, x1, ..., xm)× 1(xj = t)),
(2)
where wigen ∈ [0, 1] is the probability to generate, computed by passing the last decoder hidden state to an additional network. P icopy(j|·) is the probability to copy input token xj , and it’s computed by using the cross-attention weights in the last decoder layer at time step i. When wigen = 1, it is the original model used by DEGREE, while if wigen = 0, this model will only generate tokens from the input.
Our core idea of the adjusted copy mechanism is to encourage the model to copy more, and this is achieved by introducing a regularization term on wigen to the loss function of AMPERE:
LossAMPERE =
− log( ∑
i
P (yi|y<i, x1, ..., xm)) + λ ∑
i
wigen, (3)
where λ is a hyper-parameter. Compared to fully relying on copy from input, our method still allows the generative model to freely generate tokens not presented in the input. Compared to ordinary copy mechanisms, the additional regularizer will guide the model to copy more. Using this loss, we train the whole AMPERE end-to-end.

heading: 3 Experiments
text: We conduct experiments to verify the effectiveness of AMPERE. All the reported numbers are the average of the results from three random runs.

heading: 3.1 Experimental Settings
text: Datasets and Data split. We adopt the event annotation in ACE 2005 dataset (Doddington et al., 2004) (ACE05-E)6, and the English split in ERE
6https://catalog.ldc.upenn.edu/LDC2006T06
dataset (Song et al., 2015) (ERE-EN)7. ACE 2005 contains files in English, Chinese, and Arabic, and ERE includes files in English and Chinese. In this paper, we only use the documents in English, and split them to sentences for use in our experiments. We follow prior works (Wadden et al., 2019; Lin et al., 2020) to preprocess each dataset. After preprocessing, ACE05-E has 33 event types and 22 argument roles, and ERE-EN are with 38 event types and 21 argument roles in total. Further, we follow Hsu et al. (2022a) to select 5%, 10%, 20%, 30%, and 50% of training samples to generate the different data split as the training set for experiments. The data statistics are listed in Table 4 in the Appendix A.8
Evaluation metrics. We report the F1-score for argument predictions following prior works (Wadden et al., 2019; Lin et al., 2020). An argument is correctly identified (Arg-I) if the predicted span matches the span of any gold argument; it is cor-
7https://catalog.ldc.upenn.edu/LDC2020T19 8The license for ACE 2005 and ERE is LDC User Agreement for Non-Members. In both datasets, event types like Conflict:Attack may include offensive content such as information related to war or violence. Some passages are extracted from broadcast news, thus some real names may appear in the data. Considering the datasets are not publicly available, and these contents are likely to be the arguments to extract in our task, we do not change the data for protecting or anonymizing.
rectly classified (Arg-C) if the predicted role type also matches.
Implementation details. We use the AMR tools as we mentioned in Section 2. When training our models, we set the learning rate to 10−5. The number of training epochs is 60 when training on ACE05E, and 75 when training on ERE-EN. We simply set λ as 1 for all our models. We do hyperparameter searching using the setting that trains on 20% of data in ACE05E and selects the best model based on the development set results. We set l = 40, and batch size is set to 4 for AMPERE (AMRBART) and 6 for AMPERE (RoBERTa) in the end. This is searching from l = {30, 40, 50} and batch size = {4, 6, 8, 12}.
Baselines. We compare AMPERE with the following classification-based models: (1) DyGIE++ (Wadden et al., 2019), which extracts information by scoring spans with contextualized representations. (2) OneIE (Lin et al., 2020), a joint IE framework that incorporates global features. (3) Query and Extract (Wang et al., 2022), which uses attention mechanisms to evaluate the correlation between role names and candidate entities. (4) AMR-IE (Zhang and Ji, 2021), which captures non-local connections between entities by aggregating neighborhood information on AMR
graph, and designed hierarchical decoding based on AMR graph information. We also consider the following generation-based models: (5) PAIE (Ma et al., 2022), a framework that integrated prompt tuning, and generates span selectors for each role. 9 (6) DEGREE (Hsu et al., 2022a). The generationbased EAE model we used as our base model.
To ensure a fair comparison across models, we adopt the official codes of the above baselines to train them on the identical data and did hyperparameter tuning. For all the classification-based methods, we use RoBERTa-large, and for all the generation-based methods, we use BART-large as the pre-trained language models. Appendix §B shows details about the implementation.

heading: 3.2 Results
text: Table 1 shows the argument classification (ArgC) F1-scores in ACE05-E and ERE datasets under different data proportions. Overall, both AMPERE (RoBERTa) and AMPERE (AMRBART) consistently outperform all other baselines except the test set results of using 50% data in ERE-EN.
From the table, we can notice that AMPERE significantly outperforms our base model DEGREE in all experiments in ACE05-E, and in ERE-EN, the improvement is also considerable. When trained with less than 20% data in ACE05-E, AMPERE (RoBERTa) can consistently achieve more than 4 points of improvement over DEGREE in both the development and test sets. In the following Section 4, we will further discuss the detailed contribution of our method over DEGREE.
To quantitatively evaluate the effectiveness of AMR’s incorporation, we can first check the performance of AMR-IE. AMR-IE achieves competitive performance among classification-based models, especially under extremely low-resource settings. This is coincident with how AMPERE’s result shows. AMPERE outperforms both DEGREE and PAIE, and the gap is more obvious under lowresource settings. For example, in the 5% data proportion setting, AMPERE (RoBERTa) made over 11 points of improvement over DEGREE in ACE05E’s test Set. In the meanwhile, AMPERE (RoBERTa) achieves 4.4 points of performance gain compared with PAIE. All this shows the empirical evidence that AMR information can hint to the models’ se-
9PAIE requires manually designed prompts for model training, and ERE-EN dataset is not considered in their official codebase. Hence, we do not include PAIE as a baseline on ERE-EN.
mantic structure of the input passage, and this is especially helpful for models when training samples are limited. Despite the strong performance of AMR-IE, AMPERE can still outperform it across all the settings, indicating the effectiveness of our method.
Comparing AMPERE (AMRBART) and AMPERE (RoBERTa), we show that our proposed method does not necessarily rely on pre-trained AMR-toText models. Particularly, AMPERE (RoBERTa), which employs a pre-trained RoBERTa-large as the AMR Encoder still achieves competitive results to AMPERE (AMRBART), which uses AMR-to-Text data. Yet, the advantage of using AMR-to-Text data as an auxiliary is that we can get similar results with less parameters. The AMR encoder component of AMPERE (RoBERTa) has approximately 1.7 times more parameters than that of AMPERE (AMRBART), as we only use the encoder part of AMRBART in AMPERE(AMRBART). Nevertheless, the pre-trained knowledge from AMR-to-text data enables AMPERE (AMRBART) to perform competitively with AMPERE (RoBERTa).

heading: 4 Analysis
text: In this section, we present comprehensive ablation studies and case studies to validate our model designs. Two essential parts of our design, the AMRaware prefix, and the adjusted copy mechanism will be examined in the following studies. For all the experiments in this section, we use the setting of training on 5% and 20% ACE05-E data to simulate very low-resource and low-resource settings.

heading: 4.1 Different Ways for AMR Incorporation
text: We compare different ways to incorporate AMR information into generation-based EAE models:
• AMR Prompts. We follow the same process as AMPERE to obtain the linearized AMR graph sequence. Then, we directly concatenate the linearized AMR graph sequence to the input text as part of the prompts.
• AMRBART Encoding Concatenation. After obtaining the AMR sequence representations after the AMR encoder using AMRBART, we concatenate this encoding with the output representation in our generation-based EAE model and feed them together to the decoder.
• RoBERTa Encoding Concatenation. The method is similar to the AMRBART Encoding Concatenation method, except that we use
RoBERTa as the AMR encoder. 10
For comparison, we provide AMPERE’s performance without any AMR incorporation as a baseline. Additionally, we also consider AMPERE with frozen AMR encoder11 in the comparisons to exclude the concern of extra learnable parameters of AMPERE compared to baselines such as AMR Prompts. Note that all the mentioned models above are implemented with our proposed adjusted copy mechanism. Table 2 shows the results.
From the table, we observe that AMPERE gets the best performance among all the ways we explored and achieves 4.9% and 4.2% F1-score improvements over the model without AMR incorporation under the case of using 5% & 20% of training data, respectively.
An interesting finding is that the performance of AMR Prompts is worse than the model without any AMR incorporation in the very low-resource setting (5% data). As mentioned in Section 1, the heterogeneous nature between AMR graph and natural language sentences is an important intuitive for our model design. AMR often uses special tokens such as :ARG0 or appeal-01, and in implementation like AMR Prompts, it would be confusing for models when training samples are not sufficient.
Furthermore, due to the heterogeneous vector space between AMRBART and RoBERTa, RoBERTa Encoding Concatenation method could not work well. In comparison, the prefix design of AMPERE shows strong adaptability, as AMPERE (AMRBART) and AMPERE (RoBERTa) both outperform the other implementation methods.
10We also explore the variation that we add a linear layer to the AMR encoding to help space alignment, but there is little performance difference on both AMRBART Encoding Concatenation & RoBERTa Encoding Concatenation.
11For these type of models, during training, the AMR encoder’s parameter is fixed. Hence the number of learnable parameters is comparable to DEGREE.
Finally, we focus on the results from AMPERE with frozen AMR Encoder. We can observe that despite slight performance degradation compared to fully-trainable AMPERE, AMPERE with frozen AMR Encoder still obtain at least 1% absolute F1scores improvements over other AMR incorporation methods.

heading: 4.2 Studies of Adjusted Copy Mechanism
text: To justify the effectiveness of our adjusted copy mechanism, we compare our adjusted copy mechanism with the following method:
• AMPERE w/o any copy. For comparison, we adopt a normal generation-based model adapted with AMR-aware prefixes.
• AMPERE w/ pure copy.: In Equation 2, we directly set wigen = 0. In other words, tokens not presented in the input can not be generated.
• AMPERE w/ ordinary copy mechanism. We apply the copy mechanism but train the model with the loss function in Equation 1.
In Table 3, the experiment with AMPERE (AMRBART) and AMPERE (RoBERTa) lead to similar conclusions. Any kind of copy mechanism can lead to noticeable improvement, and the performance gap between methods with and without copy mechanism is larger in the lower data proportion setting. Our adjusted copy mechanism stably outperforms the other methods in studies. Compared to the traditional copy mechanism, our method encourages the model to copy more, hence can stably overcome the very low-resource challenges. Compared to fully relying on copy from input, our method allows the generative model to freely generate tokens not presented in the input, so as to better leverage the pre-trained language model’s power, leading to better performance when data is slightly more available.

heading: 4.3 Case Study

heading: 4.3.1 Output Examples
text: To intuitively explain the benefit of our method over previous generation-based EAE models, we present examples here to showcase the influence of incorporating AMR information. We compare AMPERE and DEGREE that both trained on 20% ACE05-E data and demonstrate two examples in Figure 2 to show the difference of their generated output text.
Example A presents a case where the edges in the AMR graph helps the model to classify the correct role type of argument "government". Without AMR information, DEGREE incorrectly predicts the "government" to be the agent that launched some organization. In the AMR graph, edge ARG1 points to the object of the action concept form-01. Thus, in the AMPERE’s output, "government" is correctly classified as the object of "form".
Example B in Figure 2 shows how the AMR graph hints AMPERE about the argument "judge". By looking up the subject of verb "order" in the AMR graph, the model is able to find the adjudicator of the event. Thus, AMPERE could correctly replace the adjudicator placeholder in the template with real adjudicator, “judge”.

heading: 4.3.2 Error Analysis
text: To point out future research direction for generation-based EAE models, we performed error analysis on 30 cases where our AMPERE (RoBERTa) made mistakes. We identified two common types of errors: (1) ambiguous span boundaries, and (2) incorrect distinction between events of the same type.
For instance, in the case of “ambiguous span boundaries,” AMPERE (RoBERTa) incor-
rectly predicted "Christian Ayub Masih" instead of the correct label "Ayub Masih." We observe that generation-based models struggle to accurately predict span boundaries, as both AMPERE (RoBERTa)’s output and the ground truth can fit into the sentence template coherently. Even with the inclusion of AMR, the model’s ability to identify potential boundaries from the AMR graph through learning remains limited.
Regarding the issue of “incorrect distinction between events of the same type,” we present an example to illustrate this. In the given input sentence, “As well as previously holding senior positions at Barclays Bank, BZW and Kleinwort Benson, McCarthy was formerly a top civil servant at the Department of Trade and Industry.”, the model becomes confused between the two "Personnel:EndPosition" events, each triggered by “previousl” and “formerly”, respectively, due to subtle differences. We suggest that incorporating additional structural knowledge, such as dependency parsing information, to separate the sentences structurally, could be a potential solution. However, we leave this research as future works.

heading: 5 Related Work
text: Generation-based event (argument) extraction models. Traditionally, most models for EAE are classification-based (Chen et al., 2015; Ma et al., 2020; Hsu et al., 2022b; Fincke et al., 2022). Recently, generation-based EAE models (Hsu et al., 2022a; Lu et al., 2021; Li et al., 2021; Paolini et al., 2021) become more and more popular due to their flexibility to present different output structures (Yan et al., 2021), to be unified considered with similar tasks (Lu et al., 2022), and their competitive performance (Hsu et al., 2022a; Liu et al., 2022a).
The development of generation-based event (argument) extraction models starts from works investigating how to reformulate event extraction problems as a generation task (Du et al., 2021a,b). Follow-up works put efforts to show the influence of different prompt designs to the generative event models. (Ma et al., 2022; Yao et al., 2022; Hsu et al., 2022a) More recently, researchers start to improve this series of work by designing different model architectures (Du et al., 2022; Zhu et al., 2022). However, very few efforts have been put into the ways and the effectiveness of incorporating auxiliary syntactic and semantic information into such models, even though this information has been shown to be beneficial in classification-based models. Hence, in this paper, we present the study and explore ways to incorporate this additional information for generation-based event models.
Improving event extraction with weaklysupervisions. Being a challenging task that requires deep natural language understanding to solve, many prior efforts have been put into investigating which auxiliary upstream task information is useful for event predictions. (Xu and Huang, 2022; Liu et al., 2019a; Huang et al., 2018; Veyseh et al., 2020) Liu et al. (2019a); Ahmad et al. (2021) leverages dependency syntactic structures of the input sentence to help cross-lingual event predictions. Huang et al. (2016, 2018) uses the similarity between AMR and event structures to perform zeroshot event extraction. More recently, Zhang and Ji (2021); Veyseh et al. (2020); Xu et al. (2022) investigates different message passing methods on AMR graph to help learn better representations for final classifications. Despite many efforts that
have been put into the community, these methods are designed for classification-based models. This highlights the open area for research — how and whether incorporating such auxiliary information can also be helpful. We take a step forward in this direction and present AMPERE to showcase the possibility to improve the generation-based event models by such way.

heading: 6 Conclusion
text: In this paper, we present AMPERE, a generationbased model equipped with AMR-aware prefixes. Through our comprehensive studies, we show that prefixes can serve as an effective medium to connect AMR information and the space of generative models, hence achieving effective integration of the auxiliary semantic information to the model. Additionally, we introduce an adjusted copy mechanism to help AMPERE more accurately and stably generate output disregarding the additional noise brought from the AMR graph. Through our experiments, we show that AMPERE achieves consistent improvements in every setting, and the improvement is particularly obvious in low-resource settings.

heading: Acknowledgments
text: We thank anonymous reviewers for their helpful feedback. We thank the UCLA PLUSLab and UCLA-NLP group members for their initial review and feedback for an earlier version of the paper. This research was supported in part by AFOSR MURI via Grant #FA9550-22-1-0380, Defense Advanced Research Project Agency (DARPA) via Grant #HR00112290103/HR0011260656, the Intelligence Advanced Research Projects Activity
(IARPA) via Contract No. 2019-19051600007, National Science Foundation (NSF) via Award No. 2200274, and a research award sponsored by CISCO.
Limitations
Our goal is to demonstrate the potential of incorporating AMR to improve generation-based EAE models. Although we have shown the strength of our method, there are still some limitations. First, our proposed techniques are based on the AMR graph generated by pre-trained AMR parsers. The generated AMR graphs inevitably have a certain possibility of being not perfect. Hence, the error propagation issues would happen to AMPERE. We hypothesize this is one of the reasons why the improvement of AMPERE is not necessarily significant when data is abundant. Yet, through our experimental results, we still show the benefit of incorporating this information, especially in the case of low-resource settings. Second, although our AMRaware prefix design should be agnostic to the used generation-based EAE model, in our experiment, we only set DEGREE as our base generation-based EAE model. We leave the investigation on the generalizability of our AMR-prefix method to other base models as future work.
Ethics Considerations
Our method relies on a pre-trained AMR parser, which is built using pre-trained large language models (AMRBART & RoBERTa). It is known that the models trained with a large text corpus may capture the bias reflecting the training data. Therefore, it is possible that the AMR graph used in our method could contain certain biases. We suggest carefully examining the potential bias before applying AMPERE to any real-world applications.

heading: A Datasets
text: We present detailed dataset statistics in Table Table 4.
B Implementation Details
This section introduces the implementation details for all the baseline models we use in this paper. Our experiments are run using our machine that equips 8 NVIDIA a6000 GPUs.
• DyGIE++: we use their official code to reimplement the model.12 Their original model is built using BERT (Devlin et al., 2019). As we mentioned in Section 3.1, we replace the used pre-trained language model into RoBERTa-large and tune with learning rates = {1e− 5, 2e− 5, 3e− 5}.
• OneIE: we use their official code13 to train the model. Their original model is built using BERT (Devlin et al., 2019). As we mentioned in Section 3.1, we replace the used pretrained language model into RoBERTa-large and tune with learning rates = {1e− 5, 2e− 5, 3e− 5}.
• Query and Extract: we use their official code14 to train argument detection model with learning rate = 1e− 5, batch size = 16, training epoch = 10. Different from the official code, we used RoBERTa-large for pre-trained language model to ensure a fair comparison.
• AMR-IE: the original AMR-IE is an end-toend event extraction model, so we adapt their official code15 to event argument extraction task by giving gold triggers in model evaluation. We fixed pre-trained language model learning rate = 1e − 5, then did hyperparameter searching from graph learning rate = {1e− 3, 4e− 3} and batch size = {8, 16}.
• PAIE: we use their official code16 to train the model with the default parameters for BART-large.
12https://github.com/dwadden/dygiepp 13http://blender.cs.illinois.edu/software/
oneie/ 14https://github.com/VT-NLP/Event_Query_ Extract/ 15https://github.com/zhangzx-uiuc/AMR-IE 16https://github.com/mayubo2333/PAIE/
• DEGREE: we use their official code17 to train the model with the default parameters for BART-large.

heading: C Detailed Result
text: Table 5 shows the detailed results of our main experiments. We repeat running every experiment setting with three random seeds, and report their average Arg-I and Arg-C F1-scores, and the corresponding standard deviation scores.
17https://github.com/PlusLabNLP/DEGREE
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations Section

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Considerations Section

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract Section & Section 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3 (Datasets)

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix A

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix A

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Appendix A

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3 (Datasets)
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
C 3 Did you run computational experiments? Section 3 & Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3 (Implementation Details)

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 2 (AMR Parser, AMRBART)
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave the investigation on the generalizability of our AMR-prefix method to other base models as future work