Venue: ACL 2023

# Event Extraction as Question Generation and Answering

Di Lu  Shihao Ran  Joel Tetreault  Alejandro Jaimes

Dataminr Inc.

{dlu,sran,jettreault,ajaimes}@dataminr.com

###### Abstract

Recent work on Event Extraction has reframed the task as Question Answering (QA), with promising results. The advantage of this approach is that it addresses the error propagation issue found in traditional token-based classification approaches by directly predicting event arguments without extracting candidates first. However, the questions are typically based on fixed templates and they rarely leverage contextual information such as relevant arguments. In addition, prior QA-based approaches have difficulty handling cases where there are multiple arguments for the same role. In this paper, we propose QGA-EE, which enables a Question Generation (QG) model to generate questions that incorporate rich contextual information instead of using fixed templates. We also propose dynamic templates to assist the training of QG model. Experiments show that QGA-EE outperforms all prior single-task-based models on the ACE05 English dataset.1

Footnote 1: Our code is available at [https://github.com/datumir-ai/Event-Extraction-as-Question-Generation-and-Answering](https://github.com/datumir-ai/Event-Extraction-as-Question-Generation-and-Answering) for research purposes.

## 1 Introduction

Event Extraction (EE) aims to extract core information elements (e.g. who, what, where, when) from text, and is a very important task in Natural Language Processing (NLP). It provides inputs to downstream applications such as Summarization Filatova and Hatzivassiloglou (2004), Knowledge Base Population Ji and Grishman (2011), and Recommendation Lu et al. (2016).

Previous work Li et al. (2013); Nguyen et al. (2016); Sha et al. (2018) is typically based on a pipeline approach, which first identifies the event trigger word/phrase and argument candidates, and then applies a classifier to the pair-wise features to classify the roles of the candidates. Unfortunately, errors tend to propagate down the pipeline. Recently, some approaches have formulated EE as a Question Answering (QA) problem Du and Cardie (2020); Li et al. (2020); Lyu et al. (2021) to mitigate the issue, in which questions for each argument role are manually defined by templates. For example, extracting the Attack argument from the Conflict.Attack event in the sentence _'That's because coalition fighter jets pumneled this Iraqi position on the hills above Chanchamal and Iraqi troops made a hasty retreat.'_ is reframed as answering the question _'Who was the attacking agent?'_ These approaches have shown promising results, but template-based questions are limiting: since the templates are built manually, they are fixed and rarely include contextual information (i.e., specific to the inputs), except for trigger words in some work Du and Cardie (2020). Formulating good questions, however, has been shown to improve performance for standard QA tasks Rajpurkar et al. (2018). For QA-based EE, a question that incorporates richer contextual information such as other event arguments could yield better results (e.g. _'Who used jets in the attack in hills?'_ in Figure 1).

In this paper, we propose QGA-EE, which consists of 1) a QG model for generating a context-aware question conditioned on a target argument

Figure 1: An event mention example from ACE. An ACE Conflict.Attack event with _pummled_ as trigger word and three event arguments: _coalition_ (Attacker), _jets_ (Instrument) and _hills_ (Place).



role and 2) a QA model for answering the context-aware question to extract the event argument. We also design dynamic templates to generate the gold context-aware questions for QG model training.

To the best of our knowledge, this is the first QA-based EE work that utilizes dynamic templates and focuses on generating context-aware questions. Li et al. (2020) also propose a model to generate questions that incorporate contextual information for both event trigger and arguments. However, our work has two main advantages. First, in Li et al. (2020) the question only incorporates the contextual information at the ontology level (e.g. argument role, event type). In our work, the generated questions incorporate contextual information at an event mention-level. For example, the question generated by our model includes the real event argument rather than just the argument role (e.g. 'hills' vs 'Place'). Second, the questions in their work are generated by filling in the templates, but our templates are dynamic and used to train the QG model which can automatically generate the optimal question given a specific event mention and the concerned argument role.

Experimental results show that QGA-EE outperforms all of the single-task-based models on the Automatic Content Extraction (ACE) 2005 English dataset Doddington et al. (2004) and even achieves competitive performance with state-of-the-art joint IE models.

## 2 Model

Figure 1 shows the overall framework of QGA-EE. It focuses on Event Argument Extraction (EAE) only, but can be paired with any event trigger tagger to perform end-to-end EE. In Section 4, we pair it with a standard sequence labeling trigger tagger to evaluate its end-to-end EE performance.

### Question Generation Model

Previous QA-based EE work Du and Cardie (2020) fills in pre-designed templates with trigger information to generate the input questions to the QA model. However missing contextual information in the questions is a bottleneck for the performance of the QA model.

QGA-EE uses a QG model to generate context-aware questions conditioned on the input sentence and target role, which is based on a sequence-to-sequence architecture (e.g. BARTLewis et al. (2020), T5Raffel et al. (2020)). In order to train the QG model, we design **Dynamic Templates** for each role in the ACE ontology.2 We design multiple templates for each role, and each of them includes different combinations of other argument roles.

Footnote 2: [https://www.ldc.upenn.edu/sites/www.ldc](https://www.ldc.upenn.edu/sites/www.ldc).

Footnote 3: footnotemark:

For example, the Conflict.Attack event in ACE has four predefined argument roles: Attacker, Target, Instrument and Place.3 For the Attacker role, we exhaustively design eight templates using all of the possible combinations of the other roles included in the question (Table 1). When the model fills in the templates given a specific event mention, it is common that some of the predefined argument roles do not exist in the event mention. Thus the model only keeps the templates that contain the slots for argument roles appearing in the event mention. For the example in Figure 1, the Target role is not mentioned. So we ignore all of the templates that contain the [Target] slot, and we obtain four candidate questions for the Attacker role with corresponding arguments filled in: (1)_Who was the attacking agent?_ (2) _Who used jets in the attack?_ (3) _Who made the attack in hills?_ (4) _Who used jets in the attack in hills?_

Footnote 3: footnotemark:

To train a QG model to generate the questions that cover as many contextual information as possible, we use the question that contains the most contextual arguments as the ground truth. For the example in Figure 1, we choose the question _'Who used jets in the attack in hills?'_, because it contains two arguments: _'jets'_ and _'hills'_, the other three candidate questions listed above contain one or zero arguments. If more than one candidate question contains the most contextual arguments, we then pick the first one. The input and output examples for the QG model are as follows:

\begin{table}
\begin{tabular}{l} \hline Who was the attacking agent? \\ Who attacked [Target]? \\ Who used [Instrument] in the attack? \\ Who made the attack in [Place]? \\ Who attacked [Target] using [Instrument]? \\ Who attacked [Target] in [Place]? \\ Who used [Instrument] in the attack in [Place]? \\ Who attacked [Target] using [Instrument] in [Place]? \\ \hline \end{tabular}
\end{table}
Table 1: Dynamic templates for Attacker role in Conflict.Attack event with different combinations of known argument roles based on ACE ontology.

 

* role: attacker context: That's because coalition fighter jets
* this Iraqi position on the hills above Chamchamal and Iraqi troops made a hasty retreat.

**Output:** Who used jets in the attack in hills?

### Question Answering Model

Different from prior QA-based EE work that adapt an encoder-only architecture and predict the offsets of the event arguments Chen et al. (2019); Du and Cardie (2020); Li et al. (2020); Liu et al. (2020); Feng et al. (2020); Lyu et al. (2021); Zhou et al. (2021), our QA model is based on a sequence-to-sequence architecture (e.g. BART, T5), and generates the answer string directly. This enables prediction of multiple event arguments that are associated with the same role. Li et al. (2021) also adapts a generation model, but the input template is fixed. The examples of input and output are as follows:

```
Input:question:Who was harmed in * injured * event?context:Injured Russian diplomats and a convoy of America's Kurdish comrades in arms were among unintended victims caught in crossfire and friendly fire Sunday. Output:diplomats; convoy;victims \(<\)/s \(>\)
```

Post-processingWe split the output into a list of candidates (by ';'), and retrieve the arguments with offsets by exactly matching against the original sentence. We dynamically change the start position for searching to preserve the order of the retrieved event arguments. If an argument candidate cannot be matched with the original sentence, we discard it. Unlike the QG model, we use all of the possible questions as inputs during training for data augmentation purposes, and the size of the training data increases from 15,426 to 20,681. But in the testing phase, we use the single question generated by the QG model for each argument role.

## 3 Experimental Setup

### Dataset and Evaluation Metrics

We conduct the experiments on the ACE 2005 English corpora, which has 33 event types and 22 argument roles. It contains 599 documents collected from newswire, weblogs, broadcast conversations, and broadcast news. More specifically, we follow the pre-processing steps in Wadden et al. (2019),4 and evaluate our models on the resulting ACE05-E dataset.

Footnote 4: [https://github.com/dwadden/dygiepp](https://github.com/dwadden/dygiepp)

For evaluation, we use the same criteria as prior work Li et al. (2013): An **event trigger** is correctly identified if its offsets exactly match a reference. It is correctly classified if both its offsets and event type match a reference. An **event argument** is correctly identified (Arg-I) if its offsets and event type match a reference in the ground truth. It is correctly classified (Arg-C) if all of its offsets, event type, and argument role match a reference.

### Compared Baselines

**Model Variants.** To evaluate the generalizability of our approach, we evaluate two QGA-EE variants: **QGA-EE\({}_{BART}\)** and **QGA-EE\({}_{T5}\)**, which use BART and T5 as backbones respectively.

We compare the proposed models against SOTA EE models. **BERT QA**Du and Cardie (2020) use BERT as the encoder and predict the positions of the argument directly with role-driven questions. **TANL**Paolini et al. (2021) transfers input sentences into augmented natural language sentences for structured prediction. **TEXT2EVENT**Lu et al. (2021) is a sequence-to-structure network for event extraction.5 Ma et al. (2020) utilizes dependency parses as additional features. **BART-Gen**Li et al. (2021) is a BART-based generation model proposed for document-level event extraction.

Footnote 5: DEGREE Hsu et al. (2022) is not included because it is not evaluated on all of the argument roles.

We also compare with joint IE models trained on all of the ACE annotations which include entities, relations, and events. They benefit from additional information from other tasks and usually achieve better performance than the models trained on a single task. It is not fair to directly compare our model with the joint models since they incorporate more information beyond the standard EE training sets, but we still list their scores as a reference. **DYGIE++**Wadden et al. (2019) is a BERT-based model that models span representations with within-sentence and cross-sentence context. **ONEIE**Lin et al. (2020) leverages global features. **FourIE**Nguyen et al. (2021) and **GraphIE**Van Nguyen et al. (2022) are Graph Convolutional Networks-based models and **AMRIF**Zhang and Ji (2021) utilizes AMR Banarescu et al. (2013) parser.

### Implementation Details

We conduct all of the experiments on a single V100 GPU. For finetuning, we use the Adafactor Shazeer and Stern (2018) optimizer with a 

learning rate of \(1*10^{-4}\), weight decay of \(1*10^{-5}\), and clip threshold of \(1.0\). We train the model for 20 epochs. Further details such as hyperparameters and data statics for model training and evaluation are in Appendix C.

## 4 Results

### Event Argument Extraction Performance

Table 2 shows the performance of QGA-EE models on ACE05-E test set with gold triggers.6 Both QGA-EE variants outperform all other approaches, and using T5 as backbone provides an improvement of 2.5% over BART. The improvement over the prior QA-based models BERT_QA shows that generation-based QA models are more effective than position-based QA models for EE. QGA-EE\({}_{BART}\) outperforms the BART-based baseline BART-Gen and QGA-EE\({}_{75}\) outperforms the T5-based baseline TANL, which demonstrates the effectiveness of our models with different backbones. Our models even outperform the joint IE models DYGIE++ and ONEIE, which leverage additional information from entities and relations.

Footnote 6: Performance of FourIE, AMR-IÉ and GraphIE in gold triggers are not available in their original papers.

### Event Extraction Performance

We also evaluate our models on ACE05-E in a more "real world" fashion with _predicted_ triggers extracted by an ALBERT-based Lan et al. (2019) sequence-labeling model (Table 3).7 Similar to the performance on gold triggers, QGA-EE benefits more from the T5 backbone on predicted triggers. Both QGA-EE variants outperform all the EE-task-centered baselines by more than 1% on Arg-C.

Footnote 7: The model is trained on ACE05-E and the F1 score on test set is 72.96%. More details in Appendix.

We also include the scores from SOTA joint IE models, DYGIE++, ONEIE, FourIE, AMR-IE and GraphIE, as reference. But, as stated earlier, it is not fair to compare our models directly with them, as they benefit from being trained with all of the annotations from entities, relations, and events. Also it should be noted that their trigger labeling models have more complicated architectures and thus perform significantly better than the sequence-labeling based tagger we use (F1 75.4% from FourIE and F1 74.7% from OneIE). This further boosts the end-to-end EE performance.

### Ablation Study

Table 4 shows the ablation study of the QGA-EE\({}_{T5}\) model on the ACE05 test set with gold triggers. By replacing the QG model with simple context-unaware templates, the F1 score decreases by \(1.65\%\). It demonstrates that the context-aware questions generated by our QG component enhance the end-to-end event argument extraction performance. Additionally, the generation-based QA model deals with multi-argument situations better and provides an improvement of \(4.24\%\).

### Impact of Data Augmentation

As we mentioned in Section 2.2, the size of the training data increases from 15,426 to 20,681 as a benefit of our proposed dynamic templates. To eval

\begin{table}
\begin{tabular}{l|c c} \hline  & Arg-I & Arg-C \\ \hline QGA-EE\({}_{75}\) & 75.04 & 72.78 \\ \hline “w/o pretrained QG & 73.57 & 71.13 \\ \hline - w/o pretrained QG \& mutli-arg support & 69.61 & 66.89 \\ \hline \end{tabular}
\end{table}
Table 4: Ablation study with gold triggers on ACE05-E test set (F1, %).

\begin{table}
\begin{tabular}{l|c|c} \hline  & Arg-I & Arg-C \\ \hline BERT\_QA Du and Cardie (2020) & 68.2 & 65.4 \\ TANL\({}^{+}\) Paolini et al. (2021) & 65.9 & 61.0 \\ Ma et al. (2020) & - & 62.1 \\ BART-Gen Li et al. (2021) & 69.9 & 66.7 \\ \hline DYGIE++\({}^{+}\)Wadden et al. (2019) & 66.2 & 60.7 \\ ONEIE\({}^{++}\)Lin et al. (2020) & 73.2 & 69.3 \\ \hline QGA-EE\({}_{BART}\) (ours) & 72.4 & 70.3 \\ QGA-EE\({}_{75}\) (ours) & **75.0** & **72.8** \\ \hline \end{tabular}
\end{table}
Table 2: Event Extraction Results on ACE05-E test data (F1, %) with gold triggers. \({}^{*}\) models are trained with additional entity and relation data. \({}^{+}\) numbers are reported from Hsu et al. (2022), and others are from the original papers.

\begin{table}
\begin{tabular}{l|c|c} \hline  & Arg-I & Arg-C \\ \hline BERT\_QA Du and Cardie (2020) & 54.1 & 53.1 \\ TANL\({}^{[n]}\) & 50.1 & 47.6 \\ TEXT2EVENT Lu et al. (2021) & - & 53.8 \\ Ma et al. (2020) & 56.7 & 54.3 \\ BART-Gen Li et al. (2021) & - & 53.7 \\ \hline DYGIE++\({}^{+}\)Wadden et al. (2019) & 54.1 & 51.4 \\ ONEIE\({}^{+}\) Lin et al. (2020) & 59.2 & 56.8 \\ FourIE\({}^{*}\) Nguyen et al. (2021) & 60.7 & 58.0 \\ AMR-IE\({}^{*}\)Zhang and Ji (2021) & **60.9** & 58.6 \\ GraphIE\({}^{*}\)Van Nguyen et al. (2022) & **-** & **59.4** \\ \hline QGA-EE\({}_{BART}\) (ours) & 57.1 & 55.6 \\ QGA-EE\({}_{75}\) (ours) & **59.8** & 57.9 \\ \hline \end{tabular}
\end{table}
Table 3: Event Extraction Results on ACE05-E test data (F1, %) with predicted triggers. \({}^{*}\) models are trained with additional entity and relation data. All numbers of baselines are reported from the original papers.

 

uate the contribution of the data augmentation, we evaluate the performance of QGA-EE on ACE05 test data with partial training data (with gold triggers). With 40% of the training examples after data augmentation (8,272), QGA-EE achieves a F1 score of 71.42% on ACE05-E test set with gold triggers. It outperforms all of the baselines in Table 2, which demonstrates the effectiveness of our proposed model.

### Analysis and Discussion

The average length of the questions generated by QGA-EE\({}_{T5}\) is 10.5 tokens, compared with 6.7 in Du and Cardie (2020). They contain more context. For example, QGA-EE generates '_Who was attacked by mob in state?_' for the Target role in '_At least three members of a family in Indias underneath state of Tripura were **[hacked\({}_{Conflict.Attack}\)]** to death by a tribal mob for allegedly practicing witchcraft, police said Thursday._' It incorporates Attacker ('mob') and Place ('state') information.

We categorize the errors into four groups:

1. Bad question generated by the QG model. For example, QGA-EE generates '_What did state buy in * sell * event?_' for the Artifact role in '... that the Stalinist state had developed nuclear weapons and hinted it may sell or use them, depending on US actions._'. It should have been '_What did state sell in * sell * event?_' and this introduces an error to the QA model.
2. Errors resulting from a mismatch of the QA output result. QGA-EE may retrieve wrong offsets if a target candidate matches with multiple text strings in the original sentence. For example, QGA-EE matches the candidate '_Welch_' with the first mention in '_He also wants to subpoena all documents maintained in Jane Beasley Welch's personnel file by Shearman; Sterling, a prestigious corporate law firm where she worked before she **[married\({}_{Life.Marryl}\) Welch.'_, where the correct one is the second mention.
3. Errors resulting from missing entity conference. For example, QGA-EE identifies '_Jacques Chirac_' as the Entity of the Contact.Phone-Write event in '_French President Jacques Chirac received only a reserved response when he tried to mend fences by placing a telephone call Tuesday to Bush._' But '_he_' is the ground truth and refers to '_Jacques Chirac_'.
4. Predictions not explicitly mentioned. For example, in '_Kelly, the US assistant secretary for East Asia and Pacific Affairs, arrived in Seoul from Beijing Friday to brief Yoon, the foreign minister._', QGA-EE infers '_Seoul_' as thePlace of the Contact.Meet event, but it is not explicitly mentioned in the context, thus not covered by the gold annotations.

We manually analyzed a subset of the errors from the test set (50 examples), and show the portion of each category of error in Figure 2.

## 5 Conclusion

In this paper, we present QGA-EE, a novel sequence-to-sequence based framework for EE, which utilizes a QG model to generate context-aware questions as inputs to a QA model for EAE. Our model naturally supports the cases in which multiple event arguments play the same role within a specific event mention. We conduct experiments on the ACE05-E dataset and the proposed model outperforms all of the single-task-based models and achieves competitive results with state-of-the-art joint IE models. In the future, we plan to utilize the extensibility of the QA framework to incorporate knowledge from semi-structured event-relevant data such as Wikipedia Infoboxes. We also plan to extend our approach to multilingual EE and joint IE.

Figure 2: The portion of each category of error based on 50 error examples in test set.

 



### Limitations

The design of the dynamic templates requires knowledge of the event ontology and is time-consuming. The authors of the paper spent 30 hours designing the exclusive templates that cover all of the possible argument combinations for each argument role in ACE ontology. With a more complicated ontology, a much larger amount of time is required.

Another limitation of our approach is the offset retrieval method. If one sentence contains multiple mentions of the same entities, or even multiple text strings that have the same spellings but refer to different entities, the QGA-EE model always retrieves the position where the mention appears for the first time in the sentence as the offset of the extracted target. It may be improved by asking the model to generate contextual text as a position reference.

## Acknowledgements

We thank our colleague Aoife Cahill and the anonymous reviewers for their constructive comments and suggestions.

## References

* L. Banarescu, C. Bonial, S. Cai, M. Georgescu, K. Griffitt, U. Hermjakob, K. Knight, P. Kehn, M. Palmer, and N. Schneider (2013)Abstract meaning representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pp. 178-186. Cited by: SS4.
* Y. Chen, T. Chen, S. Ebner, A. Steven White, and B. Van Durme (2019)Reading the manual: event extraction as definition comprehension. arXiv preprint arXiv:1912.01586. Cited by: SS4.
* tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04), Cited by: SS4.
* X. Du and C. Cardie (2020)Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS4.
* R. Feng, J. Yuan, and C. Zhang (2020)Probing and fine-tuning reading comprehension models for few-shot event extraction. arXiv preprint arXiv:2010.11325. Cited by: SS4.
* E. Filatova and V. Hatzivassiloglou (2004)Event-based extractive summarization. In Text Summarization Branches Out, Cited by: SS4.
* I. Hsu, K. Huang, E. Boschee, S. Miller, P. Natarajan, K. Chang, and N. Peng (2022)DEGREE: a data-efficient generation-based event extraction model. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Cited by: SS4.
* H. Ji and R. Grishman (2011)Knowledge base population: successful approaches and challenges. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Cited by: SS4.
* Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut (2019)Albert: a lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. Cited by: SS4.
* M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2020)BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Cited by: SS4.
* F. Li, W. Peng, Y. Chen, Q. Wang, L. Pan, Y. Lyu, and Y. Zhu (2020)Event extraction as multi-turn question answering. In Findings of the Association for Computational Linguistics: EMNLP 2020, Cited by: SS4.
* Qi Li, H. Ji, and L. Huang (2013)Joint event extraction via structured prediction with global features. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Cited by: SS4.
* S. Li, H. Ji, and J. Han (2021)Document-level event argument extraction by conditional generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Cited by: SS4.
* Chin-Yew Lin (2004)Rouge: a package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81. Cited by: SS4.
* Y. Lin, H. Ji, F. Huang, and L. Wu (2020)A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Cited by: SS4.
* J. Liu, Y. Chen, K. Liu, W. Bi, and X. Liu (2020)Event extraction as machine reading comprehension. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),

Di Lu, Clare Voss, Fangbo Tao, Xiang Ren, Rachel Guan, Rostyslav Korolov, Tongtao Zhang, Dongang Wang, Hongzhi Li, Taylor Cassidy, Heng Ji, Shih-fu Chang, Jiawei Han, William Wallace, James Hendler, Mei Si, and Lance Kaplan. 2016. Cross-media event extraction and recommendation. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations_.
* Lu et al. (2021) Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. 2021. Text2Event: Controllable sequence-to-structure generation for end-to-end event extraction. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_.
* Lyu et al. (2021) Qing Lyu, Hongming Zhang, Elior Sulem, and Dan Roth. 2021. Zero-shot event extraction via transfer learning: Challenges and insights. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_.
* Ma et al. (2020) Jie Ma, Shuai Wang, Rishita Anubhai, Miguel Ballesteros, and Yaser Al-Onaizan. 2020. Resource-enhanced neural model for event argument extraction. In _Findings of the Association for Computational Linguistics: EMNLP 2020_.
* Van Nguyen et al. (2021) Minh Van Nguyen, Viet Dac Lai, and Thien Huu Nguyen. 2021. Cross-task instance representation interactions and label dependencies for joint information extraction with graph convolutional networks. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
* Nguyen et al. (2016) Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016. Joint event extraction via recurrent neural networks. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
* Paolini et al. (2021) Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages. In _9th International Conference on Learning Representations, ICLR 2021_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67.
* Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_.
* Shazeer et al. (2018) Lei Sha, Feng Qian, Baobao Chang, and Zhifang Sui. 2018. Jointly extracting event triggers and arguments by dependency-bridge rnn and tensor-based argument interaction. In _Proceedings of the AAAI Conference on Artificial Intelligence_.
* Shazeer and Stern (2018) Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In _International Conference on Machine Learning_, pages 4596-4604. PMLR.
* Nguyen et al. (2022) Minh Van Nguyen, Bonan Min, Franck Dernoncourt, and Thien Nguyen. 2022. Joint extraction of entities, relations, and events via modeling inter-instance and inter-label dependencies. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4363-4374.
* Wadden et al. (2019) David Wadden, Ulme Winnberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pieric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Pu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_.
* Zhang and Ji (2021) Zixuan Zhang and Heng Ji. 2021. Abstract meaning representation guided graph encoding and decoding for joint information extraction. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 39-49.
* Zhou et al. (2021) Yang Zhou, Yubo Chen, Jun Zhao, Yin Wu, Jiexin Xu, and Jinlong Li. 2021. What the role is vs. what plays the role: Semi-supervised event argument extraction via dual question answering. In _Proceedings of the AAAI conference on artificial intelligence_, volume 35, pages 14638-14646.

## Appendix A ACE05-E Data Preprocessing

We follow the preprocessing steps in Wadden et al. (2019) to preprocess ACE2005 corpora. More specifically, we use the preprocessing script at [https://github.com/dwadden/dygiepp](https://github.com/dwadden/dygiepp). In addition, we retrieve the character positions of the event triggers and arguments, because T5 uses a 

SentencePiece tokenizer. Table 6 shows the statistics of the ACE05-E dataset.

## Appendix B Complete Dynamic Templates for ACE ontology

Table 12 shows the complete list of templates with different combinations of known argument roles for each ACE event argument role.

## Appendix C Implementation Details

We use Huggingface Transformers library Wolf et al. (2020) to load the model checkpoints.

### Event Trigger Labeling Model

We implemented an ALBERT-based sequence labeling model for event trigger detection. We simply apply Softmax on top of the ALBERT encoder to predict the BIO schema based event label. We finetune the alert-xxlarge-v2 checkpoint provided by Huggingface during training. 8. We use the Adam optimizer with clip threshold of 1.0 and warmup proportion of 0.1. Table 7 shows the hyperparameter to train the Event Trigger Labeling Model.

Footnote 8: [https://huggingface.co/albert-xLarge-v2](https://huggingface.co/albert-xLarge-v2)

### QG model

When generating the groundtruth for QG model training, we use the basic template (e.g. 'Who was the attacking agent?') without incorporating any arguments if the target event role does not exist in the event mention. And we do not restrict the QG model to generate verbs that only appear in the templates. They are preserved for training the QA model.

We finetune the T5-large checkpoint provided by Huggingface during training. 9 with the Adafactor optimizer with clip threshold of 1.0 and warmup proportion of 0.1. Table 8 shows the hyperparameter to train the QG Model. And Table 9 shows the numbers of examples used to train and evaluate the QG model.

Footnote 9: [https://huggingface.co/t5-large](https://huggingface.co/t5-large)

### QA model

For the QA model training, we use the Adafactor optimizer with a learning rate of 2e-4, and weight decay of 1e-5, and clip threshold as 1.0. We set all of the relative_step, scale_parameter, and warmup_init parameters to False. For optimizer scheduler, we set the warmup proportion to 0.1.

If there are no event arguments for the argument role, the output is empty, as the following example. We include them to train the QA model. Table 9

\begin{table}
\begin{tabular}{c|c|c|c} \hline Split & \#Sents & \#Events & \#Arguments \\ \hline Train & 17,172 & 4,202 & 4,859 \\ Dev & 923 & 450 & 605 \\ Test & 832 & 403 & 576 \\ \hline \end{tabular}
\end{table}
Table 6: Data statistics of the ACE05-E dataset.

\begin{table}
\begin{tabular}{c c} \hline Hyperparamter & Value \\ \hline Learning rate & 3e-5 \\ Learning rate decay & 1e-5 \\ Epoch & 20 \\ Batch size & 4 \\ Gradient accumulation steps & 4 \\ \hline \end{tabular}
\end{table}
Table 7: Hyperparameter for Event Trigger Labeling Model training.

\begin{table}
\begin{tabular}{c|c|c} \hline Hyperparamter & Value \\ \hline Learning rate & 2e-4 \\ Learning rate decay & 1e-5 \\ Epoch & 20 \\ Batch size & 2 \\ Number of beam & 4 \\ Length penalty & 0.0 \\ \hline \end{tabular}
\end{table}
Table 10: Hyperparameter for QA Model training.

\begin{table}
\begin{tabular}{c|c|c} \hline Hyperparamter & Value \\ \hline Learning rate & 1e-4 \\ Learning rate decay & 1e-5 \\ Batch size & 2 \\ Batch size & 2 \\ Batch size & 2 \\ Batch size & 2 \\ Gradient accumulation steps & 32 \\ Number of beam & 4 \\ Length penalty & 0.0 \\ \hline \end{tabular}
\end{table}
Table 8: Hyperparameter for QG Model training.

\begin{table}
\begin{tabular}{l l|l|l} \hline  & Train & Dev & Test \\ \hline QG Model & 15,785 & 1,767 & 1,434 \\ QA Model & 20,681 & 1,713 & 1,391 \\ \hline \end{tabular}
\end{table}
Table 9: Number of examples used to train and evaluate the QG and QA models.

 

shows the numbers of examples used to train and evaluate the QA model.

* event? context: Injured Russian diplomats and a convoy of America's Kurdish comrades in arms were among unintended victims caught in crossfire and friendly fire Sunday. Output: \(<\) /s \(>\)**

In postprocessing, we dynamically change the start position for searching to keep the order of the retrieved event arguments.

## Appendix D Experiment Details

For all of the scores reported in the paper, the numbers are based on a single run with a fixed random seed 42.

### Event Trigger Labeling Model

Table 11 shows the performance of the Event Trigger Labeling model on ACE05-E test set.

### QG Model

We use Rouge Lin (2004) score (ROUGE-1) as the evaluation metric for QG model training, and the score on ACE05-E test set is 0.892.

\begin{table}
\begin{tabular}{c c|c c|c c} \hline \multicolumn{3}{l|}{Trigger Identification} & \multicolumn{3}{l}{Trigger Classification} \\ \hline P & R & F1 & P & R & F1 \\ \hline
72.52 & 79.9 & 76.03 & 69.59 & 76.67 & 72.96 \\ \hline \end{tabular}
\end{table}
Table 11: Performance of our event trigger labeling model on ACE05-E test data (%).

 

* [10] M. C. Dickson, J. R. Friedman, and A. K. S. Russell, "The automated computation of the number of events in a single event using the data set", _Journal of Statistical Planning Research_**26**(4):357-389, 2000.

[MISSING_PAGE_POST]

 

* [10] M. C. Bock, J. R. K. McCulloch, and D. A. Schultz, "A new approach to the design of a distributed system for the problem of linear programming," in _Proceedings of the 27th International Conference on Machine Learning Research_, pages 35-46. ACM, 2009.

[MISSING_PAGE_POST]

 

* [10] M. C. Dickson, _The Theory of Computer Science_, vol. 249, no. 3, pp. 57-68, 1996.
* [11] J. R. Friedman and A. G. Krizhevsky, "A Course on Machine Learning for Machine Learning", (Cambridge University Press), 2000.
* [12] P. F. Berman, "Learning the complexity of machine learning algorithms", in _Proceedings of the 1st International Conference on Machine Learning_, volume 1, pages 1-10, 2006. ACM, 2006.
* [13] S. H. Lee, "An Introduction to Machine Learning", (Cambridge University Press), 2006.
* [14] T. L. Jordan, "On Machine Learning: An Overview of Machine Learning", (Cambridge University Press), 2006.
* [15] W. J. Freeman, "Finding the number of words in a database", (Cambridge University Press), 2006.
* [16] W. J. Freeman, "Classification theory", Cambridge university press, 2002.

[MISSING_PAGE_POST]

 



[MISSING_PAGE_POST]

 

* [10] M. C. Collins, "The New Event: A Modern Perspective for the LHC Run II", _JHEP_**07** (2008) 043, doi:10.1007/JHEP07(2008)043, arXiv:0805.0691.
* [11] J. L. Hewett and R. G. Roberts, "A Brief Introduction to the Standard Model Higgs Cross Sections", _Phys. Rev. D_**58** (1998) 074002, doi:10.1103/PhysRevD.58.074002, arXiv:hep-ph/9703231.
* [12] S. Weinberg, "The Theory of Leptons", _Nucl. Phys. B_**108** (1974) 1, doi:10.1016/S0550-3213(74)90045-X.
* [13] H. Georgi and P. Lepage, "Broken Symmetries in Gauge Theories", _Phys. Lett. B_**100** (1975) 1, doi:10.1016/0370-2693(75)90045-X.
* [14] T. Hanada and K. Yamagata, "Chiral Symmetry Breaking at the Large Hadron Collider", _Phys. Rev. D_**58** (1998) 064004, doi:10.1103/PhysRevD.58.064004, arXiv:hep-th/9703015.
* [15] T. Hanada and K. Yamagata, "Chiral Symmetry Breaking at the Large Hadron Collider", _Phys. Rev. D_**59** (1999) 064004, doi:10.1103/PhysRevD.59.064004, arXiv:hep-ph/9907301.
* [16] T. Hanada, K. Yamagata, and K. Yamagata, "Chiral Symmetry Breaking at the Large Hadron Collider", _Phys. Rev. D_**60** (1999) 064004, doi:10.1103/PhysRevD.60.064004, arXiv:hep-ph/9907301.
* [17] T. Hanada, K. Yamagata, and K. Yamagata, "Chiral Symmetry Breaking at the Large Hadron Collider", _Phys. Rev. D_**61** (2000) 064004, doi:10.1103/PhysRevD.61.064004, arXiv:hep-ph/0007181.
* [18] T. Hanada, K. Yamagata, and K. Yamagata, "Chiral Symmetry Breaking at the Large Hadron Collider", _Phys. Rev. D_**62** (2000) 064004, doi:10.1103/PhysRevD.62.064004, arXiv:hep-ph/0007181.
* [19] T. Hanada, K. Yamagata, and K. Yamagata, "Chiral Symmetry Breaking at the Large Hadron Collider", _Phys. Rev. D_**63** (2001) 064004, doi:10.1103/PhysRevD.63.064004, arXiv:hep-ph/0007181.
* [20] T. Hanada, K. Yamagata, and K. Yamagata, "Chiral Symmetry Breaking at the Large Hadron Collider", _Phys. Rev. D_**64** (2001) 064004, doi:10.1103/PhysRevD.64.064004, arXiv:hep-ph/0007181.
* [21] T. Hanada, K. Yamagata, and K. Yamagata, "Chiral Symmetry Breaking at the Large Hadron Collider", _Phys. Rev. D_**65** (2002) 064004, doi:10.1103/PhysRevD.65.064004, arXiv:hep-ph/0107181.
* [22] T. Hanada, K. Yamagata, and K. Yamagata, "Chiral Symmetry Breaking at the Large Hadron Collider", _Phys. Rev. D_**66** (2002) 064004, doi:10.1103/PhysRevD.66.064004, arXiv:hep-ph/0107181.
* [23] T. Hanada, K. Yamagata, and K. Yamagata, "Chiral Symmetry Breaking at the Large Hadron Collider", _Phys. Rev. D_**66** (2002) 064004, doi:10.1103/PhysRevD.66.064004, arXiv:hep-ph/0207181.
* [24] T. Hanada, K. Yamagata, and K. Yamagata, "Chiral Symmetry Breaking at the Large Hadron Collider", _Phys. Rev. D_**66** (2002) 064004, doi:10.1103/PhysRevD.66.064004, arXiv:hep-ph/0207181.
* [25] T. Hanada, K. Yamagata, and K. Yamagata, "Chiral Symmetry Breaking at the Large Hadron Collider", _Phys. Rev. D_**66** (2002) 064004, doi:10.1103/PhysRevD.66.064004, arXiv:hep-ph/0207181.
* [26] T. Hanada, K. Yamagata, and K. Yamagata, "Chiral Symmetry Breaking at the Large Hadron Collider", _Phys. Rev. D_**66** (2002) 064004, doi:10.1103/PhysRevD.66.064004, arXiv:hep-ph/0207181.
* [27] T. Hanada, K. Yamagata, and K. Yamagata, "Chiral Symmetry Breaking at the Large Hadron Collider", _Phys. Rev. D_**66** (2002) 064004, doi:10.1103/PhysRevD.66.064004, arXiv:hep-ph/0207181.

 

* [10] M. C. Bender, R. J. Baxter, and D. A. Kessler, "Theoretical Results of the Theory of Computation", _Journal of Computational Physics_**267**(3):459-483, 1996.
* [11] G. E. Hinton, "A Practical Guide to Machine Learning", _IEEE Transactions on Neural Networks_**5**(1):105-118, 1997.
* [12] H. S. Tehrani, "An Introduction to Machine Learning", _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 1-10, 2006.
* [13] T. F. Hinton, "Deep learning for image recognition", _arXiv preprint arXiv:1407.6098_ (2014).
* [14] P. I. Goodfellow, "Learning from tiny images", _Advances in neural information processing systems_**5**(1):2373-2382, 2014.
* [15] L. Bottou, Y. Bengio, and V. LeCun, "Generative adversarial nets for large scale machine translation", _CoRR_**406**(4):1140-1157, 2015.
* [16] N. Srivastava, "Dropout normalization with deep convolutional networks", _Nature_**406**(7488):749-750, 2015.
* [17] O. Russakovsky, "Image classification by reducing dimensionality", _arXiv preprint arXiv:1412.03017_ (2014).

[MISSING_PAGE_POST]

 

* [10] M. C. Dickson, R. K. Sridharan, and A. J. Lathammer, "The automated computation of the number of events in a single event using the data set", _Journal of Statistical Analysis_**26**(4):357-389, 2000.

[MISSING_PAGE_POST]

 

* [164] M. C. Bader, R. J. Brocke, and D. K. S. Murray, "The role of the social network in a social network: A review on the nature of social networks", _Social Networks_**20**(3), pp. 57-89.
* [165] G. F. Hinton, "A survey on machine learning with applications to data mining", _Journal of Machine Learning Research_**10**(1), pp. 1-12.
* [166] H. P. Kingma and J. Ba, "Adam: A method for stochastic optimization", _arXiv preprint arXiv:1412.6150_**, 2014.
* [167] L. Breiman, "Learning from deep neural networks", _Advances in Neural Information Processing Systems_**10**(1), pp. 1-12.
* [168] T. Hastie, "An introduction to artificial intelligence", _IEEE Transactions on Knowledge Discovery and Data Engineering_**10**(1), pp. 1-12.
* [169] V. I. Goodfellow, "Image recognition by image recognition", _Neural information processing systems_**1**(1), pp. 1-12.
* [170] Y. Bengio, "Generators and representations", _Proceedings of the IEEE conference on computer vision and pattern recognition_, vol. 1, no. 1, pp. 1-10.
* [171] W. Teh, "Deep residual learning for large scale visual recognition", _CoRR_**2014**(4), pp. 1-12.
* [172] X. He, "Recursive representation learning for large scale visual recognition using convolutional neural networks", _CoRR_**2014**(4), pp. 1-12.

[MISSING_PAGE_POST]

 

* [10] M. C. Collins, J. L. Rosner, and A. Soni, "The anti-jet production of a jet in the final state at hadron colliders: The case for the \({}^{36}\)He and \({}^{42}\)He targets", Phys. Rev. D **89** (2014) 054017, doi:10.1103/PhysRevD.89.054017
* [11] G. Farrarini, R. Kalloshin, and P. Nason, "A new method to study the parton distribution function using shower Monte Carlo algorithms", arXiv:1207.0581v1 [hep-ph].
* [12] T. Sjostrand et al., "An introduction to PYTHIA 8.2", _Comput. Phys. Commun._ **191** (2015) 159, doi:10.1016/j.cpc.2015.01.024, arXiv:1410.5020.
* [13] CMS Collaboration, "Event generator tunes obtained from underlying event and multiparton scattering measurements", _Eur. Phys. J. C_ **76** (2016) 155, doi:10.1140/epjc/s10052-016-3988-x, arXiv:1512.0015.
* [14] GEANT4 Collaboration, "GEANT4--a simulation toolkit", _Nucl. Instrum. Meth. A_ **506** (2003) 250, doi:10.1016/S0168-9002(03)01368-8.
* [15] CMS Collaboration, "Measurement of differential cross sections with proton-proton collisions at \(\sqrt{s}=7\TeV\)", _JHEP_ **07** (2014) 040, doi:10.1007/JHEP07(2014)040, arXiv:1407.6097.
* [16] ATLAS Collaboration, "Measurements of inclusive dijet production cross sections in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Lett. B_ **716** (2015) 263, doi:10.1016/j.physletb.2015.01.026, arXiv:1502.02701.
* [17] ATLAS Collaboration, "Measurement of differential cross section in pp collision events at \(\sqrt{s}=7\TeV\)", _Phys. Lett. B_ **716** (2016) 263, doi:10.1016/j.physletb.2016.03.026, arXiv:1607.0581.
* [18] ATLAS Collaboration, "Measurement of differential cross sections in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Lett. B_ **716** (2016) 263, doi:10.1016/j.physletb.2016.03.026, arXiv:1607.0581.
* [19] ATLAS Collaboration, "Measurement of differential cross sections in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Lett. B_ **716** (2016) 263, doi:10.1016/j.physletb.2016.03.026, arXiv:1607.0581.
* [20] ATLAS Collaboration, "Measurement of differential cross sections in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Lett. B_ **716** (2016) 263, doi:10.1016/j.physletb.2016.03.026, arXiv:1607.0581.
* [21] ATLAS Collaboration, "Measurement of differential cross sections in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Lett. B_ **716** (2016) 263, doi:10.1016/j.physletb.2016.03.026, arXiv:1607.0581.
* [22] ATLAS Collaboration, "Measurement of differential cross sections in pp collisions at \(\sqrt{s}=7\TeV\)", _Phys. Lett. B_ **716** (2016) 263, doi:10.1016/j.physletb.2016.03.026, arXiv:1607.0581.

 

* [10] M. C. Dickson, J. R. Friedman, and A. K. S. Thorne, "Theoretical Computer Science: The Art of the Philosophy of 



\begin{tabular}{|l|p{142.3pt}|p{142.3pt}|}  & Plaintiff, Defendant & Where did [Plaintiff] see [Defendant]? \\ Place & Plaintiff, Adjudicator & Where did [Plaintiff] see defendant being adjudiacted by [Adjudicator]? \\  & Defendant, The Predantal, And Judicator & Where was [Defendant] used being adjudicated by Adjudicator [Adjucidator]? \\  & Plaintiff, Default, Defendant, and Adjudicator & Where did [Plaintiff] see [Defendant] being adjudicated by [Adjudicator]? \\  & Adjudicator & Judge is not subject to the same rule as in [Pace]? \\ \hline Justice. & - & Who convicted defendant for crime? \\ Convict & Addjuctor & When confronted with the same rule as in [Pace]? \\  & & Where did the connection take place? \\  & Defendant & Where was [Defendant] convicted for crime? \\ Place & Adjudicator & Where did [adjudicator] convict the defendant for crime? \\  & Defendant, And Judicator & Where did [Adjudicator] convict [Defendant] for crime? \\ \hline Defectandant & - & Who was sentenced for crime? \\  & Defendant & Who was sentenced by [Adjudicator] for crime? \\  & Defendant, Pace & Who was sentenced for crime in [Place]? \\  & Adjudicator, Place & Who was sentenced by [Adjudicator] for crime in [Place]? \\ Justice. & - & Who sentenced the defendant for crime? \\ Sentence & Adjudicator & Where id [Adjudicator] for crime? \\  & Palce & Who sentenced [Defendant] for crime in [Place]? \\  & Defendant, Pace & Who sentenced [Defendant] for crime in [Place]? \\  & Place & Where did the sentence take place? \\ Place & Adjudicator & Where was [Defendant] sentenced for crime? \\  & Place & Any did [Adjudicator] sentence the defendant for crime? \\  & Defendant, Pace & Where did [Adjudicator] sentence [Defendant] & Pace are different from [Pace]? \\  & Policy & Where did [Adjudicator] sentence the defendant for crime? \\  & Defendant, Pace & Where did [Adjudicator] sentence the defendant for crime? \\  & Adjudicator & After



\begin{table}
\begin{tabular}{|p{28.5pt}|l|p{143.9pt}|p{76.8pt}|} \hline  & - & Where was the person extradited to? \\ Justice. & Destination & Agent & Where did [Agent] extradite person to? \\ Justice. & Origin & Where was the person extruded to from [Origin]? \\ Extradite & Agent, Origin & Where did [Agent] extradite person to from [Origin]? \\ \hline  & - & Where was the person extradited from? \\ Origin & Agent & Where did [Agent] extradite person from? \\  & Destination & Where was the person extradited from to [Destination]? \\  & Agent, Destination & Where did [Agent] extradite person from to [Destination]? \\ \hline Justice. & Defendant & - Adjudicator & Who was acquired of crime by [Adjudicator]? \\ Acquit & Adjudicator & - Defendant & Who acquited the defendant of crime? \\ \hline  & - & Who was pardoned for crime? \\ Defendant & Adjudicator & Who was pardoned by [Adjudicator] for crime? \\ Defence & Place & Who was pardoned for crime in [Place]? \\ Adjudicator, Place & Who was pardoned by [Adjudicator] for crime in [Place]? \\ Justice. & - & Who pardoned defendant for crime? \\ Pardon & Adjudicator & Defendant & Who pardoned [Defendant] for crime? \\  & Place & Who pardoned defendant for crime in [Place]? \\  & Defendant, Place & Who pardoned [Defendant] for crime in [Place]? \\  & Place & Where did [Adjudicator] & Where was [Defendant] pardon the defendant for crime? \\ Place & Adjudicator & Where did [Adjudicator] & Before crime? \\  & Defendant, Adjudicator & Crime? \\ \hline  & - & Who made the appeal? \\ Defendant & Adjudicator & Who made the appeal to [Adjudicator]? \\ Defence & Place & Who made the appeal in [Place]? \\ Police & Adjudicator, Place & Who made the appeal to [Adjudicator] in [Place]? \\ Justice. & Adjudicator & The Defendant & Who adjudicated the appeal made by [Defendant]? \\ Appeal & Adjudicator & Place & Who adjudicated the appeal in [Place]? \\  & Defendant, Place & Who adjudicated the appeal made by [Defendant] \\  & Defendant, Place & Who adjudicated in [Place]? \\  & Passing, Discretension & Where did [Defendant] & After a change of variables is present, it is not possible that the original variable is present. \\ State. & Defendant & Where did [Defendant] & Once again, if there are no changes in variables present, it is not possible that the original variable is present. \\ State. & Defendant & Where did [Defendant] & Once again, if there are no changes in variables present, it is not possible that the original variable is present. \\ State. & Defendant & Where did [Defendant] & Once again, if there are no changes in variables present, it is not possible that the original variable is present. \\ State. & Defendant & Where did [Defendant] & Once again, if there are no changes in variables present, it is not possible that the original variable is present. \\ State. & Defendant & Where did [Defendant] & Once again, if there are no changes in variables present, it is not possible that the original variable is present. \\ State. & Defendant & Where did [Defendant] & Once again, if there are no changes in variables present, it is not possible that the original variable is present. \\ State. & Defendant & Where do not change in variables present, it is not possible that the original variable is present. \\ State. & Defendant & Where does not change in variables present, it is not possible that the original variable is present. \\ State. & Defendant & Where do not change in variables present, it is not possible that the original variable is present.





**ACL 2023 Responsible NLP Checklist+**

Footnote †: margin: The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

A For every submission: A1. Did you describe the limitations of your work? Section 4.5 and Limitation Section A2. Did you discuss any potential risks of your work? the work is for foundational research and experiments are conducted on public dataset A3. Do the abstract and introduction summarize the paper's main claims? Abstract and Section 1 A4. Have you used AI writing assistants when working on this paper? Left blank. B Did you use or create scientific artifacts? section 3.1, section 3.2, section 4.1, section 4.2 B1. Did you cite the creators of artifacts you used? section 3.1, section 3.2, section 4.1, section 4.2 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? ACE 2005 Corpus was released by Linguistic Data Consortium with license: LDC User Agreement for Non-Members The code of our work will be released on Github with license TBD. B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? ACE 2005 Corpus is for research purposes only Our work is for research purposes only_B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? _The task itself is designed for identifying person/organization involved in events. The dataset contains violent events such as attack, die, and injure._B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? section 3.1 and appendix B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. C Did you run computational experiments? section 4 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? section 3.3 and appendix

C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? _section 3 and appendix_ C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? _section 4 and appendix_ C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? _appendix_
* D **Did you use human annotators (e.g., crowdworkers) or research with human participants?** _Left blank._
* D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? _No response._
* D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? _No response._
* D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? _No response._
* D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? _No response._
* D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? _No response._