A New Task and Dataset on Detecting Attacks on Human Rights Defenders
Year: 2023

Abstract:
The ability to conduct retrospective analyses of attacks on human rights defenders over time and by location is important for humanitarian organizations to better understand historical or ongoing human rights violations and thus better manage the global impact of such events. We hypothesize that NLP can support such efforts by quickly processing large collections of news articles to detect and summarize the characteristics of attacks on human rights defenders. To that end, we propose a new dataset for detecting Attacks on Human Rights Defenders (HRDsAttack) consisting of crowdsourced annotations on 500 online news articles. The annotations include fine-grained information about the type and location of the attacks, as well as information about the victim(s). We demonstrate the usefulness of the dataset by using it to train and evaluate baseline models on several sub-tasks to predict the annotated characteristics.

heading: 1 Introduction
text: It is essential for human rights organizations to track, analyze and summarize attacks on human rights defenders over time and across locations for better personnel protection and situational analysis. To do so, multiple event attributes denoting different aspects of the attacking event need to be extracted from textual sources. However, this would be a time-consuming process if done manually. Figure 1 gives an example of the kinds of information that such organizations need to extract.
In order to train and evaluate an NLP model to extract this information automatically, a relevant dataset is necessary. The ideal dataset requires accurate annotations for both the breadth (the number of extracted event attributes) and depth (the levels of granularity for each event attribute) of the events. However, all existing Event Extraction (EE) datasets (e.g. ACE05 (Doddington et al., 2004), ERE (Song et al., 2015), ACE05-E (Wadden et al.,
Algerian journalist sentenced to six months’ jail for ‘defamation’ 2021-03-29 00:00:00
A court in Algeria's capital Monday sentenced journalist Abdelhakim Setouane to six months in jail for defamation of a former speaker of the lower house of parliament, his lawyer said.
Abdellah Heboul told AFP that the journalist, who has been detained since October 20, would be released on April 20, having spent six months behind bars.
Follow us on Facebook, Twitter and Instagram to stay connected
News Article
2019), ACE05-E+ (Lin et al., 2020)) do not contain annotations at a sufficiently fine-grained level. Although some existing ontologies and datasets do include annotations related to attacking events, e.g. the ATTACK event type in the ACE05 dataset along with the associated AGENT attribute, they are incomplete with respect to many of the details of interest to human rights organizations and do not contain annotations relevant to victim characteristics or the time/location of the attacking event. As a result, existing open-source EE models trained on these datasets (Honnibal et al., 2020; Wadden et al., 2019; He et al., 2019) are unable to predict the complete set of relevant information.
To mitigate the gap in existing resources, we present HRDsAttack, a new dataset containing
7089
crowdsourced annotations on 500 online news articles (including article title, article body text, and publication time). Each news article is annotated with 13 different event attributes to capture critical information about attacks on human rights defenders, including the type and location of the attacks, as well as information about the victim(s) and the perpetrator. With HRDsAttack, we hope to support more research opportunities for including NLP in applications related to human rights, as well as for broader AI for Social Good (AI4SG) efforts.
To summarize, our contributions are threefold:
1. We present a new dataset (HRDsAttack) that includes annotations for fine-grained event details on attacks on human rights defenders. By focusing on expanding the breadth and depth of the attacking event relative to existing EE ontologies, we aim to address the limited scope of existing NLP resources. The complete ontology for our dataset is shown in Table 1;
2. We propose a new NLP task to extract finegrained event details on attacks on human rights defenders.
3. We demonstrate the usefulness of HRDsAttack with a strong baseline model based on Question Answering (QA) using the T5 model (Raffel et al., 2020) as the backbone in a multi-task setting.
The HRDsAttack dataset along with the code for model training and evaluation is available at https://github.com/dataminr-ai/ HRDsAttack.

heading: 2 Related Work

heading: 2.1 Event Extraction
text: Event Extraction (EE) is an NLP task that aims to extract key information such as who, what, where, and when from a text. The most commonly used dataset for EE is the ACE05 English corpus (Doddington et al., 2004) which consists of 33 event types and 22 event argument roles across 599 documents from newswires, web blogs, and broadcast conversations. While the ACE ontology covers a large range of event types, only two of them are related to attacking events: the LIFE.INJURE event and the CONFLICT.ATTACK event. Some of the other datasets that focus on extracting event triggers or event arguments are based on the ACE05
ontology (Wadden et al., 2019; Lin et al., 2020), and only cover limited aspects of the information that HRDsAttack covers, e.g. the ATTACKER and TARGET attributes in the LIFE.INJURE and CONFLICT.ATTACK events. The Armed Conflict Location and Event Data (ACLED) dataset (Raleigh et al., 2010) covers political violence and protest events with annotations for event type, actors and targets, but it does not cover victim-dependent attributes. In comparison, HRDsAttack focuses on attacking events on human rights defenders and provides more event attributes for the attacks, along with more granular information regarding each event attribute.
In terms of modeling approaches, early work on EE formulated the task as a token-based classification problem which leveraged different types of features (Ahn, 2006; Liao and Grishman, 2010a,b; Li et al., 2013). More recent approaches focus on applying neural models to EE tasks, such as CNNs (Chen et al., 2015), RNNs (Liu et al., 2019), and other advanced model structures (Nguyen and Nguyen, 2019; Zhang et al., 2019).

heading: 2.2 NLP Research for Human Rights
text: Existing NLP research resources around event detection and extraction related to Human Rights are extremely limited. Previous work has focused on identifying potential human rights abuse incidents from social media posts (Alhelbawy et al., 2020; Pilankar et al., 2022), alongside more general applications such as detecting abusive language (Golbeck et al., 2017; Djuric et al., 2015; Aroyo et al., 2019), or procedure-focused applications (e.g. data modeling processes for human rights data (Miller et al., 2013; Fariss et al., 2015)), or predicting judicial decisions of the European Court of Human Rights using NLP (O’Sullivan and Beel, 2019). To our knowledge, there are no event extraction datasets which target human rights issues, which makes HRDsAttack a first in this research area.

heading: 3 Dataset
text: In this section, we describe the construction of the HRDsAttack dataset, which contains 500 annotated news articles, including article title, article body text, and publication time. We select news articles as the data source rather than other data sources (such as social media posts) since online news articles generally have higher accessibility, better trustworthiness of the source, and longer content
length. In our work, we sample online news articles from the GDELT database1, which we discuss in more detail in Section 3.2.

heading: 3.1 Annotation Labels
text: To ensure the comprehensiveness of the annotations regarding capturing event details, we first identify the event attributes or labels required for annotation. As shown in Table 1, according to the UN Human Rights SDG 16.10.1 Guidance Note2, we identify
1https://www.gdeltproject.org/ 2https://www.ohchr.org/Documents/
Issues/HRIndicators/SDG_Indicator_16_ 10_1_Guidance_Note.pdf
the following 5 categories of attributes: PERPETRATOR, VIOLATION, VICTIM, LOCATION, and TIME. Each category has one or more associated event attributes, all denoting key information about the primary event described in the original article3. If there are multiple events mentioned in the article, only the primary event (i.e. the event that happened closest to the publication time) is annotated. We also specify that the VICTIM category could have multiple entries per article, while other categories can only have one entry per article (i.e. only one entry for the primary attack event). The ontology
3All label values for each event attribute are prescribed by the SDG 16.10.1 Guidance Note.
for the annotation labels is shown in Table 1.

heading: 3.2 Data Sampling
text: To build HRDsAttack, we first scrape 80,112 online news articles in the time range of 2019/09/01 to 2022/05/01 from the GDELT database following the CAMEO codebook (Schrodt, 2012), a standard framework for coding event data (Yuan, 2016). These scraped news articles are identified as relevant to human rights defenders by an existing human rights monitoring workflow.
During our pilot studies, we identified a data imbalance issue from the annotations under random sampling. Specifically, we observed significantly skewed label distributions in event attributes VIOLATION TYPE and VICTIM TYPE, the minority classes being TORTURE and KIDNAPPING for VIOLATION TYPE, and HUMAN RIGHTS DEFENDERS and TRADE UNIONISTS for VICTIM TYPE. To address this issue, we apply keyword filtering and targeted sampling to ensure HRDsAttack is wellbalanced across classes in each event attribute.
To include more samples with a higher probability of containing events associated with these minority attributes, we first reduce the original 80,112 samples into four smaller, targeted sample sets. Each targeted sample set corresponds to the articles that contain the keyword for each of the minority classes. We then randomly sample 25 articles from each targeted sample set to form a batch of 100 samples for each round of full annotation. Table 2 shows the keywords used for minority class targeted sampling.

heading: 3.3 Annotation Process
text: The annotation is done by qualified workers (Turkers) on Amazon Mechanical Turk (AMT). We design and implement a separate qualification task to recruit top-performing Turkers, and we only release the full annotation tasks to the Turkers that surpass a predefined performance bar based on the qualification tasks.

heading: 3.3.1 Qualification Tasks
text: For the qualification task, all US-based Turkers that have a HIT (Human Intelligence Task 4) approval rate greater than 90% and a total number of HITs approved greater than 500 are able to participate. In the qualification task, we sample three different news articles and ask all participant Turkers to annotate every event attribute for each news article through three questionnaires (each HIT contains three questionnaires, one for each news article). We then evaluate their performance on this annotation task. All three news articles are also annotated by domain experts, and we use their annotations as the ground truth answers for calculating the Turker accuracy. We only recruit Turkers who have 75% or higher average accuracy across all three news articles. We launched three rounds of qualification tasks with 50 assignments in total, and ten Turkers passed the qualification tasks.
The instructions and the task interface for the qualification tasks are shown in Figures 4 to 11 in Appendix A.

heading: 3.3.2 Full tasks
text: In the full task, each HIT only contains a single news article. The instructions and the annotation interface are identical to the qualification task. We launched all 500 samples in 5 batches, each batch containing 100 HITs. During our pilot studies, we did not observe a significant quality improvement with replication factor 3 due to relatively high agreement scores between the Turkers (Table 8 in Appendix C). We hypothesize that this is because the annotation task itself is highly objective. Therefore, we did not apply replication factors during the full task.
We compensate each Turker with $7.50 per assignment in the qualification task (three news articles per assignment) and $2.00 per assignment in the full task (one news article per assignment). We also provide an additional bonus to all participant Turkers of $0.5 per assignment. The final pay rate is $15.00 per hour, which is over the US national minimal wage of $7.505.
The annotation instructions and the task interface for the full tasks are shown in Figures 12 to 15 in Appendix A.
4A HIT represents a single, self-contained, virtual task that a Turker can work on, submit an answer, and collect a reward for completing.
5https://www.dol.gov/general/topic/ wages/minimumwage

heading: 3.4 Data Statistics
text: To create a benchmark dataset from HRDsAttack, we randomly split the 500 annotated samples into train, dev, and test set with a 3:1:1 ratio. Table 3 shows the statistics of the splits. A breakdown of the label-level statistics for each event attribute can be found in Table 7 in Appendix B.

heading: 4 Our Model
text: With the construction of HRDsAttack, we now turn to developing a model for the task. We noted earlier that existing state-of-the-art EE models are not suitable as baselines, as they rely on extensive human annotations based on token-level annotations, hence cannot easily be re-trained and evaluated on this dataset. For instance, AMR-IE (Zhang and Ji, 2021) and GraphIE (Qian et al., 2018) are trained on the ACE05 dataset and ERE dataset. Some recent research casts the EE task as QA tasks or Seq2seq tasks, such as RCEE_ER (Liu et al., 2020) and Text2Event (Lu et al., 2021). In this section, we propose a new model for extracting fine-grained details regarding attacks on human rights defenders.

heading: 4.1 Overall Framework
text: Given the limited amount of training data and the range and variety of event attributes, we propose using a single Seq2Seq Question Answering (QA) model. Training a unified model has the advantageous property that it shares the training data across all the sub-tasks thus potentially leading to better performance for each sub-task. Figure 2 shows the overall framework of our proposed baseline model.
We formulate all of the subtasks as a generation task following T5 (Raffel et al., 2020), which proposes reframing all NLP tasks into a unified text-to-text format. The input to the T5 model is a natural language sentence composed of (1) a task prefix (e.g. ‘extract victims’), (2) an attributeoriented question (e.g. ‘Who is the victim of the
violation?’), and (3) a context which is the original article. The output is a text string which explicitly refers to the value of the concerned event attribute (e.g. ‘Abdelhakim Setouane’).

heading: 4.2 Input-Output Design
text: We group the event attributes into three categories: general article-dependent attributes, victim-dependent attributes, and publication timedependent attributes, and we design input and output formats for them respectively. For all of the three categories, the output is a text string that explicitly refers to the value of the relevant event attribute, e.g. ‘Yes’ for PERPETRATOR MENTION, or ‘state security forces’ for PERPETRATOR TYPE. The input formats for the three categories have minor differences 6:
• General Article-dependent Attributes: Most of the event attributes depend on the general information contained within the article (i.e. do not rely on additional input other than article’s body text). These include PERPETRATOR MENTION, PERPETRATOR TYPE, and VIOLATION TYPE. For these attributes, the input is the concatenation of a task prefix, an attribute-oriented question, and the original article (e.g. the top three examples in Figure 2).
• Victim-dependent Attributes: Some event attributes, such as VICTIM SEX TYPE, depend on the information related to a specific victim. Thus we incorporate the victim name into the input question, as exemplified in the fourth and fifth examples in Figure 2.
• Publication Time-dependent Attributes: In some cases, the YEAR, MONTH, and DAY attributes related to the attack event are not explicitly present in the article, and we need to infer them based on a combination of the article publication time and the relevant time mentioned in the article (e.g. last month, two weeks ago, yesterday). The article publication time is available as metadata in the GDELT dataset (e.g. 2021-03-29 00:00:00). For these attributes, we add publication time information into the input, as shown in the last example of Figure 2.
6The complete lists of input and output formats are provided in Table 9 in Appendix D.
Task Prefix. Following the multi-task setting in the original T5 work, we add a task prefix at the beginning of the input text. The task prefix is used to instruct the T5 model to perform a particular task. It could be any arbitrary text. In our work, we use a brief task description as the task prefix for each event attribute, e.g. ‘detect perpetrator’ for PERPETRATOR MENTION or ‘extract violation type’ for VIOLATION TYPE (Figure 2). The complete list of all the task prefixes is shown in Table 10 in Appendix D.

heading: 4.3 Long Document Resolution
text: The maximum input length allowed by the T5 model is 512 tokens, but around 75% of the articles from the GDELT dataset exceed that length limit. We explore two options to deal with articles with more than 512 tokens: Truncation and Knowledge Fusion. Additional methods for handling long documents are discussed in Appendix E.
Truncation. We only use the first 512 tokens of the input text. The articles from GDELT are news articles, and the first several sentences from a news article usually contain the most important information. Thus a simple solution is to truncate the article and ignore the cut content.
Knowledge Fusion. To mitigate the information loss in the Truncation method, we adopt a split-fuse approach (Figure 3) by (1) splitting the documents into short paragraphs using the spaCy (Honnibal et al., 2020) tokenizer7; (2) applying the model to each of the paragraphs; and then (3) merging the results from each paragraph to obtain the final results for the
7We use the en_core_web_sm spaCy pipeline.
original article. For event attributes that allow more than one value (e.g. VICTIM NAMES), we keep all of the unique results, and for other attributes, we only keep the one with the highest confidence score (beam search score).

heading: 5 Experiments

heading: 5.1 Evaluation Metrics
text: We consider the following metrics for evaluating different event attributes:
• Precision, Recall, and F1 Score: we use Precision, Recall, and F1 score to evaluate the
model performance on PERPETRATOR MENTION and VIOLATION TYPE.
• Accuracy: we use accuracy (i.e. percentage correct) to evaluate the model performance on PERPETRATOR TYPE, VICTIM TYPE, VICTIM SEX TYPE, VICTIM AGE GROUP, COUNTRY, REGION, CITY, YEAR, MONTH, and DATE.
• Fuzzy Match Precision, Recall, and F1 Score: For the VICTIM NAME attribute, we use precision, recall, and F1 score based on exact matching and fuzzy matching, respectively. For exact matching, one predicted victim name is counted as correct only if it exactly matches with a victim name in the ground truth. For fuzzy matching, one predicted victim name is counted as correct if it has overlapping tokens with a victim name in the ground truth. For example, a predicted victim name Jordan is counted as correct when it matches with a ground truth name Michael Jordan.

heading: 5.2 Baseline Models
text: We consider the following models in our evaluation:
• DyGIE++ (Wadden et al., 2019): a joint Information Extraction (IE) model and we use the checkpoint trained on the ACE05 dataset. It requires mapping from the ACE event ontology8 to HRDsAttack. As a result, it only covers two attributes: PERPETRATOR MENTION and VICTIM NAME as there is no available mapping for the other event attributes in HRDsAttack.
• T5 w/ Truncation: our proposed T5-based model with truncation.
• T5 w/ Knowledge Fusion: our proposed T5based model with knowledge fusion.
• Hybrid (final model): a hybrid model based on T5 w/ Truncation and T5 w/ Knowledge Fusion. The model only applies knowledge fusion to PERPETRATOR MENTION, VICTIM NAME, and VICTIM AGE GROUP attributes. This hybrid strategy is decided based on the evaluation results on the dev set.
8The ACE ontology covers event types such as ATTACK and INJURE.
We recognize that it would be ideal to have more baseline models for comparison, such as a retrained version of DyGIE++ on HRDsAttack. However, many existing EE models are trained on token-level annotations and are not designed for the additional event attributes that HRDsAttack covers (e.g. VICTIM TYPES). Therefore, we had to design a specialized model for this task. We plan to benchmark more Sequence-to-Sequence based models on HRDsAttack in future work.

heading: 5.3 Training Implementation
text: We use the T5-large checkpoint 9 provided by Huggingface (Romero, 2021) to initialize the model and all experiments are run on a single AWS g5.xlarge instance. The AWS g5.xlarge instance is equipped with a single NVIDIA A10G GPU with 24 GB of GPU memory. Table 4 shows the hyperparameters we use to train the model.

heading: 5.4 Overall Performance
text: Table 5 shows the performance of the four models on the test set: the DyGIE++ baseline, T5 w/ Truncation, T5 w/ Knowledge Fusion, and the Hybrid model. Both T5-based models significantly outperform the DyGIE++ baseline, except for the precision of PERPETRATOR MENTION. In addition, we get further improvement from the Knowledge Fusion method for the PERPETRATOR MENTION, VICTIM NAME, and YEAR attributes. For other attributes, we get results that are slightly worse than those without Knowledge Fusion. This aligns with our assumption that violation events may be elaborated in the later parts of the news articles with specific victim names and violation types. So by applying the Knowledge Fusion method, we can significantly improve the recall of some event attributes. But for other information such as violation time and location, they usually appear in the first several sentences of the news article. The
9https://huggingface.co/t5-large
time and location information appearing in the later parts may not be related to the primary attacking event. So based on the evaluation results on the dev set (Table 11 in Appendix F), we propose a hybrid model as our final baseline model. The hybrid model only applies Knowledge Fusion to PERPETRATOR MENTION, VICTIM NAME, and VICTIM AGE GROUP attributes. We notice that the hybrid model designed based on the dev set does not achieve the best performance for VICTIM AGE GROUP and YEAR attributes on the test set. It might be the fact that the hybrid strategy is overfitted on the dev set. And we leave the optimization of the hybrid model as future work.
While the hybrid model outperforms the DyGIE++ baseline in almost all of the event attributes and unlocks the extraction of new attributes, we do see a relatively lower model performance in attributes such as REGION and DAY. We hypothesize that the ambiguity in REGION labels and the large number of classes in DAY labels introduce additional challenges to the model, especially with a limited amount of training data. For instance, some annotators mistakenly put London under REGION instead of CITY. We acknowledge that the annotation instructions could be further improved to address this issue.
We also evaluate the end-to-end performance on the victim-dependent attributes with the modelpredicted victim names (Table 6). And we use F1 scores as the evaluation metric. One victimdependent attribute is counted as correct only when both the predicted victim name and the predicted attribute value match with the ground truth.

heading: 6 Conclusion
text: In this paper, we present a new dataset that supports extracting detailed information about attacks on human rights defenders under a new task setting. Compared with existing event extraction resources, we focus on the human rights domain and expand to more event attributes for capturing event details more comprehensively. Our new dataset (HRDsAt-
tack) contains 500 human-annotated news articles with 13 different event attributes regarding the victim(s), the type of perpetrator and violation(s), as well as the time and location of the attacks. We demonstrate the usefulness of the dataset by developing a Sequence-to-Sequence-based Question Answering model tailored for this task. While it achieves decent performance on some event attributes, there are many where there is much room for improvement. We view this model as a strong baseline for future work. We believe models trained with HRDsAttack could be generalized to detect attacking events in other domains or targeting a different population. And we hope that this work encourages additional research on the development of new AI4SG NLP resources in the future.

heading: Acknowledgements
text: We would like to thank Jessie End at Dataminr for her support during this project. We also want to thank all the reviewers for their valuable and constructive feedback during the review phase.
Limitations
While HRDsAttack is, to the best of our knowledge, the first dataset on extracting attacks on human rights defenders, there are some limitations. For one, while being the first corpus of its kind, our dataset is English-only. Second, the number of documents is limited. While the sample size of HRDsAttack (500) is on par with some of the other EE datasets, such as ACE05 (599), we do see more samples being beneficial to subsequent model training and supporting other future studies. In addition, despite the effort to balance the class labels in the event attributes, some of the labels still remain imbalanced, such as PERPETRATOR TYPE.
Ethics Statement
The construction of HRDsAttack involves human annotations on AMT. The Turkers are provided with clear annotation instructions and are informed of the conditions where they would be qualified or disqualified. We compensate the Turkers with a final paid rate of $15.00 per hour which is over the US national minimal wage of $7.50.

heading: A Annotation Interface
text: We list the screenshots of our annotation interface from Figure 4 through Figure 15, including the annotation guideline (instructions) page for both the qualification tasks and the full tasks, the example page, as well as the task pages.

heading: B Label Statistics
text: In Table 7, we list the statistics of all the labels in HRDsAttack, as well as their distributions in the train, dev, and test set.

heading: C Cohen-Kappa Scores
text: We calculate the average pair-wise Cohen-Kappa scores for each qualified Turker during our pilot study using 100 Hits with replication factor 3. While we did our due diligence to make our annotation instructions as comprehensive as possible, some of the concepts regarding Human Rights were sometimes challenging to distinguish for the Turkers. The relatively lower weighted average of Cohen-Kappa scores for some event attributes (PERPETRATOR MENTION: 0.40, PERPETRATOR TYPE: 0.41) are also potentially due to the imbalanced distributions of these attributes. The weighted averages of Cohen-Kappa scores for other attributes are all higher than 0.61 for violation and victim-related classes (VIOLATION TYPE: 0.67, VICTIM POPULATION TYPE: 0.64, VICTIM TYPE: 0.62), which are generally considered as substantially agree.
D Input-Output Design
Table 9 shows the input questions and answers for all the event attributes covered in HRDsAttack. And Table 10 shows all the task prefixes that we add to the beginning of the input text in our multitask training regime.

heading: E Long Document Solutions
text: Besides the two solutions we evaluate in the paper (Truncation and Knowledge Fusion), there are two other possible solutions that we describe here. First, some work proposes splitting a long document into shorter sequences, then using a transformer to generate sequence representations for each of them (Grail et al., 2021). Then those sequence representations are fed into another network to generate the final document representation. But in this case, a large number of training examples is required to learn the parameters of the network layers which generate the document representation. There is also the long transformer (Longformer (Beltagy et al., 2020)) approach proposed to handle long documents. But in contrast to the T5 model which is pretrained on many pretraining tasks, it is difficult to reframe all of the subtasks as a unified
Sequence-to-Sequence task based on those long transformers. In comparison, the approaches we proposed are all post-processing steps that are less expensive than the aforementioned methods.

heading: F Model Performances on the Development Set
text: Table 11 shows the performance of the models on the dev set of HRDsAttack. The best baseline model (T5 Hybrid) is chosen based on the model performance on the dev set.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section Limitations
3 A2. Did you discuss any potential risks of your work? Section Limitations
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5 Experiments
3 B1. Did you cite the creators of artifacts you used? Section 5 Experiments
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We plan on releasing our dataset under the MIT license as well pending legal approval. The LICENSE will be provided alongside the dataset as a text file on GitHub when the paper is published.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The Term of Use for GDELT is: all datasets released by the GDELT Project are available for unlimited and unrestricted use for any academic, commercial, or governmental use of any kind without fee. We plan on releasing our dataset under the MIT license as well pending legal approval. The LICENSE will be provided alongside the dataset as a text file on GitHub when the paper is published.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We included the worker ID for each annotation in the dataset, which is anonymized.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 3.3 Annotation Process
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3.5 Data Statistics
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 5 Experiments
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5.3 Training Implementation
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5.3 Training Implementation
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5.4 Results and Table 6
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.4 Long Document Resolution
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3.3 Annotation Process
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Section 3.3 Annotation Process
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3.3 Annotation Process
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? The Term of Use for GDELT is: all datasets released by the GDELT Project are available for unlimited and unrestricted use for any academic, commercial, or governmental use of any kind without fee.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? We used a review process internal to our organization with HCI research scientists.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Section 3.3 Annotation Process

heading: Future Work
text: While the sample size of HRDsAttack (500) is on par with some of the other EE datasets, such as ACE05 (599), we do see more samples being beneficial to subsequent model training and supporting other future studies. In addition, despite the effort to balance the class labels in the event attributes, some of the labels still remain imbalanced, such as PERPETRATOR TYPE