Unsupervised Selective Rationalization with Noise Injection
Year: 2023

Abstract:
A major issue with using deep learning models in sensitive applications is that they provide no explanation for their output. To address this problem, unsupervised selective rationalization produces rationales alongside predictions by chaining two jointly-trained components, a rationale generator and a predictor. Although this architecture guarantees that the prediction relies solely on the rationale, it does not ensure that the rationale contains a plausible explanation for the prediction. We introduce a novel training technique that effectively limits generation of implausible rationales by injecting noise between the generator and the predictor. Furthermore, we propose a new benchmark for evaluating unsupervised selective rationalization models using movie reviews from existing datasets. We achieve sizeable improvements in rationale plausibility and task accuracy over the state-of-the-art across a variety of tasks, including our new benchmark, while maintaining or improving model faithfulness.1

heading: 1 Introduction
text: With the advent of large pre-trained language models like GPT-3 (Brown et al., 2020), the size and complexity of deep learning models used for natural language processing has dramatically increased. Yet greater performance and complexity can come at the cost of interpretability, masking anything from implementation mistakes to learned bias.
A model architecture that justifies its output by providing relevant subsets of input text as a rationale is therefore desirable (see example in Figure 1). The unsupervised selective rationalization architecture as introduced by Lei et al. (2016) generates rationales alongside predictions by chaining two jointly-trained components, a rationale-generator and a predictor. The generator extracts a rationale: concatenated short and concise spans of the input
1Code and benchmark are available at https://github. com/adamstorek/noise_injection.
text that suffice for prediction. The predictor bases its prediction only on this rationale, which encourages faithfulness, meaning how much the rationale reveals what parts of the input were important to the model’s prediction. In practice, however, the rationale often isn’t plausible, meaning it can’t convince a human of the correct prediction, undermining the architecture’s interpretability (Jacovi and Goldberg, 2021; Zheng et al., 2022). Using a high-capacity generator can further degrade plausibility (Yu et al., 2019).
To prevent this effect, we introduce a novel training strategy that leverages online noise injection, based on word-level unsupervised data augmentation (Xie et al., 2020). By definition, if the lossminimizing generator selects an implausible rationale, then the rationale both (a) offers no plausible connection for a human to the target label and (b) locally improves prediction accuracy. This might include communicating via punctuation (Yu et al., 2019) or subtle input perturbations (Garg and Ramakrishnan, 2020). Our new approach is to inject noise into the generated rationale during training by probabilistically replacing lower-importance words with noise - random words from the vocabulary -
12647
before passing the rationale to the predictor. We observe that this strategy leads to a significant improvement in plausible rationale generation and prediction accuracy without compromising the faithfulness of the architecture. We also show that powerful generators typically interfere with plausible rationale generation but can be effectively deployed when trained with noise injection.
To test our approach, we introduce a new benchmark for unsupervised selective rationalization by integrating existing movie review datasets to replace the retracted canonical beer review dataset (McAuley et al., 2012; McAuley and Leskovec, 2013; Lei et al., 2016).2 We merge a large IMDb movie review dataset (Maas et al., 2011) for training and validation and a smaller, rationaleannotated movie review dataset (DeYoung et al., 2020; Zaidan and Eisner, 2008; Pang and Lee, 2004) for evaluation. We also evaluate our unsupervised approach on the ERASER Movie Review, MultiRC and FEVER tasks (DeYoung et al., 2020; Khashabi et al., 2018; Thorne et al., 2018).3
Our contributions therefore include: 1) characterizing the issue of implausible rationale generation from the perspective of powerful rationale generators, 2) introducing a novel training strategy that limits implausible rationale generation and enables unsupervised selective rationalization models with powerful generators, 3) proposing a new unsupervised rationalization benchmark by repurposing existing movie review datasets, and 4) achieving more plausible rationale generation, with up to a relative 21% improvement in F1 score and a 7.7 point improvement in IOU-F1 score against the baseline model across a number of tasks.

heading: 2 Related Work
text: A major challenge with selective rationalization is that discrete selection of rationale tokens is nondifferentiable, making training challenging without additional rationale supervision. Lei et al. (2016) use REINFORCE-style learning (Williams, 1992) to propagate the training signal from the predictor to the generator. Bastings et al. (2019) propose a differentiable approach leveraging the Hard Kumaraswamy Distribution. Yu et al. (2019) strive to improve rationale comprehensiveness. Chang et al. (2020) focus on avoiding spuriously correlated ra-
2The dataset has been retracted at the request of BeerAdvocate and is no longer in use.
3Licensing information can be found in Appendix A.
tionales. Yu et al. (2021) tackle the propensity of selective rationalization models to get stuck in local minima. Atanasova et al. (2022) use diagnosticsguided training to improve plausibility.
Our work builds on the previous approaches, since we also frame the generator-predictor interaction as a cooperative game and seek to improve plausibility. The previous approaches have, however, introduced additional training objectives (Atanasova et al., 2022) or involved incorporating a third adversarial (Yu et al., 2019) or cooperative (Yu et al., 2021) component. This increases model complexity significantly, leading to more resourceintensive and/or complicated training. Instead, we demonstrate the effectiveness of online noise injection, a considerably more lightweight approach.
An alternative approach is proposed by DeYoung et al. (2020) who assemble a series of datasets with labeled rationales; this enables fully supervised rationale learning. Given rationale-annotated training sets, Jain et al. (2020) train each model component separately, approaching the accuracy of an entirely black-box model. Although this is a compelling direction, requiring supervision reduces the practical usability of this technique, as many applications lack rationale annotations.
Both unsupervised and supervised selective rationalization approaches generally require a specific token selection strategy to select the output rationale from the generator model (Yu et al., 2021; Jain et al., 2020; Paranjape et al., 2020). No previous work that we are aware of, however, has tried to then modify the output rationale before it is input into the predictor. Using online noise injection to enforce prediction stability is therefore a novel approach that adds greater power to the current architectures and can be easily retrofitted.

heading: 3 Implausible Rationale Generation
text: Previous work has conceptualized the interaction between the generator and the predictor as a cooperative game (Chen et al., 2018a,b; Chang et al., 2019; Yu et al., 2019; Chang et al., 2020; Yu et al., 2021). This repeated sequential game consists of two-round stage games. In the first round, the generator accepts an input sequence X1:T and outputs a rationale selection as a binary mask M1:T ∈M whereM represents the set of all masks such that X1:T ⊙M1:T satisfies rationale constraints. In the second round, the predictor accepts an input sequence X1:T⊙M1:T and outputs prediction Y . The
joint objective is to minimize the loss (see Equation 2) based on the generated mask (see Equation 1):
M1:T ← gen(X1:T ; θgen),M1:T ∈M (1)
min θgen,θpre
L(pre(X1:T ⊙M1:T ; θpre), Ỹ ) (2)
For classification, it is customary to minimize the cross-entropy loss LCE . Such a system can be shown to maximize mutual information (MMI) of the rationale with respect to the class label provided sufficient generator and predictor capacity as well as a globally optimal generator (Yu et al., 2021; Chen et al., 2018a):
max M1:T∈M
I(X1:T ⊙M1:T ; Ỹ ) (3)
However, this property does not guarantee rationale plausibility.
First, MMI does not protect against spurious correlations (Chang et al., 2020). For example, a pleasant taste is not a good explanation for a positive review of a beer’s appearance, although the two aspects are strongly correlated.
Second, MMI does not prevent rationale degeneration if the generator and predictor already contain certain biases, for example from pre-training (Jacovi and Goldberg, 2021).
Third, MMI does not prevent rationale degeneration if the generator and predictor are sufficiently powerful to develop a common encoding. Yu et al. (2019) found that providing the generator with a label predicted by a full-input classifier led the generator to develop a communication scheme with the predictor, including a period for positive and a comma for negative examples. Jacovi and Goldberg (2021) argue that any generator with sufficient capacity to construct a good inner-representation of Y can cause rationale degeneration.
The key underlying cause is that a sufficiently powerful generator is not disincentivized to produce implausible rationales beyond the assumption that generating a plausible rationale should maximize the expected accuracy of the predictor in the current training iteration. However, since the predictor is treated as a black box, this is not guaranteed. On the i-th training iteration, the generator greedily selects a binary mask M1:T that minimizes the expected loss:
argmin M1:T∈M
E [L(p̃rei(X1:T ⊙M1:T ))] (4)
where p̃reG,i represents the generator’s learned representation of pre(·; θpre) from its previous experience interacting with the predictor for i − 1 iterations in game G. As i increases, the generator learns to leverage deficiencies and biases of the predictor that remain hidden to humans, resulting in rationale plausibility degeneration.

heading: 4 Online Noise Injection
text: We propose a strategy that disrupts the generator’s learned representation of the predictor p̃reG,i for all games G ∈ G, thereby making it harder for the generator to learn to exploit quirks of the predictor. We use online noise injection, which probabilistically perturbs unimportant words in a rationale sequence X of length T (see Algorithm 1).
Algorithm 1: Noise Injection. Input: input text X1:T ; binary mask M1:T Data: set of documents D; vocabulary V R1:T ← X1:T ⊙M1:T ; R∗1:T ← R1:T ; forall ri ∈ R1:T do
pi = ProbOfReplacementD(ri); replace← Binomial(1, pi); if replace then
r∗i ← SampleFromVocabD;V(); end
end return perturbed rationale R∗1:T
If the generator attempts to generate an implausible rationale during training iteration i, it strategically includes unimportant words from the input text in the generated rationale, relying on the predictor to pick up on the bias. By subtly perturbing the rationale - replacing the unimportant words - noise injection disrupts this attempt, and the predictor does not respond to the generator-injected bias favorably as expected by the generator. The generator is therefore forced to unlearn/reset its representation p̃reG,i of the predictor and reassess its strategy, learning that generating implausible rationales is ineffective. Across any two stages i, j of game G, noise injection therefore keeps the learned representations of the predictor more consistent:
∀G ∈ G, ∀i, j ∈ stages(G), p̃reG,i(·) ≈ p̃reG,j(·) (5)
We implement the ProbOfReplacement and SampleFromVocab functions by adapting a strategy
that probabilistically replaces words with small TF*IDF, originally proposed for unsupervised data augmentation by Xie et al. (2020). We precompute the probability of replacement of each word wi ∈ d in each document d ∈ D as its normalized TF*IDF score multiplied by the document length and a hyperparameter representing the magnitude of augmentation p:
wmax − TF*IDF(wi)∑ w∈dwmax − TF*IDF(w) p|d| (6)
wmax = max w∈d TF*IDF(w) (7)
We use these precomputed probabilities to sample which words to replace as shown in Algorithm 1. The words are replaced with random words from the vocabulary V . Nonetheless, we also strive to prevent sampling "keywords" from the vocabulary - words that are highly indicative of a label - to avoid confusing the predictor. We compute the sampling probability of wi as its normalized ATF*IDF, where ATF corresponds to term frequency macroaveraged over D:
w∗max − ATF*IDF(wi)∑ w∈dw ∗ max − ATF*IDF(w)
(8)
w∗max = max w∈d ATF*IDF(w) (9)

heading: 5 Model
text: Our baseline model builds on the A2R architecture by Yu et al. (2021) who improve training stability by using an auxiliary predictor connected directly to the generator via an attention layer - this allows for gradients to flow. A2R selects top-k2 bigrams with the highest attention scores from the generator as the rationale and input for the second predictor, with k corresponding to the number of rationale tokens selected as a fraction of the size of the input text. The two components minimize their separate criteria as well as the Jensen-Shannon divergence of their predictions Y a and Y r for the attentionbased predictor and the rationale-based predictor, respectively. A2R’s generator consists of a fixed GloVe (Pennington et al., 2014) embedding layer and a linear token scoring layer.
To take full advantage of our noise injection strategy, we replace the limited-capacity generator with BERT (Devlin et al., 2019). This allows us to use a simpler attention-based predictor than A2R (see Figure 2). To further manifest the efficacy of noise
injection, we opt for a top-k unigram selection strategy which offers less regularization compared to a bigram selection strategy. Selecting unigrams is more challenging because it allows the model to select uninformative stopwords like "a" or "the".
Our architecture is shown in Figure 2. Both the selection strategy and the noise injection are modelexternal and untrained. As in Yu et al. (2021), the attention-based (see Equation 10) and the rationalebased (see Equation 11) components are trained using identical objectives - minimizing the sum of the cross-entropy loss and the Jensen-Shannon divergence of the two predictors:
La = LCE(Y a, Ỹ ) + λJSD(Y a, Y r) (10)
Lr = LCE(Y r, Ỹ ) + λJSD(Y a, Y r) (11) We refer to our model as BERT-A2R and add +NI when noise injection is used during training.

heading: 6 USR Movie Review Dataset
text: Previous work on unsupervised selective rationalization used a decorrelated subset of the BeerAdvocate review dataset (McAuley et al., 2012) as preprocessed by Lei et al. (2016). The dataset has recently been removed at the request of BeerAdvocate and is therefore inaccessible to the scientific community. BeerAdvocate reviews consists of 80,000 labeled reviews without rationales for training/validation and ∼1,000 labeled reviews with token-level annotated rationales for testing. Alternative datasets either include rationale labels for the entire dataset (DeYoung et al., 2020) or do not provide rationale labels altogether (e.g. Maas et al. (2011)). Moreover, large datasets such as MultiRC or FEVER tend to provide sentence-level rationales compared to BeerAdvocate token-level rationales. We thus repurpose existing movie review datasets to recreate a task similar to beer review, enabling new work on unsupervised selective rationalization to evaluate their performance against models designed for beer review. We merge a smaller ERASER Movie Review dataset (DeYoung et al., 2020; Zaidan and Eisner, 2008; Pang and Lee, 2004) that has full token-level rationale annotations with the lower-cased Large Movie Review Dataset (Maas et al., 2011) which has no rationale annotations.
The movie review task is similar to the binarized beer review task as used in Chang et al. (2019); Yu et al. (2019); Chang et al. (2020); Yu et al. (2021);
both are binary sentiment classification tasks based on English user reviews. However, human rationale annotations of Eraser Movie Review are less coherent and consistent than beer review (see Figure 3) and lack single-aspect labels comparable to beer review’s appearance, aroma, and taste labels. Moreover, movie review annotations tend to be over-complete (Yu et al., 2021): the same relevant information is often repeated many times in each review. This new task therefore also evaluates previous models’ robustness to a subtle distribution shift, an increasingly important consideration for real-world systems.
The reviews from the ERASER Dataset were collected and processed by Pang and Lee (2004) from the IMDb archive of the rec.arts.movies.reviews newsgroup, whereas the Large Movie Review Dataset was scraped from the IMDb website by Maas et al. (2011). In order to avoid overlap between the train and test sets, we looked for similarity by searching for matches between lowercased, break-tag-free, stop-word-free, lemmatized sentences which spanned at least 5 tokens to avoid generic matches such as "would not recommend" or "great film !". We discovered no overlap between the datasets. We use 40,000 reviews from the Large Movie Review Dataset for training and the remaining 10,000 reviews for validation. We then test our model on the 2,000 annotated examples from ERASER Movie Review.

heading: 7 Experimental setup
text: Metrics We evaluate generated rationales across several datasets using different metrics that capture
faithfulness and plausibility. Faithfulness captures the extent to which the generated rationales truly explain the model’s output. For faithfulness, we use comprehensiveness and sufficiency metrics (DeYoung et al., 2020). A rationale is comprehensive if it extracts all the information contained in the input text that is relevant for prediction and sufficient if it contains enough relevant information to make an accurate prediction. The comprehensiveness score measures the difference between the model’s predictions on the entire input text and the input text without the selected rationale (higher is better), whereas the sufficiency score measures the difference between the model’s predictions on the entire input text and just on the rationale (lower is better).
For plausibility, we use standard alignment metrics in reference to the human-annotated rationales: precision, recall, and F1 score as well as IOU-F1 score (referred to as IOU in tables) with partial match threshold 0.1 (DeYoung et al., 2020; Paranjape et al., 2020). We use token-level metrics for Movie Review which offers token-level annotations and sentence-level metrics for MultiRC and FEVER which provide only sentence-level annotations. Finally, we report prediction accuracy for the overall classification task. All results are averaged across 5 random seeds and reported as the mean with standard deviation in parentheses.
Implementation Our BERT-A2R models are trained for a maximum of 20 epochs for ERASER Movies and 5 epochs for every other dataset, keeping the checkpoint with the lowest validation loss. All BERT-A2R variants use uncased BERT-base, A2R closeness parameter λ = 0.1, and the selection strategy of picking the top k = 20% of the highest attention-scoring tokens for movie review or sentences for MultiRC and FEVER. We compute sentence-level scores by taking sentence-level averages of token scores. For optimization, we used Adam (Kingma, D.P. et al., 2015) with learning rate 2e-5 and batch size 16. Noise injection level p was set to 0.2 for USR and ERASER Movie review, 0.3 for MultiRC, and 0.05 for FEVER. This was determined based on our hyperparameter search. All of the models were trained on a single machine equipped with a 12-core processor, 64 GB of RAM, and a GPU with 24 GB of VRAM. 4
4Training details can be found in Appendix B.

heading: 8 Results

heading: 8.1 Does noise injection improve selective rationalization?
text: To compare against previous published results, we trained a BERT-A2R model on the ERASER Movie Review dataset with and without noise injection and compared our numbers to published results from the best unsupervised selective rationalization systems on this benchmark (see Table 1). All models were trained without rationale supervision. We see that our model with noise injection improves on both the classification task accuracy and the rationale F1 score relative to previous systems. Note that noise injection improves the F1 score more than the introduction of BERT to A2R.
We then train BERT-A2R models with and without noise injection on the MultiRC and FEVER benchmarks (see Table 2) as well as on our new USR Movie Review benchmark (see Table 3). Again, our noise injection training strategy achieves statistically significant improvements in rationale alignment with human annotations (p < 0.01 on the MultiRC and USR Movies, p < 0.05 on the FEVER, and p < 0.1 on ERASER Movies), achieving up to a relative 21% improvement in F1 score over our already performant baseline. The plausibility improvement applies for both tokenlevel and sentence-level extraction tasks and across all metrics. Prediction accuracy also improves across all tasks except FEVER. Noise injection also does not seem to have a negative impact on model faithfulness. On ERASER benchmarks, neither comprehensiveness nor sufficiency worsen dramatically, and in the case that one score worsens, the other score tends to remain stable or even improve. On USR movie review, we see an improvement in both faithfulness scores from using noise injection.

heading: 8.2 How does the noise injection level p affect model performance?
text: We train variants of BERT-A2R+NI with different levels of p to examine what noise level is optimal for different datasets (see Figure 4). We average results across 5 seeds but there is still some noise given that the methodology injects noise into the process. It appears that in all cases noise injection seems to degrade performance once p becomes too high as we would expect since too much noise prevents useful signal from getting through. The optimal p varies depending on the task. Rationale alignment performance on FEVER peaks at just p = 0.05. The optimum for ERASER and USR Movie Review is at p = 0.1 and p = 0.2, respectively. The best performance on MultiRC was achieved at p = 0.3. There are numerous factors that might interact with noise injection to cause this behavior: task-specific demands, sentence vs. token-level rationale annotations, and the suitabil-
ity of other training parameters. These interactions might be complex, especially with training strategies that dynamically adjust p during training. We leave exploration of these factors for future work.

heading: 8.3 Does noise injection enable the use of powerful high-capacity rationale generators?
text: For this experiment, we train BERT-A2R with fixed or trainable BERT weights in the generator, with or without noise injection, and evaluate on our new USR Movie Review benchmark (see Table 3). The version with fixed BERT weights in the generator has much less trainable capacity and cannot learn a task-specific text representation, whereas the generator with trainable BERT weights can potentially learn much better rationales or degrade to implausible rationales.
We find that the tuned generator trained with noise injection achieves superior performance across all the rationalization metrics without compromising prediction accuracy (2.8 improvement in rationale F1 score and a 7.7 improvement in rationale IOU-F1 score relative to the fixed setting). In contrast, the tuned generator without noise injection training performed the worst in all rationale metrics as well as prediction accuracy. Noise injection with a fixed generator results in a minor improvement in both plausibility metrics and prediction accuracy. We can therefore observe not only that noise injection allows us to leverage the power of a tunable BERT model in the generator that previously would have resulted in performance degradation, but also that the benefits of noise injection are greater with a powerful high-capacity generator model.
Finally, the addition of noise injection training also slightly improves comprehensiveness for both fixed and tuned generators while improving suffi-
ciency for the tuned generator.

heading: 8.4 What errors do the models make?
text: For our qualitative analysis we randomly selected 20 reviews to evaluate the effect of adding noise injection to BERT-A2R during training. From this review sample, we include examples that we believe are characteristic for the behavior we observed. First, a BERT-A2R trained with noise injection tends to select longer spans of text as rationales (see Figure 6, 7), generally without sacrificing precision compared to the baseline. Selecting continuous rationales greatly improves readability and human-alignment as noted by Lei et al. (2016).
We also observed that BERT-A2R + NI occasionally fails to select generic words such as "film" that, nevertheless, form a part of the rationale (see Figure 5). This could be a downside to our noise injection strategy, since the model will learn to ignore words with low TF*IDF even though they are relevant in a minority of cases. A potential remedy might be to use task-specific heuristics to generate probability of replacement information instead of the general low TF*IDF strategy. We leave this for future work.
Conclusion
In this paper, we investigate a major obstacle of unsupervised selective rationalization frameworks, where the generator has a tendency to learn to generate implausible rationales: rationales that lack a convincing explanation of the correct prediction. We explain the generator’s propensity towards degeneration in terms of a flawed incentive structure, characterizing unsupervised selective rationalization as a sequential, repeated cooperative game. Through this lens, we propose a novel training strategy that penalizes implausible rationale generation, thereby realigning the incentive structure with the objective to generate plausible rationales. Using a new benchmark for unsupervised selective rationalization, we show that our noise injection approach is beneficial for training high-capacity generators, outperforming the current state of the art models.
Limitations
One of the main limitations of the noise injection training strategy is that statistics used to determine probability of replacement and sampling probability are token-specific. Although this works well on languages with limited morphology such as English, inflected languages like Czech that rely on declension and conjugation might require a lemmabased strategy or a different technique altogether. Furthermore, the model extracts a rationale of fixed length k, proportional to the length of the input text. Nevertheless, input text might include more or less information relevant to the class label; a sparsity objective as proposed by Paranjape et al. (2020) could remedy this issue. Lastly, injecting noise during training sometimes leads to more unpredictable training runs.
Additional model limitations are connected to using BERT. Despite its performance and fast training, using BERT limits the scalability to long text due to the 512-token limitation; nevertheless, tasks involving long text might be able to leverage specialized approaches such as Beltagy et al. (2020). Likewise, BERT renders BERT-A2R about 20 times larger than the GRU-based A2R, requiring greater GPU resources.
The dataset also comes with a few limitations. As Yu et al. (2021) note, some reviews contain many clear explanations for the target label, decreasing the need for the generator to include all relevant explanations in the rationale. Similarly, the sparsity of human-annotated rationales can be
inconsistent across reviews: as shown in Figure 3, some rationales include long, generous spans of text that contain irrelevant information, whereas other rationales consist of merely the most important phrases.
Ethics Statement
We believe that improving the effectiveness and efficiency of unsupervised selective rationalization in the context of large pre-trained models such as BERT (Devlin et al., 2019) can help uncover and mitigate their learned bias as well as any implementation mistakes. Enabling models to produce plausible faithful rationales increases transparency, improving the end-user’s understanding of the model’s prediction and allowing AI practitioners to make more informed ethical choices in deploying models.

heading: Acknowledgments
text: This research was conducted with funding from the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001120C0123. The views, opinions, and findings expressed are those of the authors and do not represent the official views or policies of the U.S. Department of Defense or the U.S. Government.

heading: A Licensing

heading: B Training Details
text: Total estimated GPU hours spent on training: 500. BERT-A2R has 109484547 parameters.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: No section number but directly following conclusion.
7 A2. Did you discuss any potential risks of your work? We do not see that our work introduces any new risks over the already published and publicly available previous work. In fact, we believe that better rationale plausibility improves interpretability and fairness, thereby reducing the risk that black-box models pose to the general public, especially to the historically disadvantaged groups.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 5, 8

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 1-8

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix A

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 5-8
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We haven’t collected any data ourselves.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We haven’t collected any data ourselves.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix B; For each split of each dataset, we included the number of examples.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? 7-8

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 7, Appendix B

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 7-8

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 7-8

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 7, Appendix A
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave this for future work