Revisiting Pathologies of Neural Models under Input Reduction
Year: 2023

Abstract:
We revisit the question of why neural models tend to produce high-confidence predictions on inputs that appear nonsensical to humans. Previous work has suggested that the models fail to assign low probabilities to such inputs due to model overconfidence. We evaluate various regularization methods on fact verification benchmarks and find that this problem persists even with well-calibrated or underconfident models, suggesting that overconfidence is not the only underlying cause. We also find that regularizing the models with reduced examples helps improve interpretability but comes with the cost of miscalibration. We show that although these reduced examples are incomprehensible to humans, they can contain valid statistical patterns in the dataset utilized by the model.1

heading: 1 Introduction
text: During the development stage, we put much effort into tuning neural models to achieve high accuracy on held-out data. However, when deploying such tuned models in real-world scenarios, it is also important for them to be reliable. For example, when a fact verification model judges that a claim is true with a confidence of 0.95, it should have a 95% chance of being correct. Meanwhile, lowconfidence predictions can be passed onto humans to be double-checked manually. If the model can align its confidence with the correctness, it is considered calibrated. Despite achieving human-level performance on various tasks, recent studies (Guo et al., 2017; Ovadia et al., 2019; Hendrycks et al., 2020) have shown that modern neural models tend to be miscalibrated.
Miscalibration further reveals an anomaly of neural models in which they tend to produce highconfidence predictions on inputs that appear nonsensical to humans. Figure 1 shows examples from
1Our code is available at https://github.com/ nii-yamagishilab/pathologies.
the COVIDFACT dataset (Saakyan et al., 2021) where the fact verification model still makes the same correct prediction given the reduced version of the original claim. Feng et al. (2018) first discovered such pathologies of neural models on widely used NLP datasets, such as SQUAD (Rajpurkar et al., 2016) and SNLI (Bowman et al., 2015). They attributed the main underlying cause to model overconfidence and proposed a regularization method incorporating reduced examples to mitigate the problem. While the interpretability could be improved, it is unclear how the reduced examples affect model calibration. In addition, their method is based on an entropy regularizer called the confidence penalty (Pereyra et al., 2017), and other possible techniques still remain uninvestigated.
In this paper, we explore a family of regularization methods and propose an extension that unifies label smoothing (Szegedy et al., 2016) and
11504
the confidence penalty (Pereyra et al., 2017). We conducted experiments on three fact verification datasets and found that:
• Pathologies still occur even when the model is well-calibrated or underconfident.
• Incorporating the reduced examples improves interpretability (i.e., increases the input lengths) but amplifies miscalibration (i.e., increases calibration errors).
Our results suggest that model overconfidence is not the only cause of pathological behaviors. Regularizing the objective function with the reduced examples encourages the model to output high entropy (i.e., low confidence) on such examples. However, these reduced examples can also contain valid statistical patterns that are sufficient for the model (but nonsensical to humans) to make predictions. This finding has also been observed in computer vision (Carter et al., 2021).

heading: 2 Task formulation

heading: 2.1 Datasets
text: We focus on the task of fact verification, which involves classifying a claim as supported (SUP), refuted (REF), or not enough information (NEI) with respect to evidence. We conduct experiments on three datasets:
COVIDFACT (Saakyan et al., 2021) starts from valid real-world claims and evidence sentences from peer-reviewed research documents concerning the COVID-19 pandemic. They then generated counterclaims by replacing the most salient word in the original claim using language model infilling with entailment-based quality control. The dataset consists of 3,263/419/404 samples in the training/dev/test sets with two classes: SUP and REF.
FEVER (Thorne et al., 2018) is from the Fact Extraction and VERification challenge, which has three subtasks: document retrieval, sentence selection, and fact verification. We only consider fact verification and use the data preprocessed by Schuster et al. (2021), which consists of 178,059/11,620/11,710 samples in the training/dev/test sets with three classes: SUP, REF, and NEI.
VITAMINC (Schuster et al., 2021) augments FEVER with the symmetric annotation strategy (Schuster et al., 2019). Given a claim-evidence
pair from FEVER, they first edited the evidence sentence to flip the original label (e.g., REF→SUP) and then composed a new claim that holds the original label for the new, edited evidence sentence. They also collected new samples from Wikipedia revisions, but we only use the synthetically created dataset, which consists of 121,700/20,764/20,716 samples in the training/dev/test sets with two classes: SUP and REF.

heading: 2.2 Architecture
text: We formulate our task as supervised multi-class classification. Our aim is to train a model that can assign a label y ∈ Y = {1, . . . ,K} to an input x ∈ X . Our model is a neural network h parameterized by θ:
hθ(x) = MLP(PLM(x)),
where MLP is a multilayer perceptron and PLM is a pre-trained language model. Each PLM layer transforms x into a sequence of hidden state vectors.2 Following standard practice, we obtain the fixedlength vector representation of x from the first hidden state vector of the last PLM layer. The MLP then maps the vector representation toK unnormalized logits. Finally, we apply the softmax function to obtain the predicted distribution p ∈ RK over labels:
p(y|x) = softmax(hθ(x)).
Let q ∈ RK denote the ground-truth label distribution (i.e., one-hot encoding). During training, we aim to minimize the cross-entropy loss between q and p:
Lce = H(q, p) = ∑
y∈Y q(y|x) log 1 p(y|x) . (1)

heading: 3 Input Reduction
text: Model interpretation methods offer explanations for model predictions (Ribeiro et al., 2016; Li et al., 2016; Wallace et al., 2019). The goal is to understand why the model made specific predictions. A brute-force method is to look at model weights, but they are incomprehensible. Because most modern neural architectures (including ours) rely on attention mechanisms, attention weights over inputs are often used as explanations. However, subsequent
2In our case, an input x is a concatenation of claim and evidence sentences.
Algorithm 1 Input reduction Require: Original x 1: ŷ = argmaxy∈Y p(y|x) 2: while true do 3: w∗ = argminw∈x‖∇ewLce‖ 4: x̃← x \w∗ 5: ỹ = argmaxy∈Y p(y|x̃) 6: if ỹ == ŷ and x̃ 6= ∅ then 7: x← x̃ 8: else 9: break 10: end if 11: end while 12: return Final x
studies have argued that attention weights can be manipulated (Pruthi et al., 2020) and uncorrelated with feature importance measures (Jain and Wallace, 2019).
In our work, we focus on a gradient-based method called input reduction (Feng et al., 2018). The idea is to find a minimal input subset sufficient for attaining the same prediction as the original input. This minimal input subset can be regarded as a rationale, i.e., a few substrings that are sufficient for justifying predictions (Zaidan et al., 2007).
Input reduction iteratively removes the least important word from the original input until the model changes its prediction. In our case, the basic unit is a token, which can be a word or a subword. Let w ∈ x denote a token in the input and ew denote its embedding vector obtained from the PLM. Algorithm 1 summarizes the process of our input
Evidence Epistemology studies the nature of knowledge, justification, and the rationality of belief.
Original refuted claim Epistemology has nothing to do with the study of the rationality of belief.
Dataset: FEVER
reduction. Note that the ground-truth label is unnecessary for input reduction. We estimate the importance of each w through the hallucinated gradient of the loss with respect to the embedding vector and the predicted label. At each iteration, we remove the token having the smallest gradient norm (Wallace et al., 2019). We only proceed if the new predicted label of the reduced input x̃ is the same as that of the original input x.
Our inspection
Recall that our input is a sentence pair consisting of claim and evidence sentences. To conform with Feng et al. (2018), we remove tokens from the claim only (equivalent to the hypothesis in SNLI) and keep the evidence untouched. Figure 2 shows the reduction path of the refuted claim from the COVIDFACT dev set generated using Algorithm 1. Figure 3 shows additional examples of the original and reduced claims from FEVER and VITAMINC.
Figure 4 compares the claim lengths before and after reduction on the FEVER, VITAMINC, and COVIDFACT dev sets. Unlike Feng et al. (2018), we examine the results in detail by class. Feng et al. (2018) reported that the reduced examples only contain one or two words on average across all of their tasks. However, we find that their observation holds on particular classes on our specific datasets. The NEI/REF claims can be reduced to a few tokens without changing the original predic-
tions. On the contrary, we observe that the SUP claims need to remain longer to retain the original predictions.
Our observation seems to correlate with factchecking data construction. The process usually starts with creating valid claims (i.e., SUP) and modifying them to create other types (REF/NEI), which leaves annotation artifacts (Gururangan et al., 2018) or shortcuts (Geirhos et al., 2020), enabling the model to use them for predictions.

heading: 4 Regularization methods
text: In this section, we review widely used regularization methods, inspect their properties, and introduce our extension.

heading: 4.1 Existing methods
text: Temperature scaling (Guo et al., 2017) is a simple yet effective regularization method that simplifies Platt scaling (Platt, 1999) by adjusting the unnormalized logits with only one parameter, temperature τ ∈ R:
p(y|x) = softmax(hθ(x)τ ).
We can soften the predicted distribution by setting τ > 1. Following Guo et al. (2017), we use temperature scaling as a post-processing method so that the model accuracy is preserved (i.e., the predicted labels remain unchanged). We optimize τ with
respect to Lce (defined in Eq. (1)) on the development set. This procedure differs from the softmax temperature used in knowledge distillation (Hinton et al., 2015), which involves training a small model with the soft target labels from a larger model.
Label smoothing (Szegedy et al., 2016), in contrast to temperature scaling, softens the groundtruth label distribution q. Label smoothing replaces q with q′ = (1− )q + u(y), where is a balancing parameter, and u(y) is the uniform distribution over labels (i.e., u(y) = 1K ). For notational convenience, we scale down q′ by 1/(1− ) so that:
q′s = q + βu(y),
where β = (1− ) (Meister et al., 2020). By applying Eq. (1), we can derive the label smoothing loss as:
Lls = H(q′s, p)
= ∑
y∈Y (q(y|x) + βu(y)) log 1 p(y|x)
= ∑
y∈Y q(y|x) log 1 p(y|x)
+ β ∑
y∈Y u(y) log
1
p(y|x)
= Lce + β H(u, p). (2)
The above equation consists of the usual crossentropy loss and the regularization function
H(u, p). It is also equivalent to the cross-entropy form of Szegedy et al.’s (2016) label smoothing.
Confidence penalty (Pereyra et al., 2017), as its name suggests, penalizes the confident predicted distribution p. We can measure the degree of confidence in p by using the entropy H(p). A high confidence p corresponds to a low H(p) and vice versa. Pereyra et al. (2017) defined the confidence penalty loss as:
Lcp = Lce − β H(p). (3)
The regularization function of the above equation becomes the negative entropy H(p). The balancing parameter β enables a trade-off between minimizing the cross-entropy loss and maximizing the entropy of the predicted distribution p.

heading: 4.2 Observations
text: Guo et al. (2017) empirically found that model miscalibration is due to negative log-likelihood overfitting. Here, we interpret this phenomenon from a Kullback–Leibler (KL) divergence perspective. Let H(q) denote the entropy of the ground-truth label (one-hot) distribution, which is a constant. We rewrite the cross-entropy loss in Eq. (1) as:
Lce = H(q, p)− H(q) + H(q) = KL(q ‖ p) + H(q)︸︷︷︸
constant
. (4)
Thus, minimizing Lce is equivalent to minimizing the KL divergence between the ground-truth label distribution q and the predicted distribution p (i.e., pushing p towards q). When overfitting occurs, the model places most of the probability mass to a single label, resulting in peakiness in p. Typically, mitigating model miscalibration involves making p less peaky.
We can also express the label smoothing loss in KL divergence form. We know that:
KL(u ‖ p) = H(u, p)− H(u), (5)
Therefore, we can rewrite Eq. (2) as:
Lls = Lce + β KL(u ‖ p) + β H(u)︸ ︷︷ ︸ constant .
Thus, minimizing Lls is equivalent to finding a balance between pushing p towards q (as defined
3The number of classes in COVIDFACT is 2, so β H(u) = 0.1 log(2) ≈ 0.069.
in Eq. (4)) and towards u. Likewise, we can express the confidence penalty loss in (reverse) KL divergence form. Since:
KL(p ‖ u) = H(p, u)− H(p), (6)
we reformulate Eq. (3) as:
Lcp = Lce + β KL(p ‖ u)− β H(p, u)︸ ︷︷ ︸ constant .
Since the KL divergence is always non-negative, it follows from Eqs. (5) and (6) that H(p) is upper bounded by H(u, p):
H(u, p) ≥ H(u) = H(p, u)4 ≥ H(p).
We inspect the above relationship by plotting H(u, p) and H(p) in Lls and Lcp, respectively, as shown in Figure 5. We trained the models for 10 epochs with β = 0.1. Each epoch can have many iterations depending on the mini-batch size.
Interestingly, both curves appear to be mirror images of each other in the early iterations. H(u, p) and H(p) start close to H(u), meaning that the models place almost equal probabilities on both labels. As the number of iterations increases, the models become more and more confident in their predictions, and H(u, p) and H(p) gradually diverge from H(u). Another observation is that H(p) heavily penalizes the confidence penalty loss in Eq. (3) at
4The equation H(u) = H(p, u) follows from the fact that H(u) = ∑ y∈Y u(y) log 1 u(y)
= logK and H(p, u) =∑ y∈Y p(y|x) log 1u(y) = logK.
the beginning iterations because H(p) starts close to H(u) (i.e., the maximum entropy). However, the effect of H(p) diminishes because its value approaches zero at the final iterations. This behavior is contrary to that of H(u, p).

heading: 4.3 Proposed extension
text: Being able to represent Lls and Lcp in asymmetric KL divergence forms encourages us to pursue their symmetric counterpart. A known symmetric form of the KL divergence is the Jeffreys (J) divergence (Jeffreys, 1946), defined as J(p1 ‖ p2) = KL(p1 ‖ p2) + KL(p2 ‖ p1).5 On the basis of the J divergence, we derive our loss as:
LJ = Lce + β J(u ‖ p) = Lce + β ( KL(u ‖ p) + KL(p ‖ u) )
= Lce + β ( H(u, p)− H(p) ) . (7)
The regularization term of Eq. (7) simply becomes the combination of those of Lls and Lcp from Eqs. (2) and (3), respectively.

heading: 5 Hybrid methods
text: Feng et al. (2018) proposed a regularization method to mitigate overconfident predictions on nonsensical inputs, specifically by modifying Pereyra et al.’s (2017) confidence penalty with the reduced examples. The idea resembles data augmentation, but they only used the reduced examples for computing the regularization function. They first applied input reduction (described in §3) to the original training set to obtain its reduced version X̃ . Let p̃(y|x̃) denote the predicted distribution given the reduced example x̃ ∈ X̃ . By modifying Eq. (3), Feng et al.’s (2018) loss function can be expressed as:6
Lc̃p = Lce − β H(p̃). (8)
Therefore, the model will attempt to maximize H(p̃) (i.e., making p̃ less peaky) to reduce the overall loss.
Proposed extension Because the modification in Eq. (8) only involves the regularization function, this motivates us to apply the same idea to Lls and LJ . From Eqs. (2)
5Another symmetric form is the Jensen–Shannon (JS) divergence (Lin, 1991). We discuss its properties in Appendix A.
6Feng et al. (2018) formulated their problem as maximization, so the sign of the regularization term in their paper is positive.
and (7), we derive two additional loss functions that incorporate the reduced examples:
L l̃s = Lce + β H(u, p̃), (9)
and
L J̃ = Lce + β J(u ‖ p̃). (10)

heading: 6 Experiments

heading: 6.1 Training details
text: We implemented our model (described in §2.2) on top of Hugging Face’s Transformers library (Wolf et al., 2020). For the PLM, we used RoBERTabase (Liu et al., 2019). For optimization, we used Adafactor (Shazeer and Stern, 2018) with a learning rate of 3e-5, a linear learning rate decay, a warmup ratio of 0.02, and a gradient clipping of 1.0. We trained each model for 10 epochs or until the validation accuracy had not improved after three times (i.e., early stopping with a patience of 3). Early stopping can also be regarded as a regularization method to alleviate overfitting.
We used a batch size of 256 for FEVER and VITAMINC. Following Saakyan et al. (2021), we used a batch size of 16 for COVIDFACT. We found that using a large batch size yields lower accuracy on COVIDFACT. One plausible explanation is that COVIDFACT has a much smaller training set than FEVER and VITAMINC. We fixed the model hyperparameters and searched for an optimal β in the range of {0.05, 0.1, 0.3, 0.5} for the regularization methods (§4) and their variants (§5) on the dev set. We conducted all experiments on NVIDIA Tesla A100 GPUs.

heading: 6.2 Assessing model miscalibration
text: The common practice of assessing model miscalibration is to visualize the probability outputs with confidence histograms and reliability diagrams (Niculescu-Mizil and Caruana, 2005; Guo et al., 2017). Further, these visualizations can be summarized by a single number using the expected calibration error (Naeini et al., 2015).
Confidence histograms: Let p̂j denote the confidence score of the jth sample where p̂j = maxyj∈Y p(yj |xj). We first divide the confidence range of [0, 1] into M equal-size bins. The ith bin covers the interval of ( i−1M , i M ]. We then assign each p̂j to its corresponding interval. To plot a confidence histogram, we compute the percentage of samples in each bin.
Reliability diagrams: Let ŷj denote the predicted label of the jth sample where ŷj = argmaxyj∈Y p(yj |xj) and Bi denote the set of samples belonging to the ith bin. To plot a reliability diagram, we compute the average accuracy of the ith bin:
acc(Bi) = 1 |Bi| ∑
j∈Bi 1(ŷj = yj),
where 1(·) is the indicator function.
Expected calibration error: In the same manner as acc(Bi), we compute the average confidence of the ith bin:
conf(Bi) = 1 |Bi| ∑
j∈Bi p̂j .
The expected calibration error (ECE) is the weighted average of the gaps between acc(Bi) and conf(Bi) of all bins:
ECE = M∑
i=1
|Bi| N |acc(Bi)− conf(Bi)|,
where N is the number of all samples.

heading: 6.3 Results
text: We report the accuracy (Acc), ECE, and average claim length (Len) after input reduction. The average length acts as a proxy for quick assessment of whether there are any differences among model’s predictions. An increase in the length would mean that the reduced claims are less likely to appear nonsensical to humans (Feng et al., 2018), though further inspection would be necessary.
Effect of regularization
Our proposed LJ produces the lowest ECE on COVIDFACT and VITAMINC, as shown in Table 1 (middle section). Generally, all entropy regularization models yield lower ECE than temperature scaling. Figure 6 compares the confidence histograms and reliability diagrams of the baseline model with those of the best regularization models. The baseline Lce shows severe miscalibration on COVIDFACT. Our proposed LJ helps bridge the gaps between the accuracy and confidence of all bins. Surprisingly, Lce already produces low ECE on FEVER and VITAMINC, while Lcp and LJ further improve the accuracy-confidence alignment.
The results on FEVER and VITAMINC also demonstrate that the models become underconfident in the last bin (i.e., the interval of (0.95, 1]), which contains most of the model’s predictions. Feng et al. (2018) suggested that the pathological behaviors of the models is a consequence of model overconfidence. In contrast, our results show that this problem still occurs even when the model is well-calibrated or underconfident.
Effect of incorporating reduced examples in training
Table 1 (bottom section) shows the results of the hybrid models (described in §5), which augment the training set with the reduced examples and use them in the regularization function. During training, incorporating the reduced examples encourages the model to output high entropy (i.e., low confidence) on such examples. Consequently, during testing, the hybrid models can no longer reduce the input sentence to a very short length while maintaining high confidence. While these models
increase the average length, they deteriorate ECE compared to their normal versions.
Are reduced examples valid statistical patterns in the dataset?
Following Carter et al. (2021), we constructed additional datasets from the reduced examples. Recall that input reduction relies on the predicted label from the model when producing reduced examples. The reduced example only maintains the original model prediction, which can be correct or incorrect. Here, we replaced the predicted label with the corresponding ground-truth label for each reduced example to create the reduced datasets. Thus, the reduced example is not the optimal representative of the original one with the true label. We can expect discrepancies to a certain extent.
Table 2 shows the results of our baseline Lce on various settings. The original-original rows are from Table 1. We observe slight drops in accuracy when training/evaluating on the reduced datasets (i.e., reduced-reduced rows). The reduced examples produced by input reduction yield higher accuracy than those created by randomly selecting tokens in all settings. These results indicate that although the reduced examples do not align with human intuitions, they indeed contain valid statistical patterns in the datasets.

heading: Do longer reduced examples capture more meaningful information?
text: An ideal way to check whether longer reduced examples capture more meaningful information is to ask humans to evaluate the reduced claims, but this is time-consuming and costly. Here, we exploited a characteristic of COVIDFACT in which the counterclaim differs from the original claim in only one salient word, as shown in Figure 1. This enables us to perform the automatic evaluation. We first chose all reduced claims where the predictions are correct. We then checked whether the salient word in the original claim is present in the reduced claim.
Table 3 shows that Lc̃p captures more salient words than other models on COVIDFACT. Appendix B provides additional examples where Lc̃p can successfully retain salient words. However, the ECE of Lc̃p increases to close to that of baseline Lce (13.5 vs. 15.2), as shown in Table 1. Figure 7 shows that the gaps between accuracy and confidence of Lc̃p are amplified for almost all bins compared to Lcp. A simple remedy for Lc̃p is to post-process the outputs with temperature scaling. We found that the ECE of Lc̃p decreases from 13.5 to 12.4 with a temperature τ of 1.2.

heading: 7 Conclusion
text: We revisited the pathological behaviors of neural models in which they tend to be overconfident on inputs that appear meaningless to humans. We first analyzed the commonly used fact verification benchmarks with input reduction (Feng et al., 2018) and found that we could only shorten particular types of claims into a few tokens without changing the model’s predictions. We explored various entropy regularization methods and also proposed our extensions. We found that regularizing the
0
20
40
60
% o
f s am
pl es cp w/ = 0.05
0.0 0.2 0.4 0.6 0.8 1.0 0.0
0.2
0.4
0.6
0.8
1.0
Ac cu
ra cy
ACC = 82.9 ECE = 7.3
Expected Actual
COVIDFACT
Confidence
objective function with the reduced examples improves interpretability but deteriorates calibration. Training neural models that use more meaningful features while being well-calibrated is an important direction for future work.

heading: 8 Limitations
text: Our work has several limitations. We focused on fact verification, which formulates the task sentence-pair (i.e., claim-evidence) classification. Our findings may hold for certain domains where the task format is similar (e.g., natural language inference or textual entailment recognition). We did not apply beam search on input reduction, which limits us from searching multiple versions of the reduced claims having the same length. We investigated three widely used regularization methods: temperature scaling, label smoothing, and the confidence penalty. However, other subsequent methods remain unexplored.

heading: Acknowledgments
text: This work is supported by JST CREST Grants (JPMJCR18A6 and JPMJCR20D3), JST AIP challenge program, and MEXT KAKENHI Grants (21H04906), Japan.

heading: A Relationship between Jeffreys (J) divergence and Jensen–Shannon (JS) divergence
text: We can express the JS divergence between u and p as:
JS(u ‖ p) = 1 2
( KL(u ‖ p+ u
2 )
+ KL(p ‖ p+ u 2
) ) .
Both JS(u ‖ p) and J(u ‖ p) can be used as regularization functions. Following Lin (1991), the JS divergence is bounded by the J divergence:
JS(u ‖ p) ≤ 1 4 J(u ‖ p).
Thus, the J divergence penalizes the loss more strongly than the JS divergence given the same β. We preliminarily examined the use of the JS divergence but found that it is not as effective as the J divergence in our task.

heading: B Additional examples
text: Table 4 shows examples from the COVIDFACT test set where Lc̃p can successfully capture salient words.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
9
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
3 A4. Have you used AI writing assistants when working on this paper? Grammarly
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
C 3 Did you run computational experiments? 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 6.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 6.1
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 6.3
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 6.1
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Training neural models that use more meaningful features while being well-calibrated is an important direction for future work