Easy to Decide, Hard to Agree: Reducing Disagreements Between Saliency Methods
Year: 2023

Abstract:
A popular approach to unveiling the black box of neural NLP models is to leverage saliency methods, which assign scalar importance scores to each input component. A common practice for evaluating whether an interpretability method is faithful has been to use evaluation-by-agreement – if multiple methods agree on an explanation, its credibility increases. However, recent work has found that saliency methods exhibit weak rank correlations even when applied to the same model instance and advocated for alternative diagnostic methods. In our work, we demonstrate that rank correlation is sensitive to small perturbations when evaluating agreement and argue that Pearson-r could be a better-suited alternative. We further show that regularization techniques that increase faithfulness of attention explanations also increase agreement between saliency methods. By connecting our findings to instance categories based on training dynamics, we show that the agreement of saliency method explanations is very low for easy-to-learn instances. Finally, we connect the improvement in agreement across instance categories to local representation space statistics of instances, paving the way for work on analyzing which intrinsic model properties improve their predisposition to interpretability methods.1

heading: 1 Introduction
text: Following the meteoric rise of the popularity of neural NLP models during the neural revolution, they have found practical usage across a plethora of domains and tasks. However, in a number of high-stakes domains such as law (Kehl and Kessler, 2017), finance (Grath et al., 2018), and medicine (Caruana et al., 2015), the opacity of deep learning
♣Equal contribution ♥Work done while author was at TakeLab 1Our code is available at https://github.com/
mttk/unity-in-xai-court
methods needs to be addressed. In the area of explainable artificial intelligence (XAI), one of the major recent efforts is to unveil the neural black box and produce explanations for the end-user. There are various approaches to rationalizing model predictions, such as using the attention mechanism (Bahdanau et al., 2014), saliency methods (Denil et al., 2014; Bach et al., 2015; Ribeiro et al., 2016; Lundberg and Lee, 2017; Shrikumar et al., 2017; Sundararajan et al., 2017), rationale generation bydesign (Lei et al., 2016; Bastings et al., 2019; Jain et al., 2020), or self-rationalizing models (Marasovic et al., 2022). These methods have to simultaneously satisfy numerous desiderata to have practical application in high-stakes scenarios: they have to be faithful – an accurate representation of the inner reasoning process of the model, and plausible – convincing to human stakeholders.
When evaluating faithfulness in using attention as explanations, Jain and Wallace (2019) have shown that attention importance scores do not correlate well with gradient-based measures of feature importance. The authors state that although gradient-based measures of feature importance should not be taken as ground truth, one would still expect importance measures to be highly agreeable, bringing forth the agrement-as-evaluation paradigm (Abnar and Zuidema, 2020; Meister et al., 2021). While the imperfect agreement is something one could expect as interpretability methods differ in their formulation, and it is reasonable to observe differences in importance scores, subsequent work has shown that saliency methods exhibit low agreement scores even when applied to the same model instance (Neely et al., 2021). Since a single trained model instance can only have a single feature importance ranking for its decision, disagreement of saliency methods implies that at least one, if not all methods, do not produce faithful explanations – placing doubt on their practical relevance. It has been hypothesized that unfaithfulness of attention
9147
is caused by input entanglement in the hidden space (Jain and Wallace, 2019). This claim has later been experimentally verified through results showing that regularization techniques targeted to reduce entanglement significantly improve the faithfulness of attention-based explanations (Mohankumar et al., 2020; Tutek and Šnajder, 2020). While entanglement in the hidden space is clearly a problem in the case of attention explanations, where attention weights directly pertain to hidden states, we also hypothesize that representation entanglement could cause similar issues for gradient- and propagationbased explainability methods – which might not be able to adequately disentangle importance when propagating toward the inputs.
In our work, we first take a closer look at whether the rank correlation is an appropriate method for evaluating agreement and confirm that, as hypothesized in previous work, small differences in values of saliency scores significantly affect agreement scores. We argue that a linear correlation method such as Pearson-r is less sensitive to perturbations since the exact ranking order of features is not as crucial for agreement as the relative importance values, which Pearson-r adequately captures. We hypothesize that the cause of saliency method disagreements is rooted in representation entanglement and experimentally show that agreement can be significantly improved by regularization techniques such as tying (Tutek and Šnajder, 2020) and conicity (Mohankumar et al., 2020). The fact that regularization methods, which were originally aimed at improving faithfulness of attention, also improve agreement between saliency methods suggests that the two problems have the same underlying cause. Taking the analysis deeper, we apply techniques from dataset cartography (Swayamdipta et al., 2020) and show that, surprisingly, the explanations of easy-to-learn instances exhibit a lower agreement than of ambiguous instances. We further analyze how local curvature of the representation space morphs when regularization techniques are applied, paving the way for further analysis of (dis)agreements between interpretability methods.

heading: 2 Background and Related Work
text: Explainability methods come in different flavors determined by the method of computing feature importance scores. Saliency methods perform posthoc analysis of the trained black-box model by either leveraging gradient information (Denil et al.,
2014; Sundararajan et al., 2017), modifying the backpropagation rules (Bach et al., 2015; Shrikumar et al., 2017), or training a shallow interpretable model to locally approximate behavior of the blackbox model (Ribeiro et al., 2016), all with the goal of assigning scalar saliency scores to input features. Alternatively, if the analyzed model is capable of generating text, one can resort to selfrationalization by prompting the trained model to generate an explanation for its decision (Marasovic et al., 2022). In contrast to post-hoc explanations, inherently interpretable models produce explanations as part of their decision process, either by masking a proportion of input tokens and then performing prediction based on the remaining rationale (Lei et al., 2016; Bastings et al., 2019; Jain et al., 2020), or jointly performing prediction and rationale generation in cases where datasets with annotated rationales are available (Camburu et al., 2018). For some time, the attention mechanism (Bahdanau et al., 2014) has also been considered inherently interpretable. However, the jury is still out on whether such explanations can be considered faithful (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Tutek and Šnajder, 2020; Bastings and Filippova, 2020).
Faithfulness is one of the most important desiderata of explanation methods (Jacovi and Goldberg, 2020) – faithful explanations are those that are true to the inner decision-making process of the model. Approaches to evaluating faithfulness rely on measuring how the confidence of the model changes when inputs are perturbed (Kindermans et al., 2019) or completely dropped from the model (Li et al., 2016; Serrano and Smith, 2019). However, perturbations to input often result in corrupted instances that fall off the data manifold and appear nonsensical to humans (Feng et al., 2018) or fail to identify all salient tokens properly (ROAR; Hooker et al., 2019) – raising questions about the validity of perturbation-based evaluation. Recursive ROAR (Madsen et al., 2022) alleviates the issues of its predecessor at the cost of requiring many prohibitively expensive retraining steps, further motivating us to seek efficient solutions which do not require retraining the model multiple times. Another option is to leverage the evaluation-by-agreement (Jain and Wallace, 2019) paradigm, which states that an interpretability method should be highly agreeable with other methods to be considered faithful. However, since empirical evidence has shown that
saliency methods exhibit poor agreement between their explanations (Neely et al., 2021), Atanasova et al. (2020) recommend practitioners consider alternative methods for evaluating the quality of interpretability methods, such as diagnostic tests. Finally, methods such as data staining (Sippy et al., 2020) and lexical shortcuts (Bastings et al., 2022) artificially introduce tokens that act as triggers for certain classes – creating a ground truth for faithfulness which can be used as a comparison. Nevertheless, such methods have a certain drawback in that they only offer the ground truth importance of a few artificially inserted tokens, but offer no insight regarding the relative importance of the remainder of the input. Each of the aforementioned methods for estimating faithfulness of interpretability methods has its drawbacks (Jacovi and Goldberg, 2020), and we argue each should be taken in conjunction with others to increase the credibility of their collective verdict.

heading: 3 Preliminaries
text: In this section, we delineate our experimental setup, detailing the considered datasets, models, their training procedure, the saliency methods which we use to interpret the decisions of the models, and the regularization techniques we use to improve agreement between saliency methods.

heading: 3.1 Datasets
text: Leaning on the work of Neely et al. (2021), which motivated us to explore the valley of explainability, we aim to investigate the protruding problem of low agreement between saliency methods. We investigate three different types of single-sequence binary classification tasks on a total of four datasets. In particular, we evaluate sentiment classification on the movie reviews (IMDB; Maas et al., 2011) and the Stanford Sentiment Treebank (SST-2; Socher et al., 2013) datasets, using the same data splits as Jain and Wallace (2019). We include two more tasks, examining the subjectivity dataset (SUBJ; Pang and Lee, 2004), which classifies movie snippets into subjective or objective, and question type classification (TREC; Li and Roth, 2002). To frame the TREC task as binary classification, we select only the examples labeled with the two most frequent classes (ENTY – entities, HUM – human beings) and discard the rest.

heading: 3.2 Models
text: For comparability, we opt for the same models as Neely et al. (2021). Specifically, we employ the Bi-LSTM with additive self-attention (JWA; Jain and Wallace, 2019). We initialize word representations for the JWA model to 300-d GloVe embeddings (Pennington et al., 2014). We also employ a representative model from the Transformer family (Vaswani et al., 2017) in DistilBERT (DBERT; Sanh et al., 2019).
Both models work similarly: the input sequence of tokens {x1, . . . , xT } is first embedded {e1, . . . , eT } and then contextualized {h1, . . . , hT } by virtue of an LSTM network or a Transformer. The sequence of contextualized hidden states is then aggregated to a sequence representation h, which is then fed as input to a decoder network.

heading: 3.3 Explainability Methods
text: We make use of ready-made explainability methods from the propagation- and gradient-based families used by Neely et al. (2021): Deep-LIFT (Shrikumar et al., 2017), Integrated Gradients (Int-Grad; Sundararajan et al., 2017) and their Shapley variants (Lundberg and Lee, 2017), Deep-SHAP and GradSHAP.2 Since we evaluate agreement on the entire test set instead of an instance subset (Neely et al., 2021), we exclude LIME (Ribeiro et al., 2016) from the comparison as it is not computationally feasible to train the surrogate model for all test instances across all training setups.
Each saliency method produces a set of importance scores for each input (sub)word token. When evaluating the agreement between different saliency methods for a single trained model, one would expect the importance scores for the same input instance to be similar, as the same set of parameters should produce a unique and consistent importance ranking of input tokens.

heading: 3.4 Regularization Methods
text: As alluded to earlier, we suspect one cause of disagreement between saliency method explanations to be rooted in representation entanglement. To counteract this issue, we employ two regularization schemes that have been shown to improve the faithfulness of the attention mechanism as a method
2We use implementations of explainability methods from the Captum framework: https://github.com/ pytorch/captum
of interpretability: CONICITY (Mohankumar et al., 2020) and TYING (Tutek and Šnajder, 2020). Both of these methods address what we believe is the same underlying issue in recurrent models – the fact that hidden representations ht are often very similar to each other, indicating that they act more as a sequence representation rather than a contextualization of the corresponding input token xt.
Each regularization method tackles this problem in a different manner. CONICITY aims to increase the angle between each hidden representation and the mean of the hidden representations of a single instance. The authors first define the alignment to mean (ATM) for each hidden representation as the cosine similarity of that representation to the average representation:
ATM(hi,H) = cosine(hi, 1
T
T∑
j=1
hj) (1)
where H = {h1, . . . , hT } is the set of hidden representations for an instance of length T . Conicity is then defined as the average ATM for all hidden states hi ∈ H:
conicity(H) = 1
T
T∑
i=1
ATM(hi, H) (2)
A conicity value implies that all hidden representations exist in a narrow cone and have high similarity – to counteract this unwanted effect, during training, we minimize this regularization term weighted by λcon along with the binary cross entropy loss.
Similarly, TYING also aims to incentivize differences between hidden states by enforcing them to “stay true to their word” through minimizing the L2 norm of the difference between each hidden state ht and the corresponding input embedding et = embed(xt):
tying(H,E) = 1
T
T∑
i=1
∥hi − ei∥22 (3)
where E = {e1, . . . , eT } is the sequence of embedded tokens. During training, we minimize this regularization term weighted by λtying along with the binary cross entropy loss.
By penalizing the difference between hidden representations and input embedding, one achieves two goals: (1) the embedding and hidden state representation spaces become better aligned, and (2) each hidden representation comes closer to its input
embedding. The latter enforces hidden states to differ from each other: because different embeddings represent the semantics of different tokens, their representations should also differ, and this effect is then also evident in the hidden representations.
Although both works introduced other methods of enforcing differences between hidden states, namely orthogonal-LSTM and masked language modeling as an auxiliary task, we opt for CONICITY and TYING as they were both shown to be more efficient and more stable in practice.

heading: 4 Improving Agreement
text: In this section, we present two modifications of the existing evaluation-by-agreement procedure: (1) complementing rank-correlation with a linear correlation measure more robust to rank changes caused by small differences in importance weights, and (2) regularizing the models with the goal of reducing entanglement in the hidden space, and as a consequence, improving agreement.

heading: 4.1 Choice of Correlation Metric
text: Previous work (Jain and Wallace, 2019; Neely et al., 2021) has evaluated the agreement between two explainability methods by using rank-correlation as measured by Kendall-τ (Kendall, 1938). Although Kendall-τ is generally more robust than Spearman’s rank correlation, i.e., it has smaller gross-error sensitivity (Croux and Dehon, 2010), we still face difficulties when using Kendall-τ for evaluating agreement. As Jain and Wallace (2019) also note, perturbations in ranks assigned to tokens in the tail of the saliency distribution have a large influence on the agreement score. In addition, rankings are also unstable when saliency scores for the most relevant tokens are close to one another. In Figure 1, we illustrate the deficiencies of using rank correlation on a toy example of explaining sentiment classification. While saliency scores attributed to tokens differ slightly, the differences in rank order are significant, lowering agreement according to Kendall-τ due to the discretization of raw saliency scores when converted into ranks. We believe that a better approach to approximating agreement is to use a linear correlation metric such as Pearson’s r, as it evaluates whether both saliency methods assign similar importance scores to the same tokens – which is a more robust setup if we assume small amounts of noise in importance attribution between different methods.
We now investigate how Pearson-r (pr) compares to Kendall-τ (kτ ) when evaluating agreement. In Table 1 we compare agreement scores produced by pr and kτ across all datasets for JWA and DBERT, respectively. For JWA, saliency methods display agreement greater than 0.5 only in 8/24 cases according to kτ and in 16/24 cases as per pr. For DBERT, these figures are 0/24 and 5/24, respectively. While the overall agreement is subpar, we posit that kτ further exacerbates the evaluation process. We believe this is caused by tokens with approximately equal relative importance assigned slightly different rankings by explainability methods, which kτ harshly penalizes. To address this,
we recommend using the Pearson correlation coefficient as an additional measure in the evaluation of agreement, as it is more robust to rank changes caused by small differences in saliency scores.

heading: 4.2 Regularizing Models
text: Our next goal is to improve agreement between saliency methods through intervention in the training procedure, namely by applying regularization to promote disentanglement in the hidden space. In Table 2 we report correlation scores on the test splits of all datasets for regularized models (CONICITY, TYING) and their unregularized variants (BASE). We notice that both regularization techniques have a positive effect on agreement across both correlation metrics, indicating that regularization techniques alleviate a deeper issue that also affects the interpretability of attention weights. In Table 3 we report F1 scores on the test set for the regularized and unregularized models with the best performance on the validation split. We observe that regularized models generally perform comparably well to unregularized ones on downstream tasks, indicating that the improvement in the agreement does not come at a cost for downstream performance. When selecting regularized models, we choose ones with the strongest regularization scale hyperparameter that performs within 3 F1 points on the validation set compared to the unregularized model (cf. details in Appendix A.2).

heading: 5 The Cartography of Agreement
text: We have shown that by using a more appropriate correlation measure and applying regularization, the agreement of saliency methods increases sig-
nificantly. In this section, we are interested in finding out the cause of increased agreement obtained through applying regularization – are there certain instance groups in the dataset that benefit the most, and if so, what changes in the representation space resulted in the increased agreement? We leverage methods from dataset cartography (Swayamdipta et al., 2020) to distribute instances into easy-tolearn, hard-to-learn, and ambiguous categories based on their prediction confidence and variability. Concretely, if an instance exhibits low prediction variability and high prediction confidence between epochs, this implies that the model can quickly and
accurately classify those instances, making them easy-to-learn. Instances that also exhibit low variability but low prediction confidence, align with the idea that the model is consistently unable to correctly classify them, making them hard-to-learn. Finally, instances that exhibit high variability and confidence close to the decision threshold indicate that the model is likely often changing its prediction between class labels for those instances, making them ambiguous. Since ambiguous instances are characterized by confidence near the prediction threshold, Swayamdipta et al. (2020) complement variability and confidence with another statistic introduced by Chang et al. (2017), namely closeness, defined as ci = p(i) · (1 − p(i)), where p(i) is the average correct class probability of instance x(i) across all training epochs. A high closeness value denotes that the instance is consistently near the decision boundary and, thus, is a good indicator of ambiguity within the model.
Intuitively, one would expect high agreement between saliency methods on instances that are easy to learn and low agreement otherwise. However, we find the converse is true when computing how agreement distributes across instance groups. In unregularized models, we observe that easy-to-learn instances exhibit low average agreement, while ambiguous instances have a high average agreement. In Table 4, we report average agreement scores across all pairs of saliency methods on representative samples from each cartography group.3 We observe a clear distinction in agreement for both the base and regularized models, which is higher for ambiguous instances when compared to easy- and hard-to-learn instance groups. Furthermore, we can observe a consistently high increase in agreement when the models are regularized across all instance groups for all datasets, indicating that regularization techniques reduce representation entanglement.
One might wonder how the increase in agreement distributes across instances and dataset cartography attributes. In Figure 2, we visualize how the relationship between agreement and cartography attributes changes when the models are regularized. We observe that for the JWA model, all datasets ex-
3We select representative samples for each group through the relative frequency of their correct classification. If out of 5 epochs, an instance was correctly classified 5 times, it is representative of the easy-to-learn category. If it was correctly classified 0 times, it is representative of the hard-to-learn category, and if the number of correct classifications is 2 or 3, it is representative of the ambiguous category.
hibit a consistent and significant increase in agreement. Furthermore, we notice that for the DBERT model, apart from increasing the agreement, regularization reduces the confidence of the model predictions and increases variability – indicating that it reduces the known problem of overconfidence present in pre-trained language models.

heading: 5.1 The Curvature of Agreement
text: To better understand the cause of this distinction between various feature groups, we now analyze local curvature and density in the representation space. We are interested in: (1) how densely the instances are distributed in the representation space across cartography categories and (2) whether the local space around an instance is sharp or smooth. For both models and all instances, we obtain sequence representations h used as inputs to the decoder. We estimate instance density as the average distance to the nearest instance in the dataset. We estimate local smoothness around an instance representation as the L2 norm of the gradient of the hidden representation with respect to the input embeddings. If the gradient norm is high, the local space is sharp and minor perturbations can have a large effect on the prediction probability.
In Table 5, we report correlations between each of these two statistics and dataset cartography attributes. We observe that for the unregularized model, there is a significant negative correlation between confidence and both gradient norm and minimum distance to the nearest example, indicating that the local space around easy instances is smooth
and densely populated. On the other hand, there is a high positive correlation between both closeness and variability and both gradient norm and minimum distance to the nearest example – indicating that the local space around ambiguous instances is sharp and sparsely populated. When we turn our attention to the regularized model, we observe that the correlation between the gradient norm and any of the cartography attributes vanishes, while the correlations between distance and the attributes are reduced in absolute value and their sign is flipped.
From these observations, we hypothesize that the cause of low agreement on easy-to-learn instances is the multitude of possible explanations as to why such an instance should be correctly classified. From the viewpoint of plausibility, this hypothesis is in line with the Rashomon effect (Breiman, 2001) which is about there often existing a multitude of adequate descriptions that end up with the same error rate, or in our case, prediction probability – however it should not apply to faithfulness, as a single model instance should adhere to a single explanation. However, due to a plethora of corroborating evidence for easy-tolearn instances, the representation space around them is smooth to such an extent that perturbations do not significantly affect the prediction probability, which in turn adversely affects gradient- and propagation-based explanation methods. The converse is true for ambiguous instances, where we hypothesize the model observes evidence for both classes and is unable to reach a confident decision. However, this difficulty in reaching a decision also causes saliency methods to have a precise definition of what the evidence is – as the local curvature is sharp, and any minor perturbation could significantly affect prediction probability. We believe that local curvature statistics could be used as a metric for measuring whether a trained model is better suited to analysis through explainability methods.

heading: 6 Conclusion
text: We analyzed two prototypical models from different families in JWA and DBERT with the goal of finding out the cause of low agreement between saliency method interpretations. We first take a closer look at Kendall-τ , the previously used rankorder correlation metric, and demonstrate that it can be prone to exhibiting high differences in agreement for small perturbations in importance scores. To account for this, when analyzing agreement be-
tween saliency methods, we propose researchers include a linear correlation metric such as Pearson-r, which is robust to small importance perturbations. Taking a step further, we applied two regularization techniques, TYING and CONICITY, originally aimed at increasing faithfulness of attention explanations, with the hypothesis that the issue underpinning disagreements and unfaithfulness is the same – representation entanglement in the hidden space. We showed that regularization consistently and significantly improves agreement scores across all models and datasets with a minimal penalty for classification performance. Having demonstrated that it is possible to improve upon the low agreement scores, we attempted to offer intuition on
which instance categories saliency methods agree the least and show that surprisingly, easy-to-learn instances are hard to agree on. Lastly, we offered insights into how the representation space morphs when regularization is applied and linked these findings with dataset cartography categories, paving the way for further work on understanding what properties of neural models affect interpretability.
Limitations
Our work has a number of important limitations that affect the conclusions we can draw from it. First and foremost, evaluating the faithfulness of model interpretations is problematic as we do not have ground truth annotations for token importances. Thus, when applying the agreement-asevaluation paradigm, we implicitly assume that most saliency methods are close to the truth – an assumption that we cannot verify. However, every method of evaluating faithfulness has its own downsides. Token and representation erasure runs the risk of drawing conclusions from corrupted inputs that fall off the data manifold. We argue that while agreement-as-evaluation is far from an ideal way of evaluating faithfulness, it still increases credibility when used along with other techniques.
Secondly, our work is limited both with respect to the datasets and models considered. Specifically, we only evaluate one Transformer-based model from the masked language modeling family, and it is entirely possible that the findings do not generalize to models pre-trained on different tasks. Also, we only consider single sequence classification datasets – mainly due to the fact that the issues with the faithfulness of attention were most prevalent in those setups, which we assumed would be the same for agreement due to the same hypothesized underlying issue. We believe that tasks that require retention of token-level information in hidden states, such as sequence labeling and machine translation, would exhibit higher agreement overall, even without intervention through regularization. We leave this analysis for future work.

heading: A Reproducibility
text: A.1 Experimental Results
A.1.1 Setup
For both JWA and DBERT, we use the same preprocessing pipeline on all four datasets. First, we filter out instances with fewer than three tokens to achieve stable agreement evaluation.4 Next, we lowercase the tokens, remove non-alphanumeric tokens, and truncate the sequence to 200 tokens if the sequence length exceeds this threshold. We set the maximum vocabulary size to 20k for models which do not leverage subword vocabularies.
A.1.2 Validation set performance
We report the validation set performance in Table 6.
A.1.3 Computing infrastructure
We conducted our experiments on 2× AMD Ryzen Threadripper 3970X 32-Core Processors and 2× NVIDIA GeForce RTX 3090 GPUs with 24GB of RAM. We used PyTorch version 1.9.0 and CUDA 11.4.
A.1.4 Average runtime
Table 9 shows the average experiment runtime for each model across the datasets we used.
A.1.5 Number of parameters
The JWA and DBERT models that we used contained 1, 714, 951 and 66, 954, 241 trainable parameters, respectively.
4If a sequence consists of only two tokens, rank-correlation with Kendall-τ will either result in a perfect match, or completely different observations as swapping the two ranks leads to an inverse ranking.
A.2 Hyperparameter search We used the following parameter grids for JWA: [10−1, 10−2, 10−3, 10−4, 10−5, 10−6] for learning rate, and [50, 100, 150, 200] for the hidden state dimension. We yield best average results on validation sets across all datasets when the learning rate is set to 10−3 and the hidden size is set to 150. For DBERT, we find that the most robust initial learning rate on the four datasets is 2 × 10−5, among the options we explored [5 × 10−4, 10−4, 10−5, 2 × 10−5, 5 × 10−5, 10−6]. Additionally, we clip the gradients for both models such that the gradient norm ≤ 1. We use the Adam (Kingma and Ba, 2015) optimizer for JWA and AdamW (Loshchilov and Hutter, 2017) for DBERT. We run both models for 5 epochs and repeat the experiments 5 times with different seeds: [1, 2, 3, 4, 5].
For regularization methods, we conducted a grid search with parameter grid [0.1, 0.3, 0.5, 1, 5, 10] for CONICITY and [0.1, 0.3, 0.5, 1, 5, 10, 20] for TYING. We select the models with the strongest regularization scale, which is within 3 F1 points from the unregularized model. Table 8 shows the selected values for each model across all datasets.
A.3 Dataset statistics We report the number of instances per split for each dataset in Table 10. We note that all of the datasets we used contain predominantly texts in English.

heading: B Additional Experiments
text: We show the full version of local curvature statistics in Table 11 (without averaging over datasets). In Figures 3 to 10 we plot correlation scores (kτ and pr) with standard deviation on the test splits. We include the results for all datasets across training epochs for regularized models (CONICITY, TYING) when compared to their unregularized, BASE variants.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: We leave this analysis for future work