Discovering Highly Influential Shortcut Reasoning: An Automated Template-Free Approach
Authors: Daichi Haraguchi
Kiyoaki Shirai
Naoya Inoue
Natthawut Kertkeidkachorn
Venue: EMNLP 2023 Findings (2023)
Keywords: Shortcut Reasoning
Inference
Robustness
TL;DR: Automatically discovereing irrationale reasoning process of models with minimal predefinition.

Abstract: Shortcut reasoning is an irrational process of inference, which degrades the robustness of an NLP model.
While a number of previous work has tackled the identification of shortcut reasoning, there are still two major limitations: (i) a method for quantifying the severity of the discovered shortcut reasoning is not provided; (ii) certain types of shortcut reasoning may be missed.
To address these issues, we propose a novel method for identifying shortcut reasoning.
The proposed method quantifies the severity of the shortcut reasoning by leveraging out-of-distribution data and does not make any assumptions about the type of tokens triggering the shortcut reasoning.
Our experiments on Natural Language Inference and Sentiment Analysis demonstrate that our framework successfully discovers known and unknown shortcut reasoning in the previous work.

Discovering Highly Influential Shortcut Reasoning:
An Automated Template-Free Approach
Daichi Haraguchi', Kiyoaki Shirai', Naoya Inoue
Natthawut Kertkeidkaccorm
'Japan Advanced Institute of Science and Technology
RIKEN {s21 10137, kshikksiiii......k k naoya-i, nat))eiiatttttjist....
Abstract
(IID) data (i.e., data from an identical distribution as the training data) (Geirhos et al., 2020). In the aforementioned example, the reasoning for movic reviews would not be valid for OOD data (e.g.. news articles) because the sentiment of news arti- cles containing Spielberg could be arbitrary. Although many studies have explored the detec- tion of spurious correlations or shortcut reasoning (Ribeiro et al., 2020; Pezeshkpour et al., 2022; Han et al., 2020), several challenges persist. Wang et al. (2022a) propose a state-of-the- method for dis- covering shortcut reasoning, which implements automated framework to discover shorteuts without predefining shorteut templates. Still, their approach suffers from two major limitations.
Shortcut reasoming an irrational process of inference, which degrades the robustness of an NLP model. While number of previous work has tackled the identification shortcut rea- soning, there still two major limitations: (i) a method for quantifying the severity of the discovered shortcut reasoming not provided; (ii) certain types of shortcut reasoning may be missed. To address these issues, we propose novel method for identifying shorteut rea- soning. The proposed method quantifies the severity the shortcut reasoning by leveraging data and does not make any assumptions about the type of tokens trigger- ing the shortcut reasoning. Our experiments Natural Language Inference and Sentiment Analysis demonstrate that our framework suc- cessfully discovers known and unknown short- cut reasoning in the previous work.!
Firstly, their framework lacks method for quan- tifying the severity of the discovered shortcut rea- soning OOD data. Even if the shortcuts are iden- tified, we do not have necessarily be concerned about them as long as they have little negative im- pact on the model's robustness. Secondly, their approach assumes that genuine tokens, useful to- kens for predicting labels across different datasets (e.g., "good", "bad"), do not lead to shortcut rea- soning. While this assumption seems reasonable. Joshi et al. (2022) argue that such tokens are still prevalent among spurious correlations. This is be- cause such tokens are indeed necessary to predict the label, but these tokens alone may not provide sufficient information to accurately predict labels. For example, a genuine token "good" in a sentence This movie is not good can be spurious, since "good" is a necessary but insufficient token for determining its sentiment label. Therefore, genuine tokens not be ignored when identifying shorteut reasoning.
1 Introduction
While Transformer-based large language models have remarkably improved various NLP tasks, the issue of shortcut reasoning has been identified as a severe problem (Schlegel et al., 2020; Wang et al., 2022b; Ho et al., 2022). Shortcut reasoning usu- ally refers to the irrational inference of model, which is derived from spurious correlations in the training data (Gururangan et al., 2018; Poliak et al., 2018; McCoy al., 2019). For example, senti- ment analysis models could learn to classify any sentences containing the word Spielberg into POSI- TIVE, given training dataset with many positive movie reviews containing Spielberg (e.g., Spielberg is a great director!).
Shortcut reasoning makes models brittle against Out of Distribution (OOD) data (i.e., data from different distribution from the training data) com- pared to Independent and Identically Distributed
To address these problems, we propose new method for discovering shortcut reasoning. Our contributions can be summarized as follows:
We present an automated method for identify- ing shortcut reasoning.
'Our cnde is available at https: /twpttggttttt.ut com/ homoscribens/ shor wwwpaansssoottutteeaeeeee git

---

Input
Model: f
Input Reduction
Sprelberg great! yn
Output
D LID
Triggen(c) Label())
0.77
-7.2
["Spielberg]] POSITIVE
["Spielberg'] - POSITIVE
Generality (g) How prodiction
Delta (A) dagradks accursy??
Inference Pattern (p)
Spielberg
directed
y: o.o.d. deta: DOOD
Inference
Generality Calculation (Sec..3)
Shortcut Reassoning Identification (Sec.
Extraction (Sec 1/2.2)
Figure I: Our method to discover shortcut reasoning with steps.
By applying the less subjective definition of shortcut reasoning (Geirhos al., 2020), our method does not require laborious human eval- uation for detected shortcuts.
2.1 Inference Pattern
An inference pattern a crucial pattern that acti- vates a label during a model's inference process. Given a target model f, we formally define an in- ference pattern p as follows:
Our method quantifies the severity of shortcut reasoning by leveraging OOD data and does not make any assumptions about the type of tokens triggering the shorteut reasoning.
p def 4
(1)
where denotes a trigger that induces certain label, and / is the induced label.
We demonstrate that our method successfully discovers previously unknown shortcut rea- soning as well as ones reported in previous research.
Pezeshkpour et al. (2022) classified spurious correlations into two types: (i) granular features. namely discrete units such as an individual token "Spielberg", and (ii) abstract features, namely high- level patterns such as lexical overlap.
2 Discovering Shortcut Reasoning
This paper focuses on granular features and leaves the detection of shortcut reasoning with ab- stract features for future work. We thus adopt the following definition an inference pattern:
Fig. I shows the overall procedure of the proposed method. Given (i) target model J. (ii) IID data Dun, and (iii) OOD data DooD inputs, shortcut reasoning is extracted as an output. The procedure consists of the following three steps.
def
w 41,
(2)
Step extracts inference patterns, an abstract representation that characterizes the inferential pro- cess of given model ($2.1). To extract inference patterns, we use input reduction, an algorithm that automatically derives the inference patterns ($2.2). Step 2 estimates the generality of the extracted in- ference patterns. Generality is a measure of the strength of an inference pattern, indicating its de- gree of regularity ($2.3). Step 3 identifies shortcut reasoning. We automatically determine whether inference pattern exhibits shorteut reasoning without human intervention, comparing the effec- tiveness of inference patterns in DIID with that in DOOD and leveraging the estimated generality as a proxy for the severity of the identified shortcut reasoning ($2.4).
where is sequence of tokens [w], W2, w.]. Although we limit ourselves granular features. this definition still enables us to detect shorteut reasoning with a variety of forms, such as combina- tions of tokens as well as a single token. For exam- ple, a sentiment analysis model f may have infer- ence patterns such as ["not", "bad"] + NEUTRAL or ("Spielberg") - POSITIVE (possibly shortcut reasoning).
2.2 Extracting Inference Patterns
Given a target model f and an IID dataset PIID [1 N we extract set C of inference pat- terns by applying input reduction (IR) to each input Pe.

---

INPUT:
PREDICTION. 9
p: Wi -i C. we collect set EooD of examples from DOOD such that the input contains W₃. For example, given p: = ["Spielberg"] -> POSITIVE, FOOD may contain sentences such as grew up with Steven Spielberg's 's films. His films are always great!! and Spielberg is overrated. We then estimate the generality g of the inference pattern p; follows:
don't like this movic
NEGATIVS
don't mavic [MASK]
NESAVTIVE
[MASK] dor't like this movie [MASK]
NBOKTIVS
[MASK] don't like [MASK] [MASK] [MASK]
NECATIVE
[MASK] [MASK] like [MASK] [MASK] [MАSK]]
POSITIVE
["don't", "like"] NEGATIVE
(p,) def Ex'Cooo (w:) 1 [f(x') =
100. (3)
EooD(w;)]
Figure 2: An example of inference pattern extracted by Input Reduction.
Intuitively, g(p,) explains how much the inference pattern is dominant on the OOD dataset.
IR gradually reduces the number of tokens in I; by masking each token one by one, incrementally increasing the number of masked tokens after each step. In each step. IR feeds the masked z; into and obtains predicted label 5i. IR stops when the predicted label J/ flips. As the final step, IR extracts a sequence w, of unmasked tokens in oú and ÿi that is the last predicted label before the prediction flips
2,4 Identifying Shortcut Reasoning
In this section, we define shorteut reasoning and describe the method for its detection. According Geirhos et al. (2020), shorteut reasoning satisfies both of the following conditions: (i) performs well on DID, and (ii) underperforms on DOOD-
We upply these conditions to inference patterns. Given p; = -> l extracted from DuD by IR, the condition (i) is satisfied when p; works well on DnD- In other words, when the model per- forms well on IID examples that contain W; (i.e.. FuD((;)) Thus, we evaluate the performance of each inference pattern using EnD(w:). Specifi- cally, we define new metric iid_ tid_acc;, which com- putes
as an inference pattern, namely W. 4 9i-
To prioritize which tokens should be masked, we employ Integrated Gradient (Sundararajan et al., 2017), which computes the importance of each token for prediction. IR sorts the tokens in x; ac- cording to their IG score and then incrementally applies masks to the tokens in ascending order of their rank the sorted sequence. For each mask applied, IR leaves the corresponding token masked and proceeds to the next token in the sequence.
EreEmp(w)) 1[f(z) = 4 44 yi
[f(x) 4. x 100. (4)
ExcEmp(wa)
Fig. shows example of the extracting pro- cess by IR. The tokens are replaced with [MASK] in the ascending order of the IG scores (values in orange in the figure). When the predicted label is flipped from NEGATIVE to POSITIVE, the re- mained tokens at the previous step and NEGATIVE label are extracted as the inference pattern ('don't", "like"I + NEGATIVE. See Appendix A for the pseudocode of IR.
This metric counts the right prediction for inputs that contain trigger (w;).
The condition (ii) is satisfied when p3 does not deliver accurate results on DOOD, i.e., when the model operates poorly on OOD inputs that contain w, (i.c., EOOD (w:)). As a metric to evaluate how much the p; underperforms over E(w:), we define A as follows:
The IR- extraction algorithm ensures that the trigger w; the extracted inference patterns is concise and not redundant.
A def F1 (w:), F1 f). (5)
This metric compares the Fl score on EOOD(w:) to that DOOD, employed baseline for com- parison.
2.3 Calculating Generality
In order to verify the validity of an inference pattern in C' as universal pattern, we assess its generality on DOoD. This measurement determines the de- gree to which the pattern exhibits regularity in the OOD dataset.
To sum up, shortcut reasoning is defined as f4 = w; 1 such that gP:)) is sufficiently large, iid acc acc is large enough and A; is small enough. The set of shortcut reasoning is defined as
To estimate the generality of inference pattern
(prEC C g(p:) > Yι, jid_acc; > A2, A < As). (6)

---

Å1 X2, and A are the pre-defined thresholds. Note that A and A have to be an above-chance score and less than at least, respectively. This defini- tion enables us to automatically identify shortcut reasoning that has a substantial impact on OOD, unlike previous studies (Pezeshkpour et al., 2022; Wang et al., 2022a).
shortcut reasoning is discovered in the training or the test split of IID dataset (i.c., input for IR).
NLI For OOD data, the model performed 77.8 of F1 f). "Is" denotes a separation token between premise and hypothesis. For OOD data, the model performed 77.8 of F1(DOOD: We observed that most of identified as shortcut rea- soning belonged to the hypothesis, while only small proportion present in the premises. This observation suggests that the model heavily relies on the hypothesis to predict labels, corroborating the findings of Poliak et al. (2018), Furthermore, we found that negation representation in t, such as "not" or "never", often led the model to predict CONTRADICTION. This phenomenon manifests itself even when the gold label indicates otherwise. as indicated by the values of A at the first and fourth p in the Table 1. This finding aligns with the results reported by Gururangan et al. (2018). With this observed consistency with previous work, our method seems to be effective at accurately identi- fying shortcuts.
3 Experiments
3.1 Setup
One straightforward approach for assessing our method is to annotate NLP models with their ground-truth shortcut reasoning. However, recent NLP models are known to be hard to interpret, which makes it difficult to create such reference dataset. We thus resort to existing datasets for Natural Language Inference (NLI) and Sentiment Analysis (SA) that have been shown to contain spu- rious features and check if our inference patterns can reveal such features (and unknown ones).
Datasets For NLI, we adopt MNLI (Williams et al., 2018) as DnD and ANLI (Nie et al., 2020) DOOD. ANLI is an NLI dataset based on MNLI, but adversarially redesigned, which makes it harder answer. For SA, we apply Sentiment subset in Tweeteval (Barbieri et al., 2020) as DIID and MARC (Multilingual Amazon Reviews Corpus) (Keung et al., 2020) as DOOD- For all OOD datasets, we use training split of each. See Ap- pendix B.1 for the dataset details. Note that all the datasets are trinary classifications, so the chance accuracy is 0.33.
SA The model showed 60.3 points at F1 FI We found that sentiment words, such as "worst" "Excellent", emerged in almost all t that were classified as shortcut reasoning. Further analysis showed that reviews with neutral labels in MARC frequently contained both positive and negative sentiments (e.g., / hate the wrapping, but works pretty well.). Considering a sufficient number of with small A are annotated with neutral label for the original input, we estimate that the model relies on one among multiple sentiments in the input and ignores the rest. Therefore, it is possible to say that these inference patterns are shortcuts, whose t are necessary but insufficient.
Models We apply our method to RoBERTa (Liu et al., 2019) fine-tuned with the DID mentioned above, available at Hugging Face Hub. See Ap- pendix B.2 for the details.
Configuration The inference patterns of a model are obtained by learning training data. Thus, aside from test (or validation) sets, we extract C from the training set of Dun, expecting to better simulate the model's reasoning process. In addition, we randomly choose 1,000 examples as input IR considering runtime. We set hyperparameters \ = 50, A2 = 70 and Å3 = -0.05. To reliably obtain ( we filter out p; with Eood < 100 from C.
Train/Test
No significant difference was ob- served between the train and test experiments. Al- though the extracted shortcut reasoning from the experiments differed, they were essentially similar in terms of their characteristies (such as negations in NLI or sentiment words in SA).
Unknown Shorteut Reasoning In the NLI ex- periment, it is interesting note that we revealed several previously unknown shorteut reasoning. such as ["soon"] - NEUTRAL and ["is", "always"] + NEUTRAL. Both have sufficiently small A, and large jid_acc and g to be considered as p.
3.2 Results
We show samples of the results in Table 1. We se- lected representative P. which have large g, jid_ _acc, and A). The column train/test denotes whether

---

p
9
iid acc
A
[wi)(
train/test
NLI
("/s", "never" }-> CONTRADICTION
80.3
99.3
-9.0
1515
VIV
["/"", "soon" | NEUTRAL
64.1
100.0
-6.5
142
VIV
[*/s", "is", "always"" + NEUTRAL
60.8
96.6
-5.2
102
V1-
[*/",, "not" 1 CONTRADICTION
55.0
94.5
-55.6
8708
VIV
SA
["worst" 1-+ NEGATIVE
97.5
81.7
25.44
158
V1-
["Excellent") + POSITIVE
96.2
100.0
-7.2
184
-14
["Perfect" | POSITIVE
96.0
88.6
-12.9
324
IV
["poor"] J- NEGATIVE
87.1
76.0
-12.9
458
V1-
Table I: Samples of identified shortcut reasoning on NLI (above) and SA (bottom). We show several interesting results and ş that holds large and _id_ace and small A.
4
Related work
5 Conclusion
We introduced method to automatically discover shortcut reasoning. With minimal predefinition, our method successfully identified known and pre- viously unknown examples of shortcut reasoning. For future research, we plan to adapt our method to large language models and other tasks such as machine reading comprehension. Overall, we hope that our study provides a promising approach to- wards understanding the behavior of deep learning models and improving their trustworthiness.
Numerous studies have tackled the problem of de- tecting spurious correlations or shortcut reasoning (Ribeiro et al., 2020; Han et al., 2020). One major limitation of earlier studies is that they predefine specific format structure for the shortcuts (e.g., single token or predefined of tokens). This can hinder the discovery new and unexplored short- cuts, which may be manifested in diverse forms.
Recently, Pezeshkpour et al. (2022) address this problem by combining multiple interpretability techniques, such as influence function (Koh and Liang, 2017) and feature attribution methods (e.g.. Integrated Gradient). However, they rely on human assessment identify shorteut reasoning, which can result in misjudgment between rational and ir- rational reasoning. Besides, human evaluation laborious and mon-connuming.
Limitations
Firstly, we have yet to develop evaluation pro- cess validate the discovered shortcut reasoning. Even though we indicate the metrics or measure- ment of shortcut reasoning, knowing the actual reasoning process is impossible if we use black box models. Unfortunately, this problem would require significant effort to be solved.
Wang et al. (2022a) solve this issue by automati- cally identifying genuine tokens, important tokens that appear across different datasets, and spurious tokens, important tokens that appear only in an in- domain dataset. Still, discussed in $1, there is major limitation with their approach in that they fail consider the influence of the identified short- cut reasoning on OOD data. Our work attempts to address this issue by estimating the generality of inference patterns. Besides, our definition of shortcut reasoning aligns well with more practical scenarios. While they rely a subjective defini- tion of shortcut reasoning (i.c., whether reasoning is irrational from humans' point of view), our work targets shorteut reasoning that performs well IID but underperforms on OOD (Geirhos al., 2020), namely the one that clearly hurts the robustness of NLP models by definition.
Secondly, as our method not compatible with abstract inference patterns, it cannot cover all kinds of shortcut reasoning other than the granular one.
Thirdly, preparing two datasets, i.e., IID and OOD, is challenging for low-resouree languages some tasks. This problem limits the further studies or application of this method. Fortunately, now that we can access large language models that have sur- prising linguistic capabilities and are well-aligned with the user's instruction. The generated exam- ples by LLMs have a certain distribution which can be treated as OOD for target models, or we can prompt them to generate examples with specific distribution.
The fourth limitation is about IR. If the predic- tion for the masked input does not flip during the

---

reduction, then we alternatively output the last to- ken left in the input. Therefore, in some cases, we cannot guarantee that the extracted inference pattern is genuine.
Pang Wei Koh and Percy Liang, 2017. Understanding black- box predictions via influence functions. In Proceedings of the 34th International Conference Machine Learning, page 1885-1894.
Finally, input reduction can be utilized only when MASK token is available on the input model.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pre- training approach, Computing Research Repository, arXiv:1907.1192.
References
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuris- tics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguisties.
Francesco Barbieri. Jose Camacho-Cole Collado,, Luis Es- pinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644-1650, Online, Association for Computational Linguistics.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics, pages 4885-4901 Online, Association for Computa- tional Linguistics.
Robert Geirhos, Jömn-Herii Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, page 665-673.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy. Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts natural language infer- ence data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistie:: Human Language Tech- nologies, Volume 2 (Short Papers), pages 107-112. New Orleans, Louisiana. Association for Computa- tional Linguisties.
Pouya Pezeshkpour, Sarthak Jain, Sameer Singh, and Byron Wallace. 2022. Combining feature and in- stance attribution detect artifacts. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1934-1946, Dublin, Ireland. Association for Computational Linguistics.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018 Hypothesis only baselines in natural language infer- ence. Proceedings of the Seventh Joint Confer- ence on Lexical and Computational Semantics, pages 180-191, New Orleans, Louisiana. Association for Computational Linguistics.
Xiaochuang Han, Byron C. Wallace, and Yulia Tsvetkov. 2020. Explaining black box predictions and unveil- ing data artifacts through influence functions. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5553 5563, Online. Association for Computational Lin- guistics.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Be- havioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4902- 4912. Online. Association for Computational Lin- guistics.
Xanh Ho. Johannes Mario Meissner, Saku Sugawara, and Akiko Aizawa. 2022. survey on measuring and mitigating rcasoning shortcuts in machine reading comprehension. Computing Research Repository. arXiv:2209.0182.
Viktor Schlegel, Goran Nenadic, and Riza Batista- Navarro. 2020. Beyond leaderboards: A survey of methods for revealing weaknesses in natural language inference data and models. Computing Research Repository, arXiv:2005. 14709.
Nitish Joshi, Xiang Pan, and He He. 2022. Are all spurious features in natural language alike? anal- ysis through causal lens. Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 9804-9817, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceed- ings of the 34th International Conference Machine Learning, page 33 3319-3328.
Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. 2020. The multilingual Amazon reviews cor- pus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4563 4568. Online, Association for Computational Linguistics.
Tianlu Wang, Rohit Sridhar, Diyi Yang, and Xuezhi Wang. 2022a. Identifying and mitigating spurious correlations for improving robustness in NLP mod- els. In Findings of the Association for Computational

---

Linguistics: NAACL 2022, pages 1719-1729, Seattle, United States. Association for Computational Lin- guistics.
A Details of Input Reduction
Algorithm 1 Pseudo-code of Input reduction
Xuezhi Wang, Haohan Wang, and Diyi Yang. 2022b. Measure and improve robustness in NLP models: survey. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 4569-4586, Seattle, United States. Association for Computational Linguistics.
1: function INPUT POPUU_RRDUCTION_LCE
for all (z, E D do
9) [(x); x' &- 4x;; 1/ s 9
while 4 do
Frrew < x'; Upreo < I (1
+ IG IG_mask(z')
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of Association for Computational Lin- guisticy: Human Language Technologies, Volume I (Long Papers), pages 1112-1122, New Orleans,
j' < f(x²)
8:
if all tokens in x' are mask then break
9:
10:
end if end while
11:
12:
C C {p (Tprev Hprev)
Louisiana, Association for Computational Linguis-
13: end for
tics.
14: return
15:
end function
B Experimental setup
B.1 Dataset detail
Dataset
train
validation
test
NLI
MNLI (matched)
392,702
9,815
9,796
ANLI (round3)
100,459
1,200
1,200
SA
Tweeteval (sentiment)
45,615 200,000
2,000
12,284
MARC (en)
5,000
5,000
B.2
Models detail
Task
Model
NLI
roberta-large-mnii cadiffnnpotwiitee--bbrraaaaa
SA


Reviews:
[{"review_id": "DKwwYyy4wx", "reviewer": "Reviewer_iQTs", "summary": "", "questions": "Question A: \nIn Appendix A line 11, since the while conditon is $\\hat{y}$ == $\\hat{y} ' $, then after the while condition $\\hat{y} ' $ has already flipped, so line 11 actually include [\"Spielberg\"] → negative into the shortcut reasoning set, rather than [\"Speilberg\"] → positive, which should be included. \nIs my understanding correct? It looks like an issue needs to be changed. \n\nQuestion B:\nLine 343-345, what does the following statement mean? \n> \"If the prediction for the masked input does not flip during the reduction, then we alternatively output the last token left in the input\".\n\nWhy it is processed in this way?  You can do in the way that not include this shortcut reasoning right? Also this process is not reflected in Appendix A, which is confusing. \n\nQuestion C:\nIn Line 152 - 155, \"IR sorts the tokens …. ascending order of their rank\". This statement is not clear. How the tokens $x_i$ is sorted based on IG score, ascending or descending? Also gives a brief explanation of IG score, the higher the more important or the other way? So the IR is to reduce the less important tokens first, so that the final shortcut reasoning pattern is shorter?\n\nQuestion D:\nWill the sampled test data be released for reproduce the result?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the detailed review. We respond to your comments and questions below.**\n\n## Reason To Reject\n### (1): The lack of trustworthy evaluation involving human annotation\nFirst of all, we would like to clarify the difference between Wang et al. (2022a)’s goal and ours. Wang et al.(2022a) aimed to identify shortcut reasoning that is irrational from humans’ point of view. Thus, they conducted human evaluation for the verification of the results.\n\nOn the other hand, our work employs less subjective definition: we focus on detecting shortcut reasoning that performs well on IID and underperforms on OOD (Lines 181-185; see Geirhos et al. (2020) for the detailed definition and discussion). Hence, we do not care whether the detected shortcut reasoning is irrational or not from humans’ point of view. In our work, we construct quantitative metrics (e.g., $g$, $\\Delta$) for each definition above, where they do not require any human annotators. \n\nOverall, we believe that the discovered shortcut reasoning is already trustworthy because they strictly align with the definitions. We will clarify these points in the final version.\n\n## Questions For The Authors \n### (A): An error in pseudo-code in Algorithm 1 in Appendix A\nThank you for pointing out the fault of the pseudo-code. We found that the pseudo-code was indeed incorrect and needed the following revision: \n- line 6: $\\hat{y}'\\_{prev} \\gets f(x'_{prev}) \\\\, ; \\\\, \\hat{y}' \\gets f(x')$\n- line 11: $C \\gets C \\cup \\\\{ p = (x'\\_{prev},\\hat{y}'\\_{prev}) \\\\}$\n\nTherefore, Algorithm 1 does add inference patterns with an unflipped label to C (e.g., [“Spielberg”] -> positive, in your example). We will clarify it in the final version. Importantly, we note that only the pseudo-code was incorrect and the reported results had no problem. \n\n### (B): The last token of IR iteration\nWe sequentially mask input sentences to obtain minimal sequences of tokens, starting from the lowest IG scores. Namely, a token with the highest score remains at the end of IR iteration. Since we never know the “expected” prediction after the last token is masked (i.e., a sequence of input-length <mask>), we do not observe whether the prediction flips after the final masking. As such, we leave the last token and output it as a candidate of inference patterns. Importantly, it is a “candidate” of the pattern. We then evaluate the patterns by examining its generality and the extent to which the pattern harms performance on OOD. Therefore, there is no significant concern in utilizing such patterns.\n\n### (C): The definition of IG score\nWe agree that the description (Line 152-155) is unclear. As you pointed out, it is correct that the IR is to reduce the *less important tokens* first, so that the final shortcut reasoning pattern is shorter. IG scores higher values for a token that strongly influences the prediction. We first mask tokens with lower scores, namely less important tokens for prediction, and then increase masks. We are planning to add an illustration of how IR works in the final version.\n\n### (D): Sampled data\nYes, we will release all the codes and datasets used in the experiments, including the sampled data, for better reproducibility. \n\n## Typos Grammar Style And Presentation Improvements \n### (1): “Unless” is wrong?\nYes, “if” is right.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is the second paper that can automatically discover shortcut reasoning. It addressed the following limitation of the first paper (Wang et al, 2022a):\n\n(1) Wang et al, 2022a lacks the method to quantify the severity of the discovered shortcut reasoning on OOD data. This method is necessary and interesting, since we are concerned with how the shortcut reasoning will affect the potential OOD data. \n\n(2) (Wang et al, 2022a) made a plausible assumption that _genuine_ tokens which are useful for predicting labels across different datasets do not lead to shortcut reasoning. However, (Joshi et al. 2022) argue that this assumption does not hold in the situation that these tokens alone may not provide sufficient information to accurately predict labels, for instance, in the sentence \"This movie is not good\", \"good\" is necessary but not sufficient to determine the sentiment of the sentence. \n\n(3) Since this paper can quantify the severity of the shortcut, it can rank and show the representative shortcut reasoning. In the experiment part, it shows the most representative shortcut reasoning and verifies that the result aligns with the findings mentioned in previous works (shown in Sec 3.2 when discussing dataset NLI and SA).\n\n(4) Besides, the presentation and writing of this paper is very concise and clear, easy to follow and understand.", "reasons_to_reject": "(1)  The main weakness of this paper is the lack of trustworthy evaluation involving human annotation. \n\nEvaluating identified shortcuts in machine learning or deep learning based on models is difficult. The work Wang, et al, 2022a gives very detailed and convincing evaluation involving human annotators and provides precision scores. Although this work identifies some shortcut reasoning that aligns with the findings in the previous research, it lacks an overall and trustworthy evaluation on all the shortcut reasoning it has discovered. I hope it can also show more case studies, demonstrations of the examples. \n\nI hope the evaluation issue can be addressed so that I would like it to be shown in the main conference short paper. Otherwise, I feel that the evaluation is not sufficient.", "questions_to_authors": "Question A: \nIn Appendix A line 11, since the while conditon is $\\hat{y}$ == $\\hat{y} ' $, then after the while condition $\\hat{y} ' $ has already flipped, so line 11 actually include [\"Spielberg\"] → negative into the shortcut reasoning set, rather than [\"Speilberg\"] → positive, which should be included. \nIs my understanding correct? It looks like an issue needs to be changed. \n\nQuestion B:\nLine 343-345, what does the following statement mean? \n> \"If the prediction for the masked input does not flip during the reduction, then we alternatively output the last token left in the input\".\n\nWhy it is processed in this way?  You can do in the way that not include this shortcut reasoning right? Also this process is not reflected in Appendix A, which is confusing. \n\nQuestion C:\nIn Line 152 - 155, \"IR sorts the tokens …. ascending order of their rank\". This statement is not clear. How the tokens $x_i$ is sorted based on IG score, ascending or descending? Also gives a brief explanation of IG score, the higher the more important or the other way? So the IR is to reduce the less important tokens first, so that the final shortcut reasoning pattern is shorter?\n\nQuestion D:\nWill the sampled test data be released for reproduce the result?", "paper_topic_and_main_contributions": "This paper focuses on the task of discovering shortcut reasoning from an NLP model. Shortcut reasoning is the \"lazy thinking\" of a NLP model, in other words, the irrational process of inference, which affects the robustness of an NLP model. This irrational process of inference is often derived from the spurious correlations in the training data.  For instance, for sentiment analysis, a model tends to classify a movie review with word \"Spielberg\" is positive. However, \"Spielberg\" should not be a causal feature for the classification. \n\nThe contribution of the paper are:\n1. Different from most of the previous work (except for (Wang et al. 2022a)), which based on (1) human priors and (2) error analysis over the model (3) a pre-defined series of test types,  this paper proposed an automatic method for identifying shortcut reasoning. \n\n2. This method can quantify the severity of shortcut reasoning by leveraging OOD data. \n3. This automatic process can not only discover reported ones in previous research but also can discover unknown shortcut reasoning.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 333, \"unless\" should be \"if\". you mean \"if we use black box models\", then \"knowing the actual reasoning process is impossible\" right?"}, "extra_text": {}}, {"review_id": "ToIY03diZK", "reviewer": "Reviewer_X4Zv", "summary": "", "questions": "1. For the train/test column in Table 1, does the train/test refer to the train/test split of the IID dataset? If so, please add clarification in the paper. It can be confusing as the OOD dataset is also another kind of test set.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the review and advice for the paper. We have organized our responses below.**\n\n## Reason To Reject \n### (1-1): Proposed method heavily relies on the accessibility of an OOD dataset\n\nWhen an OOD dataset is not available, one could use LLMs to generate OOD data. Now that we can access language models that have surprising linguistic capabilities and are well-aligned with the user's instruction. The generated examples by LLMs have a certain distribution which can be treated as OOD for target models, or we can prompt them to generate examples with specific distribution.\n\nAs described below, even the SoTA method still relies on OOD dataset, and discovering shortcuts without the accessibility of OOD dataset has considerable difficulty. We think that this exploration is beyond the scope of this paper.\n\nWhile we already acknowledge the difficulty about the accessibility of OOD datasets in Limitation section, we will enrich the Limitation section by adding more discussion like this in the final version.\n\n### (1-2): Prior work such as Wang et al. 2022a didn’t rely on an OOD dataset\nWe would like to point out that Wang et al. (2022a) also use an OOD dataset. They propose cross-dataset analysis based on two types of datasets:(i) the one used for training a model and (ii) the other one from different sources or domains used for discovering shortcut reasoning with the first dataset (called “cross-dataset analysis”). The second dataset corresponds to our OOD dataset. We think that Wang's method and our method are not different on this point, that is, both methods require two datasets.\n\n### (2): Proposed method is computationally expensive\nThe main use case of our method is as follows. During the development cycle, developers create an NLP model, and they apply our method to the model to find the model’s weakness. This is done only during the development phase, and they don’t need to use our method after deployment. As you pointed out, our method does need extensive computation, but it is not prohibitively costly. For example, on 1000 examples of sentiment analysis data, Input Reduction took only 2 min 29 sec, and Generality calculation, which is the most computationally expensive, took only 47 min 27 sec on NVIDIA A6000 48GB.\n\nWe suspect that the term “Computational limitation” in Line 235-236 was misleading. This was supposed to mean specifically about run time. Unless the user cares about time, our framework can handle more than 1000 examples.\n\nWe clarify all the above points in the final version.\n\n## Questions For The Authors\n### (1): Train/test column\nYou are right. We are going to add some explanation on it. \n\n## Typos Grammar Style And Presentation Improvements \n### (1): Description of IR is confusing\nThank you for the suggestion. We are planning to clarify this  by adding a figure showing a working example (e.g., “Spielberg is a great director” =>\n“Spielberg is <MASK> great director” => “Spielberg <MASK> <MASK> great director” => …) in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is conceptually straightforward and reasonable. It can be easily implemented on many different datasets.\n2. On both NLI and sentiment analysis datasets, the method discovers interesting dataset patterns.", "reasons_to_reject": "1. The pattern discovery in the proposed method heavily relies on the accessibility of an OOD dataset, which is a pretty strong assumption. One main motivation behind all the shortcut discovery work is to figure out and fix the model's weakness so that it can perform well in future unknown OOD settings. Assuming the accessibility of the OOD dataset greatly limits the impact and usefulness of the proposed method. Additionally, this assumption also seems to be one of the key differences between this work and prior work such as Wang et al., 2022a. Since the current draft contains an in-depth comparison of this work and Wang et al., 2022a, I think a discussion of the adoption and the related consequences of this assumption (instead of just one sentence in the limitations section) is needed for this paper.  \n2. Due to the mask searching step, the method is computationally expensive (as acknowledged in LINE 235-236).", "questions_to_authors": "1. For the train/test column in Table 1, does the train/test refer to the train/test split of the IID dataset? If so, please add clarification in the paper. It can be confusing as the OOD dataset is also another kind of test set.", "paper_topic_and_main_contributions": "This work presents a new method to identify shortcut reasoning patterns in the models. Specifically, the method includes three steps. First, it finds input patterns by using Integrated Gradients to sort the tokens and by gradually masking the input. Then, for each discovered pattern, it calculates the generality of the pattern on the testing set, as well as the performance drop caused by the pattern. In the experiments, the proposed method successfully discovers patterns in both NLI and sentiment analysis datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The algorithm description at LINE 145-148 is hard to follow. I only understand the algorithm after checking the pseudocode in the Appendix. An example figure will also be very useful."}, "extra_text": {}}, {"review_id": "Y63oSO8ElX", "reviewer": "Reviewer_xMKL", "summary": "", "questions": "### Question about method details\n\n1. In line 127, $\\textbf{w}$ is a sequence of words, while in line 146, $\\textbf{w}_i$ is a sequence of unmasked tokens. As for my understanding, a 'word' is the basic element of language that carries an objective or practical meaning and a 'token' is the result of any tokenizer (which could be subword tokens, e.g. roberta, the model used in the experiment, uses a byte-level BPE as a tokenizer). Which one is used in the paper? The terminology is not consistent.\n2. Line 155, Section 2.2. Whether the most important token is masked first, or the least one? According to the pseudocode, IR-based extraction algorithm recognize the pattern as all the tokens left in the sentence before the label flips. So it should filter out the least important tokens, right?\n3. Appendix A. According to the algorithm, if the label does not change all the time, then the last token will still be recognized as a pattern. The limitation also mentions this. Why? I think it is more natural to drop this sample and not produce any pattern.\n4. Appendix A. According to the algorithm, what will happen if $\\hat y \\ne \\hat y'$ even if no token is masked? i.e. wrong prediction in the IID dataset. Based on line 202, I guess these samples won't be taken into consideration. But it would be better if stated clearly.\n\n### Question about soundness\n\n1. Line 189, Section 2.4. When $l_i$ meets its gold label $y_i$, the model performs well on $D_{IID}$. Why? I think only one example does not support the claim.\n2. Relatedly, line 255, Section 3.2. Negation expressions are known to be difficult for NLP models. For IID dataset, the performance on samples with negation expressions will also be much lower than other samples[1]. Also, according to table 1, g is not very high, which may indicate that the model is not very confident at these samples rather than taking it as a shortcut.\n\n[1] Naik, A., Ravichander, A., Sadeh, N., Rose, C., & Neubig, G. (2018). Stress Test Evaluation for Natural Language Inference. ArXiv. /abs/1806.00692", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the review and interesting question about the results. We have replied to comments and introduced the result of an additional experiment below.**\n\n## Reason To Reject \n### (1): Proposed method only discovers word(token) triggered shortcuts\nAs you pointed out, our current method is restricted to discovering token-based shortcuts for classification of a sequence. We will clarify the scope of our method in Introduction. However, we successfully discover many token-based shortcuts without any human intervention, which is a significant achievement. \n\n### (2): The method description is somehow confusing in detail\nAnswered below.\n\n### (3): The experiment section is lack of details\nWe are going to add more details about the experiments in the camera ready version. We will release all the codes and datasets used in the experiments for better reproducibility.\n\n## Questions For The Authors\n## Question about method details\n### (1): word or token\nOur description may be confusing. We use tokens for $w_i$, not words. We will clarify it in the camera-ready version.\n\n### (2): Masking algorithm\nYes, IR is to reduce the *less important tokens* first, so that the final shortcut reasoning pattern is shorter. IG scores higher values for a token that strongly influences the prediction. We first mask tokens with lower scores, namely less important tokens for prediction, and then increase masks. We are planning to add an illustration of how IR works in the final version. \n\n### (3): The last token of IR iteration\nWe sequentially mask input sentences to obtain minimal sequences of tokens, starting from the lowest IG scores. Namely, a token with the highest score remains at the end of IR iteration. Since we never know the “expected” prediction after the last token is masked (i.e., a sequence of input-length <mask>), we do not observe whether the prediction flips after the final masking. As such, we leave the last token and output it as a candidate of inference patterns. Importantly, it is a “candidate” of the pattern. We then evaluate the patterns by examining its generality and the extent to which the pattern harms performance on OOD. Therefore, there is no significant concern in utilizing such patterns.\n\n### (4): what will happen if $\\hat{y} \\neq \\hat{y}′$\nFirst of all, we want to clarify that $\\hat{y}$ is a model’s prediction for original input and $\\hat{y}’$ denotes a prediction for a masked input. So, $\\hat{y} \\neq \\hat{y}′$ does not mean wrong prediction in the IID dataset. \nTo answer the question, we just ignore the pattern, when the prediction for original input from IID is wrong. We will clarify it in the camera-ready version.\n\n## Question about soundness \n### (1):  Is only one example ($l_i=y_i$) enough to say “performing well on IID”?\nThank you for pointing out this important issue. We admit that $l_i=y_i$ may not be a robust metric for evaluating the efficacy of inference patterns on an IID dataset.\n\nTherefore, we conducted an additional experiment, where we evaluated the performance of each inference pattern using all the instances from IID (as opposed to one example). Specifically:\n\n1. We define a new metric called iid_acc, which counts the right prediction for inputs that contain trigger (w) as follows: \n$$\n\\displaystyle \\mathrm{iid} \\mathrm{acc} (p_i) \\stackrel{\\mathrm{def}}{=} \\frac{\\sum_{x \\in E_{iid}(w_i)}\\mathbb{1}\\left[ f(x)=l_i \\land l_i=y_i \\right]}{\\sum_{x \\in E_{iid}(w_i)} \\mathbb{1}\\left[ f(x)=l_i \\right]}.\n$$\n2. We identify shortcut reasoning from inference patterns based on the original condition for OOD and new metrics for IID. An inference pattern with more than 0.7 of iid_acc is classified as shortcuts. Specifically, we modified Equation (5) (Line 202) as follows:\n$$\n\\tilde{P} \\stackrel{\\mathrm{def}}{=} \\\\{p_i \\in C \\mid g(p_i) > \\lambda, \\mathrm{iid}\\mathrm{acc}(p_i) > 0.7, \\Delta_i \\\\! < 0 \\\\} .\n$$\n\nWe observe the results hold a similar trend to the previous results in the paper.  For NLI, negation and some newly discovered shortcuts are observed. For sentiment analysis, sentiment words tend to appear as shortcut reasoning. The results are shown below (Note that if $E$ is more than 1000, we randomly sample 1000 examples from it. So the results are slightly different from the submitted version.):\n\n**NLI**\n| $\\tilde{p}$  | $g$ | $\\operatorname{iid acc}$ | $\\Delta$ | $\\mid E \\mid$ | $\\text{train/test}$ |\n|-----------------|--------------:|---------------:|--------------:|---------------:|:--------------:|\n| [“/s”, \"never\"] $\\rightarrow$ CONTRADICTION | 80.3         | 99.3          | -9.0         | 1515 | x/x |\n| [“/s”, \"no\"] $\\rightarrow$ CONTRADICTION | 69.0         | 94.1          | -6.7         | 2038 | x/x |\n| [“/s\", \"soon\"] $\\rightarrow$ NEUTRAL | 64.1         | 100.0         | -6.5         | 142 | x/x |\n| [“/s”, \"is\", \"always\"] $\\rightarrow$ NEUTRAL | 60.8         | 96.6          | -5.2         | 102 | x/- |\n| [“/s”, \"not\"] $\\rightarrow$ CONTRADICTION | 55.0         | 94.5          | -55.6        | 8708 | x/x |\n\n**SA**\n| $\\tilde{p}$  | $g$ | $\\operatorname{iid acc}$ | $\\Delta$ | $\\mid E \\mid$ | $\\text{train/test}$ |\n|-----------------|--------------:|---------------:|--------------:|---------------:|:--------------:|\n| [\"worst\"] $\\rightarrow$ NEGATIVE | 97.5 | 81.7  | -25.4 | 158 | x/- |\n| [\"Excellent\"] $\\rightarrow$ NEGATIVE | 96.2 |100.0 | -7.2 | 184 | -/x |\n| [\"Perfect\"] $\\rightarrow$ POSITIVE | 96.0 | 88.6 | -12.9 | 324 | -/x |\n| [\"poor\"] $\\rightarrow$ NEGATIVE | 87.1 | 76 | -12.9 | 458 | x/- |\n\n\nWe will add the new metrics $\\mathrm{iid acc}$, the new Equation (5), and new obtained shortcuts in the camera-ready version.\n\n### (2): Negation is difficult for NLP models\nWe appreciate your interesting analysis. We agree that NLP models tend to struggle to understand negation expressions in general and may not be confident at samples containing negation expressions.\n\nHowever, we would like to clarify that g actually denotes *the generality of inference patterns (i.e., how much the inference pattern is dominant on the OOD dataset)*, not the performance of a model (cf. Eq.3). In other words, a lower g does not mean that the model is not confident for the input.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-organized. The authors clearly state the problem and their contribution.\n2. The proposed method can discover and provide a quantitative analysis to the shortcuts. A natural idea that could benefit future research on shortcut reasoning.", "reasons_to_reject": "1. This paper raises a broad question: shortcut reasoning. While as mentioned in section 2.1, the proposed method only discovers word(token) triggered shortcuts. Such kind of phenomenon may only contribute to a small proportion of shortcut reasoning (shallow heuristics, artifacts). Also, the task is limited to sequence classification. I think it could be better if these settings are clearly stated at the very beginning of the paper (abstract or introduction).\n2. The method description is somehow confusing in detail. Some evidence may not support the claim. I have raised some questions below.\n3. The experiment section is lack of details. I think it is necessary to provide more experiment details (settings such as which split does OOD dataset use, results such as overall f1 score $F1(D_{OOD}, f)$, more analysis on unknown shortcut reasoning) in order for the paper to be accepted (it is fine to place them in the appendix). In addition, There is also no mention of the code being released.", "questions_to_authors": "### Question about method details\n\n1. In line 127, $\\textbf{w}$ is a sequence of words, while in line 146, $\\textbf{w}_i$ is a sequence of unmasked tokens. As for my understanding, a 'word' is the basic element of language that carries an objective or practical meaning and a 'token' is the result of any tokenizer (which could be subword tokens, e.g. roberta, the model used in the experiment, uses a byte-level BPE as a tokenizer). Which one is used in the paper? The terminology is not consistent.\n2. Line 155, Section 2.2. Whether the most important token is masked first, or the least one? According to the pseudocode, IR-based extraction algorithm recognize the pattern as all the tokens left in the sentence before the label flips. So it should filter out the least important tokens, right?\n3. Appendix A. According to the algorithm, if the label does not change all the time, then the last token will still be recognized as a pattern. The limitation also mentions this. Why? I think it is more natural to drop this sample and not produce any pattern.\n4. Appendix A. According to the algorithm, what will happen if $\\hat y \\ne \\hat y'$ even if no token is masked? i.e. wrong prediction in the IID dataset. Based on line 202, I guess these samples won't be taken into consideration. But it would be better if stated clearly.\n\n### Question about soundness\n\n1. Line 189, Section 2.4. When $l_i$ meets its gold label $y_i$, the model performs well on $D_{IID}$. Why? I think only one example does not support the claim.\n2. Relatedly, line 255, Section 3.2. Negation expressions are known to be difficult for NLP models. For IID dataset, the performance on samples with negation expressions will also be much lower than other samples[1]. Also, according to table 1, g is not very high, which may indicate that the model is not very confident at these samples rather than taking it as a shortcut.\n\n[1] Naik, A., Ravichander, A., Sadeh, N., Rose, C., & Neubig, G. (2018). Stress Test Evaluation for Natural Language Inference. ArXiv. /abs/1806.00692", "paper_topic_and_main_contributions": "This paper proposes a method to automatically discover granular-feature shortcut reasoning in NLP models and quantitatively evaluate how much the shortcut reasoning affects the model's robustness. Compared to previous works, this method does not rely on any certain heuristic assumptions about the trigger token. Experiment observations are consistent with previous works. The authors also find some shortcuts that are previously unknown.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: TLDR: The paper makes an impportant contribution proposing an automatic method for identifying shortcut reasoning; it can quantify the severity of the shortcut, and discover unknown shortcut reasoning. The concerns raised by the reviewrs are mainly about limitations of the work and a better comparison with related work. There are also some clarification points. The authors address well all raised concerns in their rebuttal and provide additional results to further support their claims.

Following is a summary of the pros and cons identified by the reviewers in order of importance:

Pros:
1. Significance/Originality - all reviewers agree that the paper makes an important contribution with an automatic method for identifying shortcut reasoning; it addresses important limitations of related work on shortcut discovery (iQTs), the method discovers interesting dataset patterns (X4Zv), could benefit future research on shortcut reasoning (xMKL).
2. Quality - the proposed method is conceptually straightforward and reasonable (X4Zv).
3. Clarity - the presentation and writing of this paper are very concise and clear (iQTs, xMKL).

Cons:
1. Clarity - some evidence might not support the claim, e.g., is only one example enough to say performing well on IID (xMKL).
2. Clarity - include a more comprehensive discussion of the limitations of the method - the need for OOD dataset (X4Zv), computational cost (X4Zv), discovering only a certain type of shortcut reasoning (xMKL).
3. CLarity - a more comprehensive comparison with related work in terms of evaluation (iQTs), and dataset usage (X4Zv).
Decision: Accept-Findings