See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknesses
Authors: Yulong Chen
Yang Liu
Jianhao Yan
Xuefeng Bai
Ming Zhong
Yinghao Yang
Ziyi Yang
Chenguang Zhu
Yue Zhang
Venue: COLM (2024)
Keywords: self-challenging, human-in-the-loop, evaluation
TL;DR: We introduce a new evaluation method combining LLMs and human insight to identify and analyze the persistent 'bug' patterns in LLMs.

Abstract: The impressive performance of Large Language Models (LLMs) has consistently surpassed numerous human-designed benchmarks, presenting new challenges in assessing the shortcomings of LLMs. 
Designing tasks and finding LLMs' limitations are becoming increasingly important.
In this paper, we investigate the question of whether an LLM can discover its own limitations from the errors it makes. 
To this end, we propose a Self-Challenge evaluation framework with human-in-the-loop.
Starting from seed instances that GPT-4 fails to answer, we prompt GPT-4 to summarize error patterns that can be used to generate new instances and incorporate human feedback on them to refine these patterns for generating more challenging data, iteratively.
We end up with 8 diverse patterns, such as text manipulation and questions with assumptions. 
We then build a benchmark, SC-G4, consisting of 1,835 instances generated by GPT-4 using these patterns, with human-annotated gold responses.
The SC-G4 serves as a challenging benchmark that allows for a detailed assessment of LLMs' abilities. 
Our results show that only 44.96\% of instances in SC-G4 can be answered correctly by GPT-4. 
Interestingly, our pilot study indicates that these error patterns also challenge other LLMs, such as Claude-3 and Llama-3, and cannot be fully resolved through fine-tuning. Our work takes the first step to demonstrate that LLMs can autonomously identify their inherent flaws and provide insights for future dynamic and automatic evaluation.

Published as a conference paper at COLM 2024
See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknesses
Yulong Chen 12+ Yang Liu Jianhao Yan' Xuefeng Bai
Ming Zhong*
Yinghao Yang! Ziyi Yang² Chenguang Zhu
Yue Zhang
Westlake University 2 University of Cambridge
Microsooft GenAI
4 UIUC Westlake Institute for Advanced Study yulongchen 1010@00010.m.mmmmmmmmmmmmmmm yaliu10@micrenn wweemicrsooft..mm com
yue zue.zhang@wass.rg cn
Abstract
The impressive performance of Large Language Models (LLMs) has con- sistently surpassed numerous human-designed benchmarks, presenting new challenges in assessing the shortcomings of LLMs. Designing tasks and finding LLMs' limitations this paper, we investigate the question of whether an LLM can discover its own limitations from the errors makes. To this end, we propose a Self- Challenge evaluation framework with human-in-the-looe seed instances that GPT-4 fails to answer, we prompt GPT-4 to summarize enror patterns that can be used to generate new instances and incorpo- rate human feedback on them to refine these patterns for generating more challenging data, iteratively. We end up with 8 diverse patterns, such as text manipulation and questions with assumptions. We then build a benchmark, SC-G4, consisting of 1,835 instances generated by GPTT44 using these patterns, with human-annotated gold responses. The SC-G4 serves challenging benchmark that allows for detailed assessment of LLMs' abilities. Our results show that only 44.96% of instances in SC-G4 can be an- swered correctly by GPT-4. Interestingly, our pilot study indicates that these error patterns also challenge other LLMs, such as Claude-3 and Llama-3, and cannot be fully resolved through fine-tuning, Our work takes the first step to demonstrate that LLMs can autonomously identify their inherent flaws and provide insights for future dynamic and automatic evaluation.
becoming increasingly important. In
Starting from
1 Introduction
Large Language Models (LLMs), such GPTT4 (OpenAI, 2023) and Llama (Touvron al., 2023a; Dubey et al., 2024), have shown remarkable performance on diverse Natural Lan- guage Processing (NLP) tasks, and have been trusted by users search engines and personal assistant due to its great capacity (Tan et al., 2023b). To better understand the capability of LLMs, much effort has been dedicated evaluating LLMSs on multiple benchmarks (Fabbri et al., 2021; Gao & Wan, 2022; Huang et al., 2023; Bang et al., 2023; Zhong et al., 2023). Early LLM evaluation work follows traditional evaluation protocols, which are designed to evalu- ate task-specific models on datasets of single tasks (Nallapati et al., 2016; Rajpurkar et al., 2016; Goyal et al., 2022), or benchmarks that assemble multiple tasks for evaluating a certain capability, e.g., math problems (Cobbe et al., 2021) summarization (Chen et al., 2023). More recent work evaluates LLMs on complex and expertise-level human tasks (Zhong et al., 2023; Skopek et al., 2023; Wang et al., 2024; Gandhi et al., 2024), such the lawyer qualification test (Shui al., 2023), GAOKAO test (Zhang al., 2023b), etc.!
'Yulong Chen completed this work during his internship at Microsoft when he was a D.Phil. student at Westlake University.
Yuu Zhang is the corresponding author.
Due to space limit, we present more Related Work in Appendix A.
1

---

Published as a conference paper at COLM 2024
Step 1: Pattern Summarization
Step 2: Pattern Evaluation
Fallure Inetances
A List of Error Patterns
One Domali Word
New Instances under Pattern Domain
Pee Alvolia
Pattern
Seete of murh hea
Step 3: Pattern Optimization
Initial pattern
Human Feedbackk
Optimized Pattern
Tmoponal Janilg eir Che wit Srreeh 28T
Dempuuut Aodijauly beserrahh Sporite apnduaaer qppiiity dine
2ng AD CONCLE
offcklyy
Nor 3PV F2 peoeec poser
Humman Feediback on Model Responses
Figure 1: The overall Self-Challenge framework. We first summarize initial error patterns from seed failure instances (Step 1). Then, we perform pattern evaluation (Step 2) to determine whether summarized patterns can be used to generate challenging queries, and obtain corresponding human feedback; pattern optimization (Step 3) to modify the original pattern, making it more accurately describe challenging features, based on human feedback (the difference between the initial pattern and optimized pattern in is highlighted by underlined text). We frame Step 2 and Step 3 iteratively.
Although benchmarking scores are widely used to assess the capabilities of LLMs, they offer somewhat superficial view, iee, mostly indicating whether LLMs can perform a specific task without deeper analyses into the systems to uncover the precise reasons for failure or the nuanced aspects of performance degradation (Zhang et al., 2023b). As result, people can be satisfied with high evaluation scores, but overlook the model's real weaknesses reflected in the negative instances, which can pertain to more fundamental issues, such as tokenization (Sennrich et al., 2015). Therefore, developing more dynamic, adaptive, and aulomated evaluation methods that can efficiently detect the limitations of LLM capabilities remains an open challenge for the research community.
To mitigate the above, we propose a Self-Challenge framework that allows dynamic and interpretable evaluation for LLMs. The key idea is to prompt LLMs to identify their own limitations based on the errors they make, and then generate queries featuring those limitations to challenge themselves. As shown in Figure 1, given a set of failure instances from an LLM, we first prompt the LIM to summarize error patterns (Step 1), and then ask it to iteratively evaluate (Step 2) and refine these patterns (Step 3), discovering new challenging features and detailing those features. During this process, we engage human experts to evaluate the quality of summarized patterns by manually identifying whether such patterns can be used to generate challenging queries. Such a framework offers two advantages: First, it enables the dynamic generation of large-scale datasets for quantitative evaluation, starting from a small set of qualitative instances. Thus, it is adaptive and free of human design for tasks. Second, the patterns provide a detailed description of features that can lead the failure of LLMs, which allows more fine-grained evaluation and thereby makes the weaknesses of LLMs more interpretable.
Employing the Self-Challenge framework on GPT-4 (OpenAI, 2023), we obtain 8 challenging patterns from 189 failure instances. Those error patterns are distinct, covering different aspects of GPT-4' limitations, such as Text Manipulation or Transformation and Temporal Ambiguity or Confusion with Specific Events. Leveraging these patterns, we construct new benchmark, SC-G4, consisting of 1,835 challenging instances in the combination of patterns and 30 domains, where GPT-4 can only answer 44.96% correctly.
We further investigate characteristics of those error patterns using SC-G4. We first bench- mark multiple LLMs in zero-shot setting and show that those error patterns from GPT-4
LLater in this paper, unless otherwise specified, the terms "challenge" or "challenging" refer to behaviours or features of queries that can induce L.Mss to generate incorrect, hallucinatory, or non- factual responses.
2

---

Published as a conference paper at COLM 2024
can generalize across different LLMs such Claude-3 only achieving 24.47% accuracy and Llama-3-70B achieving 23.32% accuracy. Interestingly, our pilot fine-tuning experiments show these errors cannot be reliably addressed by simply fine-tuning on SC-G4, suggesting that part of them can be "bugs" of LLMs, such as incompetence of text manipulation on sub-word and character levels.
We release our data at https://g/ http:/://ggghuubcooo////g/////// com cowp///caalll/see--aallleeeeeeee
2 Self-Challenge Framework
In this section, we introduce the Self-Challenge framework, designed to identify and sum- marize error patterns in LLMs from errors. Additionally, enables the generation of representative data under each pattern, facilitating quantitative evaluation for various LLMs
The overall process of Self-Challenge is shown in Figure In particular, given instances, the LLM is first prompted to discover potential challenging patterns from the instances (subsection 2.1). To ensure the generated patterns are of high quality, we perform pattern evaluation (subsection 2.2) and pattern optimization (subsection 2.3) with human-in- the-loop. We further frame the above two processes an iterative manner (subsection 2.4). Appendix B presents detailed prompt information.
of
2.1 Pattern Summarization
As shown in Step 1 in Figure 1, given a set of queries Q = responses R- = [[)) (IQI = |R) by directly prompting the LLM with the query, i.e.:
(9/1))////)) we obtain LLM
f;
(q:),
(1)
where di is the i-th query, & is the LLM function, and ri is the LLM-generated response to qi- We also collect correct responses C = {CI, C(c J, where cj is the correct response to 9;-
Then, prompt the LLM to summarize potential error patterns from those instances:
((Prompssunn (2, R, C)),
(2)
where Promptsumn is natural language instruction that guides the LLM to generate initial patterns and is the summarized pattern list (P {Pi..., Pip }).
2.2 Pattern Evaluation
In our experiments, we notice that some initially summarized patterns can be coarse- grained and of low quality, ie., generated pattern fails to capture the features of queries that challenge LLMs. Therefore, it is essential to evaluate these patterns. However, since there are no fixed answers or gold standards for what constitutes good pattern, evaluating them directly (e.g,, by comparing the similarity of two pieces of text) is not feasible.
To address this, we perform the pattern evaluation indirectly. In particular, we evaluate whether a pattern accurately summarizes and describes the challenging features of set of queries by determining if it can be used to generate new, analogous, and challenging queries. Therefore, given summarized pattern p, we prompt the LLM to generate new queries in specific domain d Le.:
Qr Φ Prrmppgen (p,d)),
where QPd is the new query set under the pattern of p in the domain of d, and Prompt gen is natural language instruction.
Similarly, for each query in Quai, we obtain its model-generated response by directly prompt- ing the L.M (Equation 1). Human annotators then provide feedback. Each human
We find that, without the domain information (i.e., Q" = Prromptee (p)), the generated queries tend to converge and lack diversity.
3

---

Published as a conference paper at COLM 2024
is natural language that describes whether the query successfully challenges the LLM, which functions both as an evaluation score and a "gradient" signal for later pattern opti- mization. Finally, can evaluate the pattern quality by accuracy, ie,, the portion of how many queries successfully challenge the LLM.
The detailed annotation information can be found in Appendix C.
2.3 Pattern Optimization
After evaluating the pattern and obtaining the feedback, then prompt the LLM to optimize it, making it more accurate and detailed.
Taking each pattern p and its newly generated queries Q" (QP USED Q14, where D = {d}), corresponding L.M-geenerted responses R and human feedback FP as input, we prompt the LLM to optimize the pattern so that the optimized pattern can better reflect the features of challenging queries and be used for generating new queries:
p' = Φ Prompptopt (p, QP,RP, RP,F)),
(4)
where p' is the optimized pattern and Prompfopt is natural language instruction.
2.4 Iterative Seff-Chllennee
We iterate the pattern evaluation and optimization processes ensure that the optimized patterns are of high quality, In particular, we replace with p' in Equation 3 and can obtain new set of queries generated by p'. Then we evaluate the query-response under p' and obtain corresponding human feedback to optimize p' by replacing with p' in Equation 4. Through this iterative process, we can continually optimize the patterns until they are considered good enough.
Finally, we can obtain set high-quality patterns, which illustrate where and how LLMs can fail and can be used to generate more diverse queries under corresponding patterns.
Discovering Error Patterns in GPT-4 and Constructing SC-G4 Benchmark
To demonstrate the usage and facilitate future research, we apply our Self-Challenge frame- work to GPT-4 (OpenAI, 2023). We first discover 8 error patterns from 189 failure instances and construct new benchmark, SC-G4, for quantitative evaluation.
3.1 Seed Instance Collection
We collect 189 diverse queries from online and from previous studies that qualitatively evaluate GPT-4 such (Zheng al., 2023b). These instances diverse in terms of task formats and domains, which cover traditional NLPP tasks such as multi-hop QA (Zheng et al., 2023b), and non-traditional NLP tasks such as word sorting (bench authors, 2023). Appendix D shows the collection details.
3.2 Error Pattern Discovery
Given the initial collection of queries, we first prompt GPT-4 to discover their initial patterns. Due the length limit, we group the instances according to their sources and prompt GPT-4 individually. In this way, we obtain a list of 14 initial patterns.
We then perform the pattern evaluation and optimization for two rounds. To smooth model bias towards certain domain, we provide 4 domain words. For each combination of pattern
4https: //tpt:://///at/rrm waw.attaaro.pp.ii....m.mm wwtmmai.com/occcooo cww/docs/moooeeeeooooooooooooooe Foor example: https:/ t/t tter. com/home and https: / /gi thub. com com//manyoso/hallt411]1
gpr-o- 3pt-4-32k version.

---

Published as a conference paper at COLM 2024
ID
Pattern Name
%
A
Assumption of Existence
179
9.75
B
Bias towards More popular or Well-known Informatton&&Oveegeeeelllazzzi
221
12.04
C
Complex Counting or Identification Tasks
232
12.64
Complex Logical Reasoning and Paradoxes
229
12.48
Complex Syntactic Structures & Multiple Clauses
256
13.95
F
Recursive and Unusual Patterns
230
12.53
Temporal Ambiguity Confusion with Specific Events
217
11.83
H
Text Manipulation or Transformation
271
14.77
Table 1: A list of names of patterns, with instance counts (#) and portions (%) for each individual pattern.
and domain, we generate 10 new queries, and thus have 4 x 10 40 new queries for each pattern. Then, we manually evaluate model responses with human feedback. During this process, we discard patterns that are consistently found less satisfying, i.e., over 50% of generated queries cannot challenge GPT-4, Finally, we obtain 8 high-quality patterns.
Table 1 presents the list of pattern names, where we see that the error patterns cover different aspects that can challenge GPT-4, ranging from underlying problems such as character (letter) processing, which can be related to tokenization issues, to more high-level problems such as logical reasoning.
Figure 2 presents one typical error pattern and its corresponding original pattern, where we highlight their difference. We see that the original patterns are general while the optimized patterns are more comprehensive and detailed in describing what kind of specific features should included in the query, such as "such as dependency or constituency... " Moreover, the optimized patterns include instructions on how to re-produce queries under such patterns.
We also apply the Self-Challenge framework to Llama-3-70B (Dubey et al., 2024) for pattern discovery shown in Appendix G, which, however, is less effective compared with GPT-4.
3.3 The SC-G4 Benchmark
After obtaining the patterns, we construct the SC-G4 benchmark consisting of 1,835 queries with human-annotated gold responses under the combination of 8 patterns and 30 domains.
We randomly select 30 Wikipedia titles from Wikipedia metadata (Appendix E). For each combination of pattern and title, prompt the LLM to generate 10 diverse queries at once. Thus, we have x30 x10 2,400 new queries in total. Generally, we find that new queries can mostly follow the pattern and are diverse in terms of specific tasks. Take pattern C (Complex Counting or Identification Tasks, Appendix F) for example, the new query: "In the sentence, "Joey Votto is known for his exceptional plate discipline and ability get on base,' how many words have the letter 'é' as the third character and end with the letter 'n'?" suits the requirement of "identify specific elements in complex sentences" Moreover, the task of identifying letter that comes after another letter is not shown in the aforementioned instruction. It suggests we can obtain diverse queries out of the limited features described in patterns.
As we aim to assemble the benchmark by collecting self-challenging data generated by GPT- 4, there remain two major questions. First, a few generated queries can still distract from corresponding patterns, in particular when we use new domains for generation. Second, the generated queries can lead to open-ended responses such as " "Please combine two names of the centers and institutes at Stanford University in an alphabet order" which poses challenges for practical evaluation. Therefore, we further filter out 565 queries to ensure the remaining 1,835 queries can be used to evaluate LLMs by comparing ggneergede responses with responses annotated by humans. We ask 4 human annotators to write the gold response with the help of GPT-4, where annotators also manually give binary label to the GPT-4

---

Published as a conference paper at COLM 2024
Original Pattern:
Complex Syntanti Struchures Muhiipe Clauses GPT-4 struggles to acnurately parse santences with complex struturss multiple dluses keading to incurrect identiiictioon of specfiic words or relationhhips between words.
To generate questions that chal enge GPT-4, create senterces with multiple clauses embeedddd pfreses, and complex relationships between perse the sentence accurately and identify the correct answer
GPT-4 perse
idientffy specific
reletionships words. This will difficult for GPT-4
Original
Complex Syntactic Structures and Muttiple Clauses, GPT-4 strupgles to accurately parse sentences with complex structures and multiple relationships
dauses leading incorrect identiiiction specific
words.
To questiors that chal enge GPT 4, reletionships (Buch as dependency or constituency between words Then, ask GPT-4 parse the sentence, identlly speclfic werdl dependenccy constituency relationshis between words rephrase the sentence.
complex sentences with multipe
embedded phuases, and compolex
For eigmole J. 1'mm student very close the VCU area that's looking for an apartment or house move in with another permon/pegple What are words in the provious sentencee which
constitueccy with the second word bepins with 'x? more
l'Unfartinnatll, currently no funding or eapacity adyse inters or students consult the EdIN Information on potential supervsiors I" What are the wards in the grevicus sentence, which hold a dependency with "unnding or what are predicative words
sentences?
will make difficult GPT-4 raguire complea Angguistic anelysis
sentence accurataly and identify the
Avoid fact hatt- questions that
Figure 2: A case of optimized pattern, coupled with its original pattern. We highlight their difference. Full patterns can be found in Appendix F.
response, indicating whether it correct or not. This binary label is further used the gold evaluation results of GPT-4 on SC-G4. The detailed annotation information can be found in Appendix C.
Finally, we obtain 1,835 queries annotated with gold responses. Our human evaluation shows that GPT-4 can only achieve an accuracy of 44. 96%. Table 2 gives the basic statistics for the overall SC-G4 benchmark. This suggests that most queries generated by GPTT4 can successfully challenge GIT-44 themselves. Also, we see that compared with GPT-4 responses, the averaged token lengths of aaman-annotater responses much less. It shows most GPT-4 responses are further corrected by human annotators.
Benchmarking LLMs on SC-G4 and Investigating Error Generalization across LLMs
In this section, we investigate the main question of whether GPT-4 error patterns are universal and can generalize across other LLMs.
4.1 Experimental Setup
In addition to GPT-4, experiment with other closed-source LLMs and open-source LLMs on SC-G4, in both zero-shot and few-shot settings
Baseline Systems. We compare Gemma-7B (Team et al., 2024), Phi3-7B-instruct (Abdin et al., 2024), M models (Touvron et al., 2023b; Dubey et al., 2024), Claude-3-Sonnet! and GPT-3.5-turbo and GPT-4o models (Achiam al., 2023; Brown et al., 2020) with GPT-4 (OpenAI, 2023).
(Jiang et al., 2024), Gemini-1-pro (Team et al., 2023), Llama
Evaluation Data. For zero-shot experiments, we evaluate LLMs on the whole SC-G4 data. For few-shot fine-tuning, we experiment under 2 settings as shown Table (1) Split randomly: We first randomly split the SC-G4 into train/ dev test sets (500/335// (00/3355/000)), where train and test sets share common patterns and domains. (2) Split by domains: We split the SC-G4 into train/ main/dev test sets by constraining domains in them. In other words, train and test sets have different domains but share common patterns. In particular, we split data of 10/5/15 domains into / dev/test sets (629/247 /959), respectively.

---

Published as a conference paper at COLM 2024
Self-Chal
#
Avg Query
Avg Gold R
Avg GPT-4 R
Overall
1,835
34.84
49.08
78.13
Random-set
500/335/1,000
18.56/48.44/499
77.48/77.00/
Domain-set
629/247 /959
34.34/35.60/44.98
43.21/56.57/51/ /51001
79.05/78.12/77.44
Table 2: Basic statistics for SC-G4 benchmark. #: instance count. Avg: averaged token length. R: Response. We also report statistics of different data split strategies (Random and Domain) as discussed in subsubsection 4.2.2.
Implementation Details. For zero-shot experiments, we directly prompt baseline LLMs given queries as input. For few-shot experiments, we use LoRA (Hu et al., 2021) to fine-tune the Llama 2-Chat (7B) in different settings (c.f. subsubsection 4.2.2 for detailed settings of different data split strategies), We fine-tune the model on 8 A100 GPUs on our training sets for 5 epochs" and choose the best checkpoint by their loss on validation sets. We use the AdamW optimizer (Loshchilov & Hutter, 2017) and set the learning rate as 2e-4.
For GPT-4, we directly evaluate their performance when annotating the gold responses (subsection 3.3). However, it is not practical to conduct manual evaluation for all models, which can be costly. Therefore, we evaluate model performance via prompting GPT-4 (Liu et al., 2023) to compare the model response and gold response, i.e.
Evaluation Protocol.
I = Φ {(Pomppee h)r),,
where 1, 9, h and r are output label ("correct" or "incorrect"), query, gold and model-generated responses, respectively. Appendix presents the prompt in detail. We show that this evaluation method achieves a moderate-to-high F-1 score of around 90 in Appendix I.
4.2 Result and Analysis
We first show results in a zero-shot setting, finding the "bugs" are universal to other LLMs, and present few-shot results, finding most error patterns cannot be simply solved via tuning.
4.2.1 Zero-shot Performance
Table 3 shows the model performance the full SC-G4 data in the zero-shot setting. Generally, GPT-4 achieves the best performance of 44.96% human-evaluated accuracy and 47.63 GPT-4 evaluated accuracy, which, however, still less satisfying. Compared with GPT-4, other LLMs show poor performance on our data, where GPT-40 outperforms other LLMs including Turbo, Llama models, and Claude-3. This suggests that although those instances are initially aimed to challenge GPT-4, they are difficult for other LLMs. For Llama 2 based models, Chat models slightly outperform standard models. Also, we see that as the sizes of LLMs increase, the overall performance increases.
We further analyze the model performance under different patterns, respectively. The break- down analysis is shown in Table 3. Note that the pattern IDs are aligned with as in Table 1 and Appendix F, respectively. First, GPT-4 achieves the best human-evaluated accuracy on patterns D (68.1 (88), A (67.609 (67.6%)) and B (65.61%), and show very poor performance on patterns H (9. (9.23%), C (22.41%) 4)) and G (38.28 25.). Also, Turbo shows a similar trend on those each pattern is much poorer than GPT-4. Interestingly,
patterns, while the performance although outperforming Turbo on most patterns, GPT-4o shows poor performance on pat- tern A. We assume that this can be because the multi-modal training does not necessarily enhance the models' ability to disambiguate hallucinatory queries. In addition, Llama-based models and other LLMs underperform GPT-4 on all patterns.
For Llama models, it is noted that Chat models show very poor performance on pattern A compared with standard models, indicating they are more prone to hallucinate on queries that include non-existent entities. Also, we see that Llama models of bigger size do not
We also trained models with more steps, but did not observe large performance improvement.
7

---

Published as a conference paper at COLM 2024
LLMs
B
C
E
F
G
H
all
Gemma-7B
25.70
8.14
8.19
11.35
16.80
6.09
5.07
0.00
9.65
Gemini-l-pro
17.88
17.65
7.76
17.03
26.17
10.87
3.07
0.00
12.59
Phi3-7B-instruct
12.29
21.72
15.09
37.55
20.31
21.30
9.22
2.21
17.33
Llama-2-7B
0.56
19.46
4.74
13.54
22.66
4.78
6.45
0.00
9.21
Ilama- 1--na-2--B-chat
8.94
15.84
1.72
3.06
19.92
3.91
7.83
1.11
7.74
Illma-2-13B
0.00
23.53
6.90
14.41
27.34
5.65
11.06
0.00
11.34
Llama-2-13B-a 2-13BBchaat
17.88
18.55
5.17
13.97
22.27
5.65
7.83
0.37
11.17
Hlama-3-70B-instrmm
43.58
31.22
19.40
30.57
32.81
20.87
15.21
0.37
23.32
Mixtral-8x1
24.58
24.43
8.19
18.34
26.56
13.04
7.37
0.37
14.93
Blaude--SSonee
41.34
34.39
20.69
35.37
29.69
24.35
15.21
1.85
24.47
GPT-3.5-turbo
49.72
42.53
14.22
39.74
43.75
23.91
23.04
2.58
28.94
GPT-4o
30.73
41.63
30.60
44.54
43.36
35.22
20.74
8.86
31.66
GPT-4
65.92
69.68
22.84
71.18
60.16
46.52
43.32
11.44
47.63
GPT-4+
67.60
65.61
22.41
68.12
57.42
41.74
38.25
9.23
44.96
Table Model performance on full SC-G4 benchmark. We the overall performance and breakdown performance under different individual patterns. t indicates human evaluation while others are evaluated by GPTT-. The IDs of patterns correspond to IDs Table
ID
#
G-4t
G-4
L-2
L-2"
G-4
G-4
L-2"
106
65.09
60.38
7.55
31.131
A
98
67.35
67.35
7.14
35.711
110
64.55
69.09
15,45
10.001
B
73.21
76.79
16.96
11.61
C
129
20.16
23.26
1,55
6.20
C
124
20.97
20.16
2.42
4.841
D
118
66.95
71.19
1.69
0.85.
D
130
70.00
71.54
2.31
2.31
145
56.55
58.62
19.31
23.451
E
125
63.20
65.60
22.40
20.001
124
40.32
45.16
3.23
5.651
118
42.37
47.46
3.39
2.54
118
34.75
40.68
9.32
8.47|
G
113
42.48
46.02
10.62
9.73
150
8.67
10.00
1.33
0.00
H
139
7.91
10.79
0.72
0.00
all
1,000
43.10
45.80
7.40
10.401
all
959
47.24
49.24
8.03
10.01
Table 4: Model performance on randomly split test set (left) and performance on test set split by domains (right). G-4: GPT-4. L-2: Llama 2-Chat (7B). *: LoRA-tuned.
significantly perform better than smaller models on certain patterns, such pattern H (Llama-2-7B: 0.00; Lllama-2-13B; 0.00; Llama-3-70B: 0.37). This indicates that simply scaling up the model size may not solve problems of such patterns. We present the breakdown analysis of different domains in Appendix
4.2.2 Few-shot Performance
We experiment with 2 settings as stated in Table 4.1. The first is to give general overview of whether fine-tuning can bring improvement, while the latter is to present more fine- grained analysis.
Random Split. Table (left) presents the results, which also include zero-shot performance of GPT-4 and Llama 2-Chat (7B) on the test set for comparison. Generally, we see that the LoRA-fine-tuned model shows 3.00% accuracy improvement. However, we see that pattern A (23.58%1), but few on the rest. On
fine-tuning only brings great improvement the other hand, fine-tuning even slightly hurls the model performance, such as (5.45%)) and H (1.33%]). The above results suggest that most challenges cannot be simply resolved by fine-tuning.
Domain Split. As shown in Table 4 (right), similarly, the LoRA-tuned model shows the best improvement on pattern A (28.57%f) and shows poorer performance on B (5.35%)), E (2.40%)), (0.85%L), G (0.89%) and H (0.72%] than zero-shot model. Moreover, the performance gap between the LoRA-tuned and zero-shot models on this dataset is less than the randomly split dataset. Such narrower gap indicates that different domain further
8

---

Published as a conference paper at COLM 2024
enhances the difficulty of generalization ability. This suggests that the model does not learn relevant pattern knowledge from fine-tuning, except for pattern A.
4.3 Findings and Highlights
The error patterns be universal in different LLMs. Evidenced by our zero-shot results, we find all LLMs show poor results on GPT-4 error patterns. The results also show that most LLMs tend to perform relatively well on patterns A and E, while relatively poor patterns C and H. Similar trends can be consistent with previous research (Huh et al., 2024; Bommasani et al., 2022) that LLMs can be homogeneous they are trained with similar techs (e.g,, transformers (Vaswani et al., 2017)) on similar data (e.g., Wikipedia) (Touvron et al., 2023b; Radford al.; Brown et al., 2020).
LLMs tend to fail on tasks that are related to tokenization. By analyzing patterns C and H, we find they are mostly related to the tokenization mechanism in LLMs. Challenging queries under these two patterns include identifying certain (sub-)words in word, or counting words with a specific feature (e.g,, "start with 'é', and end with ")"". Such tasks can be closely related to applications such as poetry generation (Oliveira, 2017), spelling and grammar checking (Soni Thakur, 2018), and teaching aide (Felten et al., 2023). We emphasize that such issues reveal the problems of underlying tokenization mechanisms, which can further influence the performance of downstream tasks. Also, reveals that the current tokenization design (Sennrich et al., 2015) in LLMs is different from humans, making LLMs less sensitive to character-level information. While humans can easily solve these tasks, LLMs fail do so.
LLMs can sense the uncertainty within themselves. Our few-shot (in particular the cross- domain) experiments show that the Llama model can benefit most from fine-tuning on pattern A. It shows LLMs may have learned that there can be non-factual information in the query even although we do not fine-tune them on more in-domain data. We hypothesize that this reflects the LLMs' ability to sense internal uncertainty (Zhang et al., 2024; Ye et al., 2024), and that fine-tuning on such data enhances their threshold for taking uncertain queries into account and responding more cautiously. We show more analyses, including case study, in Appendix L.
Simply fine-tuning may not be able to solve all problems. We observe that fine-tuning can enhance model performance on certain tasks (pattern A) (Yin al., 2023). However, as evidenced by our few-shot experiments in different settings, for the other patterns, we find that fine-tuning does not bring consistent improvement. Although we acknowledge that little improvement can be attributed to the small training data, there can be some inherent flaws in LLMs. More advanced technologies or a large amount of relevant data may help to resolve those error patterns, which we leave for future work.
5 Conclusion
In this work, we introduced Self-Challenge framework that evaluates LLMs, revealing their nuanced limitations by prompting LLMs to challenge themselves iteratively. By integrating this framework with GPTT-4, we obtained 8 typical error patterns, and assembled the SC- G4 benchmark with 1,835 challenging instances, where GPT-4 achieves 44.96% accuracy. We further showed that such patterns are universal across multiple LLMs, highlighted by low accuracy rates in both zero-shot and few-shot settings, suggesting that the limitations may not be mere oversights but inherent "bugs" within current LLM architectures or their pre-training data.
Acknowledgments
We appreciate reviewers and meveeeess from COLM 2024 for their valuable feedback. We also appreciate Bonnie Webber, Andreas Vlachos, Leyang Cui and Rami Aly for their proofreading and insightful discussion. We acknowledge funding support from the NSFC key project 62336006.

---

Published as a conference paper at COLM 2024
6 Ethics Statement
The construction of SC-G4 involves the participation of GPT-4 (input query generation) and human annotation (gold output response).
Data Usage and Safety: During our annotation, we have ensured that the GPT-4 generated text safe and does not contain any uncomfortable descriptions, such as violent crimes and events. The data in SC-G4, in particular under pattern A (Assumption of Existence), includes fake questions that contain fictional entities or events. This inclusion is crucial for testing and enhancing GPT-4's s ability to reason about hypothetical speculative information, thereby improving its performance across a variety of real-world applications.
Potential Biases of GPT-4 Generated Queries: We acknowledge that relying GPT-4 to generate instances for evaluation can lead to potential biases, in particular data homo- geneity (Ding et al., 2024; Dunlap al., 2023). Although we enrich the input pattern with additional domain information and evaluate their level of diversity by calculating the Dis- tinct score (Li al., 2015) as in Appendix M, there remains a potential problem that queries under the same pattern and domain can be similar to each other without human verification. This problem can be solved by decomposing the current patterns into more fine-grained and detailed sub-patterns or providing larger domain text as source for query generation, which we leave for future work.
Human Annotation: Please refer to Appendix C, where we provide a detailed annotation plan and how we compensate the annotators.
References
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arx preprint ar #xXi:2444.14219, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint ar Xr1::22303.8877, 2023.
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. Do not have enough data? deep learning to the rescue! In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.
7383-7390, 2020.
Xuefeng Bai, Jialong Wu, Yulong Chen, Zhongqing Wang, and Yue Zhang. Constituency parsing using Iims. Xrv preprint ar Xxi::2310.19462, 2023.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multi- modal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023, 2023.
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabili- ties of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:://ppeerevieww www.:neetfoorm222m /ttpee////////////////////////o/
Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7422-7439 Apr. 2020. doi: 10.1609/ v4ii.5341 URL https://ojs. aaai. http:////eeeee.p/////////raiiaia
Rishi Bommasani, Kathleen A Creel, Ananya Kumar, Dan Jurafsky, and Percy S Liang. Pick- ing on the same person: Does algorithmic monoculture lead outcome homogenization? Advances in Neural Information Processing Systems, 35:3663-366 3678 2022.
10

---

Published as a conference paper at COLM 2024
Ali Borji. A categorical archive of chatgpt failures. Xiv preprint arXio:2302.03349, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tiees Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL ht httpt::tttphoott http::///cocccc......ccooo neurips ips. cc/ cc/paper/2020/ hask/1457cc06bbfcc4667444888 (bce4441abbbaaac1ccc (attaaacc44644aa-- Abstraatbbb................a.... html.
neur
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss,, William H. Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Cart, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. ArXiz, 0/5/21007.0330 2021. URL https: /appi.apppp... www.nticschol..e... org/ CorpusID: C 235755472.
Yulong Chen, Yang Liu, Ruochen Xu, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Yue Zhang. UniSumm and SummZoo: Unified model and diverse benchmark for few-shot summarization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Anmual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 12833-12855, Toronto, Canada, July 2023. As- sociation for Computational Linguistics. doi: (0.18653/v1 200.//2222clcoooggee https://aclanttoolgg.ooo org/2023 acl - long. 718. 718.
URL
Bharath Chintagunta, Namit Katariya, Xavier Amatriain, and Anitha Kannan. Medically aware gpt-3 data generator for medical dialogue summarization. In Machine Learning for Healthcare Conference, pp. 354-372. PMLR, 2021.
Christopher Clark, Kenton Lee, Ming-Wes Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. 10000000000000000000 arviv preprint XXiv:1905.1004. 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, abs/ abs/2110.14168, 2021. URL https: httpss//ppp.ssemaniicccccoooaa wwtpolaaarrr//orrrroeeeeee wrw./orppssI................ /ttpCCorpuDD::29998555 239998651.
semanticscholar
Chris Cummins, Volker Seeker, Dejan Grubisic, Mostafa Elhoushi, Youwei Liang, Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Kim Hazelwood, Gabriel Synnaeve, and Hugh Leather. Large language models for compiler optimization. ArXit, abs/ 2bs00220077 2023.
doi: 10.48550 arXiv.2309.00066
Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Wei Liu, Ninghao Liu, et al. Auggpt: Leveraging chatgpt for text data augmentation. arXiv preprint arXio:2302.13001 20233
Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu, and Shafiq Joty. Data augmentation using Ilms: Data perspectives, learning paradigms and challenges. arXiv preprint ar arXiv:2403.022: 02.999990 2024.
11

---

Published as a conference paper at COLM 2024
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arxiv preprint ar Xiv:2444 21783, 2024.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezi Yang, Joseph E Gonzalez, and Trevor Darrell. Diversify your vision datasets with automatic diffusion-based augmentation. Advances in neural information processing systems, 36:79024-79034, 2023.
Alexander R Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. Qafacteval: Improved qa-based factual consistency evaluation for summarization. arXiv preprint arXiv:2112.08542, 2021.
Ed Felten, Manav Raj, and Robert Seamans. How will language modelers like chatgpt affect occupations and industries? Xiv preprint ar axXi::2003001157, 2023.
Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah Goodman. Under- standing social reasoning in language models with language models. Advances in Neural Information Processing Systems, 36, 2024.
Mingqi Gao and Xiaojun Wan. Dialsummeval: Revisiting summarization evaluation for dia- logues. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, pp. 5693-5709, 2022.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News summarization and evaluation in the era of gpt-3. ar Xiv preprint arXiv:2209.12355 2022.
Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, Yujiu Yang, Tsinghua University, and Microsoft Research. Connecting large lan- guage models with evolutionary algorithms yields powerful prompt optimizers. ArXiv, abs/2309.0855 2023. doi: 10.48550/ arXiv:330900855. 08532..
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a. URL
net/forum?id=
d7KBjmI3GmQ.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Dalasels and Benchmarks Track (Round 2), 2021b. URL https::/ wttppppwwwwwwweweeeeeeeeeeeeee net het/foorumidd77yywt2nnne
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint ar Xii:::10060968880 2021.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint ar arXio:2305.08322) 2023.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representa- tion hypothesis. arviv preprint ar Xxx::2205500787, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arviv preprint ar Ntt::401.040888 2024.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Compulational Linguistics, 7:453 466, 2019.
12

---

Published as a conference paper at COLM 2024
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-rromoting objective function for neural conversation models. ariio preprint arXic:1510.03055, 2015.
Xiao Li, Yawei Sun, and Gong Cheng. Tsqa: tabular scenario based question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 13297-13305, 2021.
Zekun Li, Wenhu Chen, Shiyang Li, Hong Wang, Jing Qian, and Xifeng Yan. Controllable dialogue simulation with in-context learning, arViv preprint xriv::2200001185, 2022.
Yin-Ting Lin, Alexandros Papangelis, Seokhwan Kim, Sungjin Lee, Devamanyu Haz- arika, Mahdi Namazifar, Di Jin, Yang Liu, and Dilek Hakkani-Tur: Selective in-context data augmentation for intent detection using pointwise v-information. Xrv preprint ar XrXi:22022.0506, 2023.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. Gpteval: Ngg evaluation using gpt-4 with better human alignment. arXiv preprint arXio:2303.16634, 2023.
lya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint ar arXiv:1711.05101, 2017.
Yu Meng, Martin Michalski, Jiaxin Huang, Yu Zhang, Tarek Abdelzaher, and Jiawei Han. Tuning language models as training data generators for augmentation--enhaned few-shot learning. In International Conference on Machine Learning, pp. 24457-24477. PML.R, 2023.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gülçehre, and Bing Xiang- Abstractive text summarization using requence-to-seqeent RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pp. 280-290, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/vii 11/1161008..
URL https: //tpaaaaaaaaaaaaaaaaaaaaaa orgooog (etg///166110 1028.
Hugo Gonçalo Oliveira. A survey on intelligent poetry generation: Languages, features, techniques, reutilisation and evaluation. In Proceedings of the 10th international conference on natural language generation, pp. 11 11-20, 2017.
OpenAl. Gpt-4 technical report. arxiv, pp. 2303-08774, 2023.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt general-purpose natural language processing task solver? ar arv preprint ar xxXin:2302..66470 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. ar preprint ar arXio:1606.05250 2016.
Gaurav Sahu, Pau Rodriguez, Issam H Laradji, Parmida Atighehchian, David Vazquez, and Dzmitry Bahdanau. Data augmentation for intent classification with off-the-shelf large language models. arXiv preprint ar XrXiv:2004.01959, 2022.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. Xrxiv preprint ar arXic:1508.07909 2015.
Ruihao Shui, Yixin Cao, Xiang Wang, and Tat-Seng Chua. A comprehensive evaluation of large language models on legal judgment prediction. In Houda Bouamor, Juan Pinoo, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7337-7348, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10.18653 v1 http/2222........... fiw.diiggs-emnlp....... URL https: /ccanthoooogyyyyoooooooo wwwpogg...oooggoooooooooooo org/2023. mandings-emnlpp 490... 490.
13

---

Published as a conference paper at COLM 2024
Ondrej Skopek, Rahul Aralikatte, Sian Gooding, and Victor Carbune. Towards better evaluation of instrutiionffolooiing A case-study in summanization. In Jing Jiang, David Reitter, and Shumin Deng (eds.), Proceedings of the 27th Conference on Compu- tational Natural Language Learning (CoNLL), pp. 221-237, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ v1 /12203..on...1..6. URL https://aclaatthllog org/ 1rtp://..00r/2222...111111111
conll-1. 16.
Madhvi Soni and Jitendra Singh Thakur. A systematic review of automated grammar checking in english language. Xiz preprint ar axXiv:1804.00540, 2018.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Cholamidavoodi,, Arfa Tabassum, Arul Menezes, Arun Kirubara- jan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong- Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhski,, Fanyue Xia, Fate- meh Siar, Fernando Martinez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambat- tista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Ha- jishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, lan Ng, Isaac Noble, Jaap Jumelet, Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoñ, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosin- ski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Bur- den, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Centtreras-cchande Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova,
14

---

Published as a conference paper at COLM 2024
Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. lyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Óli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mo- hammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pacchhhiar,, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Srihar- sha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmmkumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yid- ing Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether thain-ff-thought can solve them. Xrv preprint arX.v22210.09261, 2022.
Qingyu Tan, Hwee Tou Ng, and Lidong Bing. Towards benchmarking and improving the temporal reasoning capability of large language models. arXio preprint ar arXiv:2306.08892, 2023a.
Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. Can chatgpt replace traditional kbqa models? an in-depth analysis of the question answering performance of the gpt Ilm family. In International Semantic Web Conference, pp. 348-367. Springer, 2023b.
Zhen Tan, Alimohammad Beigi, Song Wang, Ruocheng Guo, Amrita Bhattacharjee, Bohan Jiang, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation: A survey. ar arv preprint 2002::2402. 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini family of highly capable multimodal models. Xiv preprint arXin:2312.11805, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al.
15

---

Published as a conference paper at COLM 2024
Gemma: Open models based on gemini research and technology. arXio preprint (rXio:2403.08295, 2024.
Petter Törnberg. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning, arviv preprint ar Xiio:2304.06588 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar Xtt:2332.13977,, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ar preprint arXiv:2307 009088, 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, 201 2017/2as1/ 7/hash/
2017. URL hptppee//proeceiigggg htwp:oceeeeinnnnneeiiirsesseeese CA, USA, pp. 1 3f5ee243547dee91
html.
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. Advances in Neural Information Processing Systems, 36, 2024.
Zezhong Wang, Fangkai Yang, Lu Wang, Pu Zhao, Hongru Wang, Liang Chen, Qingwei Lin, and Kam-Fai Wong. Self-guard: Empower the Ilm to safeguard itself. arv preprint ar a:Xt:2330..1881, 2023.
Yotam Wolf, Noam Wies, Yoav Levine, and A. Shashua. Fundamental limitations of align- ment in large language models. ArXiv, abs/2304.11082, 2023. doi: 10.48550/ 10.48550/arXiv.230 arX.v.2300 11082.
Jianhao Yan, Yun Luo, and Yue Zhang. Refutebench: Evaluating refuting instruction- following for large language models. arXiv preprint arXie:2402.13463, 2024.
Chengrun
Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. ArXiv, abs 2309/0349903 2023. doi:
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdi- nov, and Christopher D Manning, Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380, 2018
Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking Ilms via uncertainty quantification. arviv preprint Xit::2401.127999 2024.
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don't know? pp. 8653-8665, 2023. doi: 10.48550/ att.://3............... 18153.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? Anna Korhonen, David Traum, and Lluis Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10116553v11/19-111
URL ht http://aaaaaaalaaoaaaea/ wwwpogg..o...... 0742
Biao Zhang, Barry Haddow, and Alexandra Birch. Prompting large language model for machine translation: A case study. In International Conference on Machine Learning, pp- 41092-41110. PMLR, 2023a.
16

---

Published as a conference paper at COLM 2024
Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. Luq: Long-text uncertainty quantification for lims. arXiv preprint ar arXiv:2403.202799 2024.
Xiaotian Zhang, Chun yan Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on gaokao benchmark. ArXiv, abs/2305.12474, 2023b. doi: 10.48550/ /at/aariviv..1122.2. 12474.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's 's song in the ocean: a survey on hallucination in large language models. ar Xiv preprint ar Xio:2309.002219 2023c.
Chujie Zheng, Sahand Sabour, Jiaxin Wen, Zheng Zhang, and Minlie Huang. Augesc: Dialogue augmentation with large language models for emotional support conversation. arXiv preprint ar Xrio:2202.13047, 2022.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging Ilm-as-a-judge with mt-bench and chatbot arena, 2023a.
Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. Why does chatgpt fall short in providing truthful answers? ariiv preprint ar Xiv:2333 10513, 2023b.
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards unified nulti-dimensional evaluator for text gen- eration. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2023- 2038, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclantthlll
org/2022.... emnlp-main. 131.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. Xiv preprint arXiiv:304.06364, 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-folloiing evaluation for large language models. Xiv preprint ar Xxit:2311.07911 2023.
Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. Can chatgpt reproduce human-generated labels? study of social computing tasks. arXiv preprint 2rXi0:2304.10145, 2023.
A Related Work
Evaluating LLMs. Recently, model evaluation has gained increasing attention from the communities (Zhang et al., 2023c; Fabbri et al., 2021; Gao & Wan, 2022; Bai et al., 2023; Zhong et al., 2022; 2023; Huang et al., 2023; Bang et al., 2023; Chen et al., 2023). Before the surge of LLMs, evaluation protocols mainly focus on evaluating a certain aspect of a single task (Zheng et al., 2023b; Goyal et al., 2022). Evaluation in LLMs can be broadly divided into two categories. A line of work aims to evaluate the comprehensive performance, either on a large set of tasks (Qin al., 2023; Bang et al., 2023; Zhong et al., 2023; Hendrycks et al., 2021a; Srivastava et al., 2023), or evaluate on representative datasets of core scenarios, such as general instruction-rrsponsee (Dubois et al., 2023; Zheng et al., 2023a). Another line of research focuses on specific capabilities of LLMs, such as MATH (Cobbe et al., 2021; Hendrycks et al., 2021b), reasoning (Zellers et al., 2019; Bisk et al., 2020), instruction following (Zhou et al., 2023; Yan et al., 2024), coding (Chen et al., 2021), etc. Our work does not fall into these two categories but rather focuses on locating the specific limitations of LLMs.
There has been work that highlights the intrinsic limitation of LLMs (Yin et al., 2023; Wolf et al., 2023). For example, Yin et al. (2023) explore LLMs' self-knowledge, revealing the limits of their knowledge compared to human proficiency. Wolf et al. (2023) investigate the fundamental limitations of aligning L.M behaviours with human intentions. However,
17

---

Published as a conference paper at COLM 2024
they still focus evaluating LLMs on certain tasks. In contrast, find challenging tasks (error patterns) from data in the wild.
Recently, researchers have explored using LLMss as annotators for creating and augmenting data their interest, using both zero-shot and few- shot learning (Ding et al., 2024; Tan et al., 2024). For example, Dai et al. (2023); Anaby-Tavor et al. (2020); Sahu et al. (2022); Lin et al. (2023); Törnberg (2023); Zhu et al. (2023) use LLMs such ChatGPT and GPT-4 to label raw text for text classification tasks, and Chintagunta et al. (2021); Zhang et al. (2023a) use LLMs to generate output texts for text generation tasks, such as machine translation and text summarization. Additionally, Anaby-Tavor finetune GPT-2 with small data to annotate new training data for text classification tasks. Li et al. (2022); Zheng et al. (2022); Meng et al. (2023) use LLMs to augment dialogues with few-shot learning.
Data Augmentation using LLMs.
al. (2020)
Our work falls the above studies that use LLMs for data augmentation. However, different from those that mostly use LLMs to augment analogous data or label output text, we use LLMs to obtain diverse queries for specific task under the guidance of patterns (instruction) generated by LLMs themselves.
Using LLMs as Optimizers. Our Self-Challenge framework involves using LLMs as opti- mizers to modify the patterns. Similarly, Yang et al. (2023) introduce the optimization by prompting, which employs LLMs for generating solutions in natural language. Cummins et al. (2023) explore the application of LLMs in optimizing code. Guo et al. (2023) integrate evolutionary algorithms with LLMs for discrete prompt optimization.
B Details of Prompts
B.1 Summarization Prompt
We first ask GTP-4 to analyze the error instances:
Given the below question, incorrect model answerr, and correct answer, please ana- lyze why and how this question induces GPT-4 to generate an incorrect answer. Do not provide suggestions on how improve GPT-4 performance on such questions.
iQuestion: {q:}; GPT-4 Answer: {ri}; Correct Answer: {ci})
...
where { {ri} and } are the i-th query, model response and correct answer.
Then, ask GPT-4 to summarize possible patterns of challenging questions using the below prompt:
Below are some questions, where GPT-4 fails to generate correct answers, and corresponding analysis why and how the question challenges GPT-4:
(Question: {q: ): Analysis: {ai}]
According to the questions and analysis, please summarize the most common patterns of the above questions, which can induce GPT-4 to generate incorrect answers. Please note that:
a. The patterns can describe features from different cognitive levels in those questions, for example, syntax, semantics, pragmatics, or even the knowledge acquisition in GPT-4.
b. Each above question can contain multiple patterns.
c. Each summarized pattern should at least appear in one of the above questions.
d. For each summarized pattern, give clear and detailed explanation and analysis
to make the summarized pattern understandable.
e. Each pattern should be formulated in a reproducible way. Following each pattern, please give suggestions on: when want to generate questions using the
18

---

Published as a conference paper at COLM 2024
summarized patterns, how can we generate those questions, which contain such putterns and induce GPT-4 to fail?
where {ai} is the analysis to the i-th query.
B.2 Generation Prompt
We use the below prompt to generate new queries:
Below is a pattern, which is summarized from set of questions that induce GPT-4 to generate incorrect answers.
Pattern: [P:}
Please generate 10 new questions that contain such a paltern in the {domain) } domain. Please note that:
1. The question should induce GPT-4 to generate incorrect answers.
2. The question should be answered by short sentence or phrase. The answer should not be complicated, for example, not paragraph of hundreds of words.
3. The new questions should be diverse, and are different from each other in terms of their main contents, entities, events, and question types. Do not generate questions consistently asking the same thing, or having large lexical overlaps.
4. You should generate more diverse and novel questions of different types, as long as those new questions contain the pattern and successfully induce GPT-4 generate incorrect answers.
5. Please generate questions with entries (e.g., "1.") and do not generate possible answers or generate pafterns again.
where {P:} is individual pattern (full pattern with name and description), and {domain) is a domain word, such as Martin Nolan.
B.3 Optimization Prompt
After collecting model responses and their corresponding human feedback, we use the below prompt to optimize the initial pattern:
Please first read the below original pattern, which is summarized from questions where GPT-4 fails, and their paired example questions.
Pattern: Pt
Below are new questions, which are generated using the above pattern and should induce GPT-4 to fail, GPT-4 answers those new questions and their correspond- ing human feedback:
(Question: {q:); GPT-4 Answer: (ri}; Human Feedback: {fi}]
Given the new questions which are generated using that pattern, GPT-4 answers, and human feedbuck, please:
1. Check whether the new questions successfully induce GPT-4 generate incorrect answers: if not, analyze reasons why GPT-4 can successfully generate correct answers for those new questions, which are supposed to induce GPT-4 to generate incorrect answers.
2. Then consider how to modify the pattern that it can be used to generate questions where GPT-4 is more likely to fail. In particular, you should treat the questions where GPT-4 answers correctly as negative samples.
The modification can include but are not limited to:
Modify a vague pattern into a more fine-grained and specific pattern. Add descriptions on what types of questions that GPT-4 can answer correctly, and how we should avoid them when generating questions.
19

---

Published as a conference paper at COLM 2024
However, please ensure that questions generated by new patterns should be an- swerable and can be answered by a short phrase.
where {qi}, {ri} and {fi are the i-th query, model response and human feedback.
B.4
Evaluation Prompt
Given a query and its correct response, we evaluate the model outputs by using the below prompt:
Please first read the below question, its correct answer and model-generated
answer:
Question: {question}
Correct Answer: {correctresponse)
Model--emerated Answer: (modelresponse)
Is the model- model-generated answer correct or incorrect? Please only output "correct" or "incorrect".
C Details of Annotation
C.1 GPT-4 Response Verification and Gold Response Annotation
We ask annotators to write gold response to query with the help of GPT-4 response. In particular, given a query, we first prompt GPT-4 to generate a response and then ask the human annotator to check whether the model response is correct or not. In addition to questions and responses, annotators are also presented with corresponding patterns, to help them better understand what the task the LLMs are doing. For example, for the Assumption of Existence questions, responses including "Sorry, cannot find relevant information on [non-existent entities/eventss" would be regarded as "correct".
For responses that contain reasoning steps (such as text manipulation and counting tasks), they should examine whether the reasoning is correct. For example:
Query: In the sentence "Henrik Lundqoist is often regarded one of the best goaltenders in the history of the NHL," identify the total number of words that end with the letter "s" (ignoring case-sensitivity).
GPT-4 Response; In the given sentence, there are words that end with the letter "s": "Lundqvist," " "as," and "goaltenders."
Human Label: incorrect (although there are words ending with "s", GPt- mistakenly take "Lundqvist" as a candidate)
If the response correct, the annotator can use it as the gold response with modification. If the response is found error, the annotator is asked to perform post-editing to correct the model response, which then can be used as the gold response.
When evaluating GPT-4 responses and annotating gold responses, annotators are encour- aged to utilize various tools, including search engines (e.g., Google) and syntax analyzers, to check the factuality and correctness of GPTT4 answers.
C.2 Quality Control
To ensure quality, we require annotators to practice on 20 training samples and achieve satisfactory performance on our assessment test before proceeding to real annotations. We pay extra attention to (1) Incorrect: If any error (in particular factual) is found in GPT4 response, annotators label it as correct (or vice versa); annotated responses contain errors, in particular, annotators incorrect GPT4 responses gold responses without modification; (2) Unreadable: annotated response is oversimplified particularly if annotators modify GPT4 responses, but annotated responses need GPT4 response context to understand.
20

---

Published as a conference paper at COLM 2024
id
Wiki Title (domain)
#
id
Wiki Title (domain)
#
1
Abney Park Cemetery
62
16
Marfin Nolan
50
2
Anatoly Karpov
Maryborough Base Hospital
55
3
Chaka Khan
69
Mixtec language
4
Child Workers in Nepal
67
19
Muirchertach Ua Briain
56
5
Chinese calligraphy
67
20
Murgon State School
39
6
Corruption in Yemen
63
21
Philippine drug
69
Development of COVID-19 tests
Prespa Agreement
8
Edinburgh and Northern Railway
72
23
Ray Cooper
50
9
Emergency medical services in Germany
62
24
Scouting in Illinois
65
10
Henrik Lundquist
77
25
Shirley Chisholm
70
Joey Votto
Stanford University centers & institutes
12
John Mowbray, 3rd Duke of Norfolk
Sunflower Student Movement
13
Josh Shapiro
69
28
The Battles of Coxinga
47
14
Laura Siegemund
61
29
Transportation in Washington, D.C.
71
15
Long QT syndrome
55
30
Zhejiang University
61
Table 5; Domains that are used for generating new queries. #: the count of valid queries in our final (1,835) data.
For the test annotation, require that their accuracy in determining the correctness of GPT-4 responses should reach at least 90%, and 90% of their annotation on gold responses should be reasonable and correct to the queries.
After annotation, we conduct quality control process where we randomly select 10% of the annotations for manual review. If we discover that more than 5% of an annotator' submissions are deemed unsatisfactory, we request the annotator to redo the entire set of annotations meet our quality standards.
C.3 Annotator Background and Compensation
All annotators junior undergraduate students who study in English programs, ensuring they have a solid foundation in reasoning and language necessary for the task. Annotators are compensated by payment based their annotation quantity, which is around 1.4 USD per instance. This compensation structure and rate is applied across both the annotation of human-in-loop feedback and the construction of the SC-G4 benchmark.
C.4 Estimated Human Performance on SC-G4
We provide three estimated human performance (binary-label) on SC-G4 benchmark. (1) We measure untrained annotators' performance by calculating their first trial annotation on 20 instances. annotators were involved (1 did not pass our follow-up test annotation). Their Fleiss' Kappa score is 0.56 (moderate agreement). The average accuracy by majority vote is 81.00%. (2) We measure their trained performance by evaluating their test annotation on 100 instances, where we provide gold labels. Their averaged accuracies are 90.20% (w who did not pass) and 92.50% (wo), respectively. (3) We measure their formal annotation performance by comparing their first round of formal annotation, and the final data (gold) after checking and revision. The accuracy is 96.73%.
D Seed Instance Collection
We collect those failure cases from two main sources: (1) papers that investigate the GPT- failure cases (Zheng et al., 2023b; Borji, 2023; Qin et al., 2023; Tan et al., 2023a) The original sources of those papers include HotPotQA (Yang et al., 2018), TSQA (Li et al., 2021), NQ (Kwiatkowski et al., 2019), BigBenchHard (Suzgun et al., 2022), BoolQ (Clark et al., 2019). (2) failure cases from online (twitter) and our usage. For example:
21

---

Published as a conference paper at COLM 2024
"If knaves always lie and knights always tell the truth, and you ask a person if they are a knight and they say no, what are they?"
For cases presented via screenshots, we manually convert them into text. All data are then tested using our GPT-4 API.
Full Domain Information
Table presents the full domains that we use to generate new queries in the SC-G4 bench- mark and the number of valid queries of individual domains. These domains are Wikipedia titles, which are randomly selected from Wikipedia metadata (2023).
F
GPT-4 Error Patterns
We present the full GPT-4 error patterns, with each coupled with their ID and initial patterns for comparison. We show the major difference in italics.
E1
Pattern A
Initial Pattern A
Assumption of Existence: GPT-4 may generate incorrect answers when question assumes the existence non-existent entity or concept. This can lead the model to provide answer based on incorrect assumptions associations. Such entities or concepts are created using the combination of existing ones.
To generate questions containing this pattern, create the existence of a non-existent entity or concept by slightly modifying existent entities, combining multiple existent entities into one, and then ask questions that assume the existence of such non-existent entities concepts within real context.
Final Pattern
Assumption of Existence: GPT-4 may generate incorrect answers when question assumes the existence of non-existent entity or concept. This can lead the model to provide answer based on incorrect assumptions associations. Such entities or concepts are created using the combination of existing ones.
To generate questions containing this pattern, create the existence a non-existent entity concept by slightly modifying existent entities, combining multiple existent entities into one, and then ask questions that assume the existence of such non-existent entities concepts within real context. Additionally, make the questions more specific and detailed, so that GPT-4 is more likely to generate existent entity or concept, rather than recognizing its non-existence. For example, instead of asking about the impact of a non-existent study, ask about specific findings or methodologies used in the study. This will make it more challenging for GPT-4 recognize the non-existence of the study and may lead to incorrect answers.
answer based on the assumed existence of the non-
F2
Pattern B
Initial Pattern B
Bias towards More popular or Well-known Information and Overgeneral- ization: GPT-4 might be more familiar with popular or well-known topics
22

---

Published as a conference paper at COLM 2024
due to their frequency in the training data. This could lead the model to generate answers related to more popular topics, even if they are incorrect. Also, GPT-4 might overgeneralize from its knowledge of a topic, leading it to choose a more well-known or common answer instead of the correct one. To generate questions with this pattern, create questions that involve less well-known aspects of popular topics or require the AI differentiate between popular and less popular information. To generate questions with overgeneralization create questions that require the Al to avoid common assumptions or generalizations and focus on specific details or less well- known aspects of a topic. Or focus on subjects or situations that share similarities with other, more well-known examples, but have distinct differ- ences that GPT-4 might overlook.
Final Pattern B
Bias towards More popular Well-known Information and Overgeneral- ization: GPT-4 might be more familiar with popular or well-known topics due to their frequency in the training data. This could lead the model to generate answers related to more popular topics, even if they are incorrect. Also, GPT-4 might overgeneralize from its knowledge of topic, leading it to choose more well-known or common answer instead of the correct one.
To generate questions with this pattern, create questions that:
1. Involve less well-known aspects of popular topics require the AT to differentiate between popular and less popular information, but also ensure that the information is not outdated. And avoid asking questions with words like "unsung" or "lesser well- well-known""
2. Require the AI to avoid common assumptions or generalizations and focus on specific details or less well-known aspects of a topic.
3. Focus on subjects or situations that share similarities with other, more well-known examples, but have distinct differences that GPT-4 might over- look.
4. Imoolve information that is not easily accessible or requires a deeper understand- ing of the subject matter.
5. Avoid questions that can be answered by general knowledge or information that is widely available
Ensure that questions generated by the new pattern are answerable and can be answered by short phrase
E3 Pattern C
Initial Pattern C
Complex Counting or Identification Tasks: GPTT44 may struggle with count- ing tasks, leading to incorrect answers. Examples: "How many words in the following sentence have an odd number of letters?", "How many characters are in this sentence, not counting spaces and punctuation?", and "How many words in this sentence contain exactly three vowels?"
GPT-4 may struggle with accurately identifying specific letters or characters within words. Example: "How many words in the following sentence start with a consonant?"
Final Pattern
Complex Counting or Identification Tasks: GPT-4 often struggles with accurately counting or identifying specific elements in complex sentences lists. This can lead to incorrect answers when the question requires counting words, characters, or instances of a specific criterion in a more intricate context.
23

---

Published as a conference paper at COLM 2024
Create questions that require GPT-4 to count identify specific elements in complex sentences or lists. This can include tasks like counting words with a specific number of letters in long sentence, identifying words that start with specific letter in a paragraph or that contain a specific letter at specific position, or counting characters in sentence with multiple punctuation marks and special characters. Avoid simple and straightforward questions that GPT-4 can easily answer correctly.
E4 Pattern D
Initial Pattern D
Complex Logical Reasoning and Paradoxes: GPT-4 may not be able to han- dle questions involving paradoxes or complex logical reasoning. Example: "If knaves always lie and knights always tell the truth, and you ask person if they are knight and they say no, what are they?"
Final Pattern
Complex Logical Reasoning and Paradoxes: GPT-4 may struggle with questions involving complex logical reasoning, hypothetical scenarios with multiple variables, paradoxes. These questions often require more phisticated understanding of logic and reasoning than the model capable of handling.
Example: "If knaves always lie and knights always tell the truth, and you ask person if they are knight and they say no, what are they?"
Focus on questions that involve multiple variables and logical steps that need to be connected in order to arrive at a correct answer.
Example: "Imagine a scenario where the Edinburgh and Northern Railway has rule that trains can only stop at stations with nrmbeenumbered platforms, and every station has a platform number that is doubled each time a train arrives. If train starts at platform 1 and continues to the next station with prime-numbered platform, what would be the platform number of the 4th station stops at?"
Create hypothetical scenarios that involve conditions or constraints that make the question more challenging to answer.
Include paradoxical elements or contradictions in the question that make it difficult to provide straightforroard answee.
Avoid questions that can be answered with general analysis by providing relevant information about a specific topic, as GPT-4 is likely to answer these correctly.
E5 Pattern E
Initial Pattern E
Complex Syntactic Structures and Multiple Clauses: GPT-4 struggles to accurately parse sentences with complex structures and multiple clauses, leading to incorrect identification of specific words or relationships between words.
To generate questions that challenge GPT-4, create sentences with multiple clauses, embedded phrases, and complex relationships between words and ask GPT-4 to parse the sentence or identify specific words or relationships between words. This will make it difficult for GPT-4 parse the sentence accurately and identify the correct answer
Final Pattern E
Complex Syntactic Structures and Multiple Clauses: GPT-4 struggles to accurately parse sentences with complex structures and multiple clauses,
24

---

Published as a conference paper at COLM 2024
leading to incorrect identification of specific words or relationships between words.
To generate questions that challenge GPT-4, create long and complex sen- tences with multiple clauses, embedded phrases, and complex relationships (such as dependency or constituency) between words. Then, ask GPT-4 to parse the sentence, identify specific words or dependency or constituency relationships between words, or rephrase the sentence.
For example: 1. "I'm student very close to the VCU area that's looking for an apartment or house to move in to with another person/people. What are the words in the previous sentence, which form constituency with the second word that begins with letter 'a'?
2. "UInfotunatell, I currently have no funding capacity to advise interns or visiting students. Please consult the EdiNLP web site for more information potential supervisors." What are the words in the previous sentence, which hold a dependency with "funding" or what are the predicative words in the above sentences?
This will make it difficult for GPT-4 parse the sentence accurately and identify the correct answer. Avoid fact-based questions that do not require complex linguistic analysis.
E6
Pattern F
Initial Pattern
Recursive and Unusual Patterns: Questions that require recursive opera- tions, such as replacing characters within word multiple times, can be challenging for GPT-4. Example: "What is the outcome of replacing all the 't's in the word 'tomato' with "potato'?"
GPT-4 might struggle with questions involving unusual patterns or tasks that require non-standard processing. Examples: "How many words in the following sentence have an odd number of letters?", "How many words in this sentence contain exactly three vowels?", and "How far is Aigre located north of Angoulême?"
Final Pattern F
Recursive and Unusual Patterns: GPT-4 might struggle with questions involving unusual patterns or tasks that require ntn-standard processing, such as recursive replacements unconventional counting tasks.
a. Increase the complexity of the unusual patterns or tasks, such as incorporating more steps or multi-level replacements in the questions.
b. Combine multiple unusual patterns or tasks within single question to increase the difficulty.
c. Focus on generating questions that require uncorventional counting tasks or identifying patterns across multiple words or phrases.
By increasing the complexity and incorporating multiple unusual patterns or tasks within a single question,
generate questions that are more likely to induce GPT-4 to generate incorrect answers. However, it is essential to ensure that the generated questions remain answerable and can be answered by short phrase.
E7
Pattern G
Initial Pattern G
Temporal Ambiguity Confusion with Specific Events: GPT-4 might have difficulty understanding the significance of specific time frames mentioned in the question and could provide information about the subject's careet
25

---

Published as a conference paper at COLM 2024
life in general, rather than focusing the specific time frame. To generate questions with this pattern, include precise time frames (e.g. specific months years) that might be challenging for GPT-4 to pinpoint accurately.
Final Pattern
Temporal Ambiguity or Confusion with Specific Events: GPT-4 might have difficulty understanding the significance of specific time frames mentioned in the question and could provide information about the subject's career or life in general, rather than focusing on the specific time frame. To generate questions with this pattern, include precise time frames (e.g., specific months or years. But do not be too specific, e.g., specific dates or hours) that might be challenging for GPT-4 to pinpoint accuralely, and also mention specific events or accompliihmmets that occurred during those time frames. This will make it more difficult for GPT-4 provide a correct answer without addressing the specific event or accomplishment.
E8
Pattern H
Initial Pattern H
Text Manipulation or Transformation: GPT-4 can make errors when per- forming text manipulation or transformation tasks, such as replacing letters, shifting letters in the alphabet, or sorting words. These errors can be due to the complexity of the task or limitations in GPT-4's processing capabilities.
Create questions that require GPT-4 to perform complex text manipula- tion or transformation tasks, such replacing specific letters with others, shifting letters in the alphabet, or sorting words alphabetically.
Final Pattern
Text Manipulation or Transformation: GPT-4 can make errors when per- forming complex text manipulation or transformation tasks, such as replac- ing specific letters with others based certain conditions, shifting letters the alphabet by varying amounts, or sorting words based on multiple criteria. These errors can be due to the complexity of the task limitations in GPT-4's processing capabilities.
Create questions that require GPT-4 to perform complex text manipulation or transformation tasks, such as:
Replacing specific letters with others based on their position in the word the alphabet.
Shifting letters in the alphabet by different amounts depending on their position in the word or other conditions.
Sorting words based on multiple criteria, such as alphabetically and by length.
Avoid questions that involve simple text manipulation tasks, such as sorting words alphabetically or replacing a single letter with another. To increase the difficulty, ask questions that combine multiple manipulation or transformation.
G Self-Challenge Llama-3-70B
We apply our framework to Llama-3-700 (Dubey et al., 2024) and discover 19 patterns (due to limited space, we only show the pattern names) as shown in Table 6.
However, find that (1) those patterns mostly covered by GPT-4. For example, GPT-4
pattern B covers Llama patterns H and J, GPT-4 pattern covers Llama patterns O, Q
and S. (2) some GPT-4 patterns are not found by Llama3. For example, GPT-4 patterns C
26

---

Published as a conference paper at COLM 2024
ID
Pattern Name
LA
Misleading context with provided date
L-B
Irrelevant information with date
L-C
Lack of explicit current date Inconsistent or implicit information
L-D
LA
Assumption of Existence
LA
Ambiguity and Vagueness
L-G
Lack of Context
LA
Overfitting and Bias
L-
Misinterpretation of Syntax and Semantics
LJ
Inability to Handle Novel Unseen Data
LKK
Misinterpretation of Context
L-L
Incomplete or Partial Analysis
L-M
Misunderstanding of Instructions
LAN
Limited Knowledge Training Data
LO
Linguistic Complexity
L-P
Format or Presentation Errors
L-Q
Complex Sentence Structures
L-R
Pronoun Resolution Dependency Identification
LS
Table 6: list of names of patterns discovered by Llama-3-70B.
(Complex Counting or Identification Tasks), D (Complex Logical Reasoning and Paradoxes), H (Texl Manipulation Transformation).
We further optimize Llama pattern E. We then find Llama-3-70B cannot generate queries that induce Llama-3-70B hallucinate.
H Examples of New Queries
Table 7 presents the examples of new queries for each pattern. We see that those queries contain the features of corresponding patterns, and are diverse in domains.
I Evaluation Validation
To validate the effectiveness, we first compare the gold labels and model-evaluate labels of GPT-4 responses on the full 1,835 instances, where GPT-4 evaluation achieves 85.35 precision, 95.89 recall and 90.31 F-1. As GPT-4 evaluation can be biased to GPT-4 output (Liu et al., 2023), we further construct small that consists of 115 instances, where each gold response is completely annotated by human annotators without the assistance of GPT-4. The GPT-4 evaluation achieves 87.14 precision, 96.83 recall and 91.73 F-1 on this set, which we consider is good enough. Such a higher-recall and lower-precision evaluation indicates that GPT-4 highly reliable in identifying correct answers but with a few errors in classifying incorrect responses as "correct". It also suggests that the model performance evaluated by GPT-4 can slightly overestimated or higher than its real performance.
We also experiment with the evaluation method following Wang et al. (2023), i.e., asking LLMs to explain why the model-generan response is correct or not, and then give a label for the response. However, we do not observe significant improvement.
J Analyses of Model Performance on Different Domains
Figure 3 shows the breakdown of model performance on different domains. Generally, GPT- shows the best performance on all domains, and gives the most smooth curve compared with other models. In contrast, Llama 2 models show worse performance, and give more sharp curves, indicating they are less stable and generalizable across domains, compared with GPT-4 and Turbo.
27

---

Published as a conference paper at COLM 2024
ID
Pattern Name
Example
Assumption of Existence
During the "Karpov Invitational" tournament, what unique rule did Anatoly Karpov introduce encourage faster and more aggressive play? play?
Bias towards More popular or Well-known Information and Overgeneralization
In the context of the Philippine drug war, which specific law enforcement unit was disbanded in 2017 due to its controversial involvement in extrajudicial killings?
C Complex Counting or Identi- fication Tasks
In the sentence, "Joey Votto is known for his exceptional plate discipline and ability to get on base," how many words have the letter 'e' as the third character and end with the letter 'n'?
D Complex Logical Reasoning and Paradoxes
If Muirchertach Ua Briain had rule that he could only levy taxes on years with a prime number of battles, and the number of battles doubled each year, which year would be the fourth time he levied taxes if his first tax year had 5 battles?
E Complex Syntactic Structures and Multiple Clauses
The entrance of Abney Park Cemetery features elaborate iron gates, designed by the architect William Hosking, that exhibit the Victorian Gothic architectural style. What are the words in the previous sentence that hold a dependency relationship with "iron gates"?
Recursive and Unusual Pat- terns
If the words " "rapid," "accurate " and "scalable" are used to describe the ideal characteristics of COVID-19 test, and you were to arrange these words based on the sum of the positions of their vowels in the alphabet, which word would come first?
G Temporal Ambiguity or Con- fusion with Specific Events
During the fall of 1989, Ray Cooper participated in con- cert tour with famous rock band. What was the name of that band?
H Text Manipulation or Trans- formation
In the context of Martin Nolan's 's work, replace the first and last letters of each word in the phrase "auction house exec- utive director" " with their corresponding opposite letters in the alphabet (A >7, B <- >Y, etc.), and then sort the transformed words by length in descending order.
Table 7: Examples of new queries generated by GPT-4.
Train
Dev
Test
Count of instances
500
325
1,010
Avg Query
34.22
36.94
4.48
Avg Gold R
61.61
63.67
38.19
Avg G
82.40
84.50
73.97
Table 8: Data statistics for fine-tuning data split by difficulty. Avg: averaged token length. R: Response.
K Analysis of Few-shot Performance on Data Split by Difficulty
We investigate whether fine-tuned models can be improved on data that is most challenging to GPTT4. In particular, we take all instances where GPT-4 fails (human evaluation) as test set (1,010), and randomly split the rest data into train (500) and dev sets (325). The statistic is shown in Table 8.
Table 9 shows the results. First, model performance on this most challenging test set compared with on other test sets. This can be because that those instances are very to challenge to GPT-4 itself in terms of diverse
that GPT-4 evaluation can highly overestimate the
28

---

Published as a conference paper at COLM 2024
GPT-41
GP7-4
GPT-35- TuroO
Lams 2 Chan DN
Larr 2 Ohan (3781338
Uana 2.79)
Uama (138)
24
13
16
16
Figure 3: Breakdown analysis of zero-shot performance under different individual domains. The IDs of domains are corresponding to IDs in Table 5 in Appendix E, respectively.
G-4
G-4
L-2"
A
58
0.00
10.34
1.72
29.31
76
0.00
13.16
2.63
3,95
C
180
0.00
3.33
1.11
5.00
D
73
0.00
15.07
1.37
1.37
109
0.00
9.17
5.50
6.42*
10.45
0.75
1.49
G
134
0.00
9.96
2.24
7.46
H
246
0.00
2.85
1.22
0.00
all
1,010
0.00
7.52
1.81
4.95*
Table 9: Model performance data split by difficulty.
factors, including their internal knowledge, and GPT-4 fail to detect errors in model response, and thus be more biased.
Analyses on Pattern A
Pattern A finds that GPT-4 tends to hallucinate when the input queries contain non-existent entities, which are mostly created by combining several existent entities into one. Interest- ingly, fine-tuned models show the most improvement on this pattern, thus we manually investigate further by looking into the model outputs.
Overall, we find that fine-tuned Llama 2 gets higher accuracy on such pattern because has learnt reject answer ("There no ...""..
Take the below case for example:
Query: How did the Edinburgh and Northern Railway's experimental solar-
powered train perform in terms of energy efficiency during its trial run in 1858?
Gold: There was no solar-powered train in 1858 actually. But the first solar cell was invented in 1883 by Charles Fritts.
GPT-4 (zero): I'm sorry, but there seems to be a mix-up in the information you provided. The Edinburgh and Northern Railway was railway company in Scotland that operated from 1847 to 1862. Solar power technology was not available during that time period, and the first solar cell was invented in 1954 by
29

---

Published as a conference paper at COLM 2024
ID
Dist-3
Dist-4
A
87.02
93.93
B
87.71
94.42
C
74.56
84.33
D
81.68
90.76
E
82.03
88.74
F
75.30
86.65
82.62
92.00
46.19
59.41
all
70.23
82.05
Table 10: Distinct scores of SC-G4.
word sorting
Dist-3
Dist-4
wo. prefix
41.63
49.39
prefix
8.72
9.98
Table 11: Distinct scores of BigBench word-sorting task.
Bell Labs. Therefore, it is not possible for the Edinburgh and Northern Railway to have had an experimental solar-powered Irain in 1858. (correct)
Llama 2-Chat (zero): The Edinburgh and Northern Railway's experimental solar-powered train, which ran in 1858, was able to achieve an energy efficiency of around 15%. This means that for every unit of energy used to power the train, it was able to travel approximately 15 units of distance. While this may not seem particularly impressive by modern standards... (incorrect)
Llama 2-Chat (fine-tuned on data randomly-split): There is no record of an Edinburgh and Northern Railway's 's experimental solar-powered train trial run in 1858. (correct)
Llama 2-Chat (fine-tuned on data spilt by domain): Thank you for your question! However, I must point out that there is no record of an Édin- burgh and Northern Railway's experimental solar-powered train in 1858....
(correct)
we see that the fine-tuned Llamas successfully identify that this query can contain non- factual content, which is considered correct, but fails respond informatively as GPT-4 and human. We further investigate the outputs of Llama-2-Chat (7B) fine-tuned on randomly split data, and find that over 50% of queries under pattern A rejected in similar way by it.
Moreover, it is noticeable that the cross-domain fine-tuned Llama (domain split) also suc- cessfully rejects such questions. Recall that in this setting, we constrain the domains in the training and testing sets. In other words, pattern knowledge is shareable while domain knowledge is not. The results indicate that the fine-tuned Llama effectively learns the hid- den features of these patterns and can apply them across different domains. We hypothesize that it is possible that LLMs can sense such uncertainty within themselves (also note the uncertainty can exist across domains). By fine-tuning on such data, LLMs can be better at distinguishing between factual and certain entities and non-factual ones, and refuse to answer questions with non-existent entities or events, rather than generating non-factual responses.
30

---

Published as a conference paper at COLM 2024
M Diversity Evaluation of SC-G4 Queries
We use the Distinct score (Li et al., 2015) to evaluate the diversity of generated queries. We report dist-3 and dist-4 in Table 10. Higher scores indicate the data are more diverse. For comparison, we show the results of the BigBench word-sorting task (bench authors, 2023) in Table 11, which is similar to our pattern H. Generally, we see that queries in SC-G4 are diverse from each other.
31


Reviews:
[{"review_id": "XBX6FXIzZL", "reviewer": "Reviewer_H8gk", "summary": "This paper introduces a self-challenge assessment framework that engages both LLM and humans to iteratively generate more challenging data and identify the limitations of LLM. This work is of the average quality, good clarity, average originality and good significance.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable review.\n\n**‘Self-Challenge’**\n\nWe will revise it by adding *Human-in-the-loop*, referring to the process of LLM finding questions that can challenge themselves with the guidance of humans.\n\n **Annotation**\n\nFor the *unsatisfactory* for each data, we check both label and response:\n\n1. Incorrect: If any error (in particular factual) is found in GPT4 response, annotators label it as *correct* (or vice versa); annotated responses contain errors, in particular annotators use incorrect GPT4 responses as gold responses without modification\n2. Unreadable: annotated response is oversimplified particularly if annotators modify GPT4 responses, but annotated responses need GPT4 response as context to understand\n3. Unnatural: annotators use GPT4 response, but it contains words such as '*As an AI…*'\n\nThe binary-label annotation is in Appen C (first para).\n\n**Why GPT4**\n\nWe used GPT4 as it is regarded as the most powerful LLM on textual tasks. \n\nPlease also see Response to xmpu. We conduct experiments using Llama-3-70B-instruct. But it is less satisfying compared with GPT4.\n\n**Human involvement**\n\nYes, our work involves the process of human-in-the-loop. In particular, after generating/optimizing error patterns, we evaluate the pattern and obtain feedback from human evaluation (as mentioned in Sec2.2 and 3.2). This process cannot be automated as LLMs do not know what is correct and is incorrect.\n\n**4 domain words**\n\nThose 8 patterns are not summarized from 4 domains, but from 189 seed data, and are then optimized under 4 domain words with human feedback. Here the *smooth* of 4 domains is compared with 1. \n\nWe will clarify it as a limitation.\n\n**Pattern selection**\n\nWe select 8 *qualified* patterns from the initial 14 ones. As in Sec 2.2, we evaluate the pattern by evaluating the new queries generated by them. The new queries should meet 2 criteria. 1. they should contain important features of the pattern, e.g. an Assumption of Existence (AOE) query should contain non-existent content. 2. the new queries should be challenging enough, e.g. an AOE query should induce LLM to hallucinate.\n\n**Pattern requirement**\n\nThe requirement of *identify specific ...* refers to the features in pattern (Fig5, Appen E), which is completely generated by the LLM.\n\n**More LLMs**\n\nPlease see Response to xmpu, we benchmark more LLMs.\n\nFor all above issues, we will clarify & highlight them in our revision. We would really appreciate it if you could reconsider our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper explores a question that deserves to be examined\n2. We all know that large models vary in their answers depending on the questions asked, so how they ask questions is critical. The paper proposes an idea of self-challenge, through self-iteration, optimization, and further enable the questions to be asked to the model can be more structured.\n3. Although there is room for improvement, the experiments in this paper are relatively complete and can cover most of the questions that need to be verified.", "reasons_to_reject": "See the above questions", "questions_to_authors": "1. Almost all answers are manually labeled, especially gold answers, which seems to contradict the claim of ‘self-challenge’. Suggested modification of this statement. If it must be manually labeled. Can the criteria for manual annotation be described in detail,. For example, the 5% annotation requirement for quality control is unsatisfactory, what is the standard weighing of unsatisfactory here?\n 2.1 Regarding the optimization part, why did you choose to use GPT-4 to optimize the answers obtained by GPT-4 instead of using other LLMs. According to you, GPT-4 has some inherent flaws that lead to limitations, then to test the optimized results, shouldn't it be better to use other large models, especially after multiple rounds of iterations, instead of just discarding them.\n2.2 Regarding the iterative process you mentioned, doesn't it require a human to critique whether this model is good enough for each iteration, or even to screen the entire iteration, which doesn't seem to be a function of automatic screening? \n3.1 Why does only provide four domain words to smooth the model? And the 8 patterns from these 4 domains  are used throughout all the experimental design. In my opinion, affect more than just the smoothing of the model?\n3.2 Can you introduce the logic of selecting these eight patterns in detail?\n3.3 Where does requirement come from, and how does it relate to patterns, \"identify specific elements in complex sentences\n3.4 Can you introduce in more detail about the content of binary tags, how to annotate, especially for the gold tag binary annotation, I do not see the relevant content of the introduction in Annex C\n4.1 This research direction actually has quite a few researchers working on it at the moment. Can you add some more natural datasets and baseline experiments, comparing other methods as well as practical effects?\n4.2 Regarding the evaluation metrics section, it is recommended to add evaluation metrics from other genres or articles, such as [1], even though you may consider this evaluation metric to be simple or not as advanced as yours on specific issues, like in [1], i.e., evaluating by distinguishing between labels, is relatively simple, but it would add credibility to the readers.\n[1] Wang, Zezhong, et al. \"Self-guard: Empower the llm to safeguard itself.\" arXiv preprint arXiv:2310.15851 (2023).", "extra_scores": {}}, {"review_id": "FsZch2cGQ0", "reviewer": "Reviewer_nH6u", "summary": "This paper presents Self-Chal, a self-challenging evaluation framework where LLMs summarize the error patterns (Step 1), evaluate the error patterns (Step 2), and optimize the patterns (Step 3). During this process, this study engages human experts to evaluate the quality of summarized patterns. This study obtains 8 patterns from 189 failure instances and constructs a new benchmark, Self-Chal, that consists of 1,835 challenging instances in the combination of 8 patterns and 30 domains. The experiments with multiple LLMs show that those error patterns are universal across different LLMs (e.g., Llama 2). In addition, fine-tuning LLMs cannot address these failure cases reliably.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review.\n\n**Human Performance**\n\nWe provide 3 estimated human performance (binary-label) on Self-Chal benchmark. (1) We measure untrained annotators’ performance by calculating their first trial annotation on 20 instances (before training). 5 annotators were involved (1 did not pass our follow-up test annotation). Their Fleiss' Kappa score is 0.56 (moderate agreement). The average acc by majority vote is 81.00%. (2) We measure their trained performance by evaluating their test annotation on 100 instances, where we provide gold-labels. Their averaged acc are 90.20% (w 1 who did not pass) and 92.50% (wo). (3) We measure 4 annotators' formal annotation performance by comparing their first round of formal annotation, and the final data (gold) after checking and revision. The acc is 96.73%.\n\n**Unanswerable Questions**\n\nYes, there are questions that cannot be answered by one gold response. For example: \n```Please combine two names of the centers and institutes at Stanford University in an alphabet order```. \n\nSuch questions are filtered out of our benchmark, and are not used for evaluation.\n\n**Tokenization Issues**\n\nThank you for acknowledging the tokenization tasks are interesting. From a practical view, such tasks can be closely related to applications such as poetry generation, spelling checking, and Teaching Aide in the area of NLP4Education. We also emphasize that such issues reveal the problems of underlying tokenization mechanisms, which can further influence the performance of downstream tasks. Also, it reveals that the current tokenization design in LLMs is different from humans.\n\nLastly, we agree that our methodology mainly relies on prompting LLMs, which has been adapted by previous study [1, 2, 3, 4]. We also highlight that this framework is novel in that we propose a new semi-automatic evaluation protocol that can help humans better mine and understand the error patterns in LLMs.\n\n1. NLG Evaluation using GPT-4 with Better Human Alignment\n2. Better patching using LLM prompting, via Self-Consistency\n3. Tree of thoughts: Deliberate problem solving with large language models\n4. Chain-of-thought prompting elicits reasoning in large language models\n\nWe thank you for your appreciation of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ This paper is well-written and easy to read.\n+ The idea of letting LLMs analyze failure cases semi-automatically (with humans in the loop) and create difficult instances is interesting.\n+ The new benchmark dataset (Self-Chal) may be useful for the development of LLMs.", "reasons_to_reject": "+ The method presented in this paper is mostly about prompting LLMs (limited technical novelty); this may have pros and cons.", "questions_to_authors": "+ Where can I find the human performance on the new benchmark dataset (Self-Chal)? Is there any possibility that this method creates questions that cannot be solved by anyone?\n+ It is interesting to see that LLMs suffer from answering questions about tokenization. At the same time, do we want LLMs to solve these questions (or \"bugs\" of LLMs) (from the practical point of view)?", "extra_scores": {}}, {"review_id": "LGoGY0S2DX", "reviewer": "Reviewer_xmpu", "summary": "The authors propose a Self-Challenge (Self-Chal) evaluation framework designed to uncover weaknesses in LLMs by having LLMs and humans collaboratively identify challenging patterns and generate new instances based on those patterns. They start with seed instances that GPT-4 struggles to answer, prompting GPT-4 to summarize error patterns. These patterns are used to generate new instances, which are iteratively refined with human feedback. The authors present eight patterns, such as text manipulation and questions with assumptions, and create a benchmark, Self-Chal, consisting of 1,835 instances with human-annotated gold responses. Their results show that GPT-4 can correctly answer 44.96% of the Self-Chal instances. They find that these patterns cannot be fully resolved through fine-tuning, indicating that some error patterns may represent inherent flaws in LLMs.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review.\n\n**Self-Challenging of Llama3-70B-instruct**\n\nThe full patterns summarized by Llama3 can be found at https://anonymous.4open.science/r/Self_chal_anonymous-5FE3/README.md\n\nGenerally, we find (1) Llama3 summarized patterns are mostly covered by GPT4. For example, P-B in our paper covers P-8 and 10, P-E covers P-15, 17 and 19. (2) some GPT4 patterns are not found by Llama3. For example, P-C (counting), P-D (logical), P-H (text manipulation).\n\nWe further optimize P-5 and find Llama3 cannot generate queries that induce Llama3 to hallucinate.\n\nWe will include full results in our revision.\n\n**Benchmarking other LLMs**\n\n|LLMs|A|B|C|D|E|F|G|H|all|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Gemma-7B|33.52|7.69|7.76|12.23|15.62|5.65|6.45|0|10.35|\n|Phi3-instruct-7B|12.29|20.36|14.22|35.37|20.70|22.61|7.83|2.21|16.84|\n| mixtral-8x7B | 24.02|25.79|7.33|19.65|26.17|13.48|7.83|0.37|15.15|\n|Llama3-70B-instruct|43.58|31.67|20.26|33.19|32.81|20.87|14.75|0.74|23.81|\n|Gemini-1-pro|22.91|16.82|7.79|16.81|25.78|10.09|7.01|0|13.06|\n|Claude-3-Sonnet|35.75|34.39|20.69|35.37|28.52|24.35|16.59|2.21|23.98|\n|GPT-4o|25.70|41.63|31.03|44.54|41.80|36.52|18.43|10.70|31.17|\n\nGenerally, all LLMs show poor results. Similar trends can be consistent with previous research [1, 2] that LLMs can be homogeneous as they are trained with similar techs (e.g., transformers) on similar data (e.g., Wikipedia).\n\n1. Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization? \n2. The Platonic Representation Hypothesis\n\n**Human Performance**\n\nDue to space limit, please see response to nH6u, where we provide estimated human performance.\n\n**Evaluation Protocol**\n\nWe acknowledge that the validation on 115 data is small. However, this is a supplement to our validation on full 1835 data.\n\n**Diversity**\n\nFollowing your suggestion, we evaluate the diversity of our data using Distinct score. We report dist-3 and dist-4 here. The higher, the more diverse.\n\n|group|dist-3|dist-4|\n|:-|:-:|:-:|\n|A|87.02|93.93|\n|B|87.71|94.42|\n|C|74.56|84.33|\n|D|81.68|90.76|\n|E|82.03|88.74|\n|F|75.30|86.65|\n|G|82.62|92.00|\n|H|46.19|59.41|\n|all|70.23|82.05|\n\nWe compare BigBench word-sorting task, which is similar to pattern H.\n\n|word_sorting|Dist-3|Dist-4|\n|:-|:-:|:-:|\n|wo. prefix|41.63|49.39|\n|w. prefix|8.72|9.98|\n\nIt shows our data are diverse and are of slightly higher diversity compared with similar task.\n\nWe would really appreciate it if you could reconsider our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The Self-Challenge framework offers a relatively unique way to categorize weaknesses in LLMs according to interesting categories through an iterative protocol that harnesses LLMs themselves, rather than relying solely on accuracy. There are some recent similar efforts in the same spirit, e.g., The Generative AI Paradox in Evaluation: \"What It Can Solve, It May Not Evaluate\" (Oh et al, 2024) and WSC+: Enhancing The Winograd Schema Challenge Using Tree-of-Experts (Zahraei and Emami, 2024), which the authors should compare/contrast with in Related Work.\n\n- The paper includes a commendable assessment of LLMs using multiple strategies beyond zero-shot, including few-shot and fine-tuning experiments.", "reasons_to_reject": "**1. Lack of Comprehensive Comparative Analysis:**\n- The paper relies exclusively on GPT-4 for error pattern generation, missing comparisons with other LLMs during the pattern discovery process. This is particularly important, since the idea of their challenge is to pit LLMs against their own recognitions of weaknesses, then LLM-diversity in the generated dataset is crucial.\n- The evaluative experiments include a limited set of models (primarily GPT and LLaMA families), neglecting many other available LLMs, both open-source and proprietary (e.g., Mixtral, Gemma, Phi, Claude 3) that could provide a more comprehensive understanding of error patterns across models.  Because of this, the claim that error patterns are \"universal\" (which the authors often make) across LLMs is too strong given the limited evaluation on just a few families of models.\n- The training data for fine-tuning is too small to draw general conclusions about the impact of fine-tuning on model improvement (although the authors acknowledge this).\n\n**2. Inadequate Experimental Depth:**\n- The experiments lack a proper comparison to human performance on the Self-Chal benchmark.\n- The results focus only on GPT-4-generated questions without sufficient discussion on potential biases.\n- Qualitative error analyses would be very important to enhance the depth of the experiments. \n- Although the evaluation protocol was validated with human labels, the validation set was relatively small, which could lead to inaccuracies in automated evaluation. \n- The paper does not provide a baseline comparison with human performance on the Self-Chal benchmark. Of course, I know that humans were involved in the labelling, but if so, then at the very least,a \"majority vote\" (supported with inter-annotator agreement statistics) using multiple annotators can used to proxy a \"human baseline\". \n\n**3. Insufficient Coverage in the Related Work Section:**\n- The related work section is too brief and currently resides in the appendix, which is problematic (particularly because there are many relevant works that need to be discussed as a matter that is of high priority as the main content of the paper). For example, it lacks coverage of a significant body of literature concerning dataset augmentation and creation using LLMs.", "questions_to_authors": "- The paper introduces the initial list of error patterns derived from GPT-4’s performance. What methodology was used to inform the initial pattern list? Was there a comprehensive and systematic approach, or was it primarily exploratory? Also, are the 8 error patterns (derived after two iterations of refinement) exhaustive in identifying all possible areas where GPT-4 fails, or are there additional patterns that could have been explored?\n\n- I noticed that the prompts (e.g., the summarization prompt in B.1) are quite intricate and have \"disclaimers\" and warnings for the model to very specifically guide it/steer it away from certain behaviors. How did this prompt evolve over the iterations, and were earlier prompts less effective in summarizing patterns? If so,\n\n- Why was GPT-4 specifically chosen for constructing the benchmark, and could other LLMs have been equally effective in generating challenging questions? Does the use of GPT-4 to evaluate its own responses lead to inherent bias in the evaluation, and how could this be mitigated? If there were iterative changes to the prompts, how can you ensure that the final prompts are not simply optimized for gaming the results in your experiments rather than revealing genuine weaknesses in GPT-4 and other LLMs? \n\n- I have noticed that particularly GPT-4 tends to become overly focused on the unimportant specifics of provided examples/details in a prompt, leading to generated instances that are not diverse. How did you address this issue, which I fear might lead to instances in your benchmark that lack diversity and are too similar to one another? If you did combat against it, is there a way you can demonstrate the diversity of the uniqueness of queries in your dataset? On that note, it would be quite important to have a table of examples of instances in the paper so that the reader can \"see for themselves\" the types of instances.", "extra_scores": {}}, {"review_id": "9hpyrsEYAA", "reviewer": "Reviewer_oXPQ", "summary": "This paper proposes a framework to analyze error patterns in mistakes made by LLMs by using analysis by LLMs and feedback by humans. By using the proposed framework, the experiment identifies 8 patterns of errors made by GPT-4. In addition, they introduce a benchmark that includes new queries in the 8 patterns. The analysis on the benchmark shows that GPT-4 makes mistakes in more than 50% of cases and fine-tuning cannot solve the benchmark.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review.\n\n**Seed Instance Collection**\n\nThe full list can be found at https://anonymous.4open.science/r/Self_chal_anonymous-9205/seed_instance.json.\nIn this research, we are motivated by the failure of GPT4, and we collect those failure cases from:\n\n(1) 4 papers that study the failure of ChatGPT(4) [1,2,3,4] (67.7%), where we focus on QA data. \n\n1. Why Does ChatGPT Fall Short in Providing Truthful Answers?\n2. A Categorical Archive of ChatGPT Failures\n3. Is ChatGPT a General-Purpose Natural Language Processing Task Solver?\n4. Towards Benchmarking and Improving the Temporal Reasoning Capability of LLMs\n\nFor example:\n\n```The school in which the Wilmslow Show is held is designated as what?```\n\n```Who did Jessica Valenti work for in 2005?```\n\nThe original sources of those papers include HotPotQA, TSQA, NQ, BigBenchHard, BoolQA.\n\n(2) failure cases from online (twitter) and our usage (32.3%). For example:\n\n```If knaves always lie and knights always tell the truth, and you ask a person if they are a knight and they say no, what are they?```\n\nFor cases presented via screenshot, we manually convert them into text. All data are then tested using our GPT-4 API.\n\nWe will further clarify those sources and provide full seed instances in our revision.\n\n**Evaluation in Limited Tasks**\n\nFirst, we acknowledge that we did not involve traditional NLG tasks, e.g. MT, due to the fact as you mention. We will discuss it as a limitation.\n\nSecond, 8 tasks (patterns) are summarized from 189 data. With more data collected, we believe there would be more interesting tasks (patterns).\nIn fact, the Self-Chal benchmark involves arithmetic reasoning tasks such as Patterns C and D. For example:\n\n```... If a student starts with a paper containing 5 grid squares and each subsequent paper's number of squares is doubled, what will be the total number of characters written on the 4th paper?```\n\n**Using More LLMs for Experiments**\n\nWe use Llama3-70B-instruct, which summarizes 21 patterns. However, we find Llama3 is less effective compared with GPT4: (1) one G-4 pattern can cover multiple L-3 pattern; (2) L-3 patterns cannot be used to generate challenging queries; (3) multiple G-4 patterns are not shown in L-3 patterns. \n\nWe benchmark more LLMs on Self-Chal. Please also see Response to xmpu.\n\n**Suggestion on clarification**\n\nWe will follow your suggestion to revise our paper. We use gpt-4-32k-0613.\n\nWe would really appreciate it if you could reconsider our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes an idea for analyzing mistakes made by LLMs using LLMs. This idea provides a relatively reproducible framework for diagnosing problems in responses from LLMs, beyond evaluating performance such as accuracy on specific tasks. This framework may help future research in creating new LLMs or tuning LLMs for specific applications.", "reasons_to_reject": "The main problem with this paper is the lack of information about the Seed Instance Collection (Section 3.1). This paper should at least clearly show the sources of 189 queries, and I recommend providing the full list of the actual queries in the appendix or supplemental materials. If details of these seed instances are not provided, all experiments in this paper provide no information to readers. (The other issues below are much more minor. Please first focus on addressing this problem.)\n\nIn addition, it looks like this paper only evaluates tasks in limited domains (factual questions and logical tasks like word sorting). If this is true (I'm not sure because details are not provided), this paper should clearly mention that many other popular tasks, such as arithmetic reasoning or text generation, are not covered.\n\nThe experiments in this paper are insufficient to claim that the observations apply to diverse LLMs. (1) Although this paper analyzes other LLMs on the proposed benchmark, it mostly only focuses on GPT-4. It is better to show whether we see different patterns when the proposed framework (in Figure 1) is applied to other LLMs. (2) Experiments in Section 5 only evaluate GPT and small Llama, which are insufficient. I recommend evaluating at least one more strong closed model (e.g., Gemini, Claude) and a large open model (e.g., Mixtral, Llama 70B).\n\n(Updated June 5th) I've updated the overall rating from 5 to 6 since the rebuttal by the authors partially addresses my concerns.", "questions_to_authors": "(comments and suggestions)\n* Word sorting is not a novel task. For example, this task is in Big-Bench (https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/word_sorting).\n* Step 3 in Figure 1 does not explain almost anything about pattern optimization. I recommend to add more details.\n* I recommend providing the versions of the GPT models you used (e.g., 0613) to improve reproducibility.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers found the paper clear and supported its motivation (better identification and measurement of LLM shortcomings). Reviewers liked the idea of including humans in the loop in the process of finding model challenges. The experiments in the paper seem to be clearly explained and executed, but some reviewers point out limitations of the Self-Chal benchmark (especially the types of tasks/problems it covers and its reliance on GPT-4).

During the discussion period, the authors provided results from some follow-up experiments to address some of these issues (e.g., benchmarking more models, getting human performance numbers), which should be added to the next version of the paper. However, some of the limitations are unavoidable, given the design of the framework (e.g., the type of tasks it can include).
Decision: Accept