Capturing Failures of Large Language Models via Human Cognitive Biases
Authors: Erik Jones
Jacob Steinhardt
Venue: NeurIPS 2022 Accept (2022)
Keywords: open-ended generation
ai safety
codex
robustness
cognitive biases

Abstract: Large language models generate complex, open-ended outputs: instead of outputting a class label they write summaries, generate dialogue, or produce working code. In order to asses the reliability of these open-ended generation systems, we aim to identify qualitative categories of erroneous behavior, beyond identifying individual errors. To hypothesize and test for such qualitative errors, we draw inspiration from human cognitive biases---systematic patterns of deviation from rational judgement. Specifically, we use cognitive biases as motivation to (i) generate hypotheses for problems that models may have, and (ii) develop experiments that elicit these problems. Using code generation as a case study, we find that OpenAI’s Codex errs predictably based on how the input prompt is framed, adjusts outputs towards anchors, and is biased towards outputs that mimic frequent training examples. We then use our framework to elicit high-impact errors such as incorrectly deleting files. Our results indicate that experimental methodology from cognitive science can help characterize how machine learning systems behave.

Capturing Failures of Large Language Models via Human Cognitive Biases
Erik Jones
Jacob Steinhardt
UC Berkeley erjones@bekeeley.coo
UC Berkeley jsteiinhardttbeezkkleey..k edu
Abstract
Large language models generate complex, open-ended outputs: instead of out- putting class label they write summaries, generate dialogue, or produce working code. In order to asses the reliability of these open-ended generation systems, we aim to identify qualitative categories of erroneous behavior, beyond identifying individual errors. To hypothesize and test for such qualitative errors, we draw inspiration from human cognitive sysse--ystematicc patterns of deviation from tional judgement. Specifically, we use cognitive biases as motivation to (i) generate hypotheses for problems that models may have, and (ii) develop experiments that elicit these problems. Using code generation as a case study, we find that OpenAI's Codex errs predictably based on how the input prompt framed, adjusts outputs towards anchors, and is biased towards outputs that mimic frequent training exam- ples. We then use our framework elicit high-impact errors such incorrectly deleting files. Our results indicate that experimental methodology from cognitive science can help characterize how machine learning systems behave
Introduction
Recent large language models have achieved new, exciting capabilities. In contrast traditional classifiers, these models can generate open-ended text, enabling use cases like summarization
Stiennon et 2020 dialog Thoppilan et 2022 and code generation (Chen et al. 2021
The open-ended power of these systems, however, poses new reliability challenges. We must understand not only when systems err, but also the kinds of errors they make, as some errors are much more costly than others. For example, erroneous code that does not compile is less dangerous than code that deletes all files in the home directory. Studying how frequently an error occurs is difficult, as the same error (e.g. delete all files) can appear in a wide range of syntactically diverse outputs. In order to better reason about how complex systems err, we need methods test whether systems make the same qualitative eтror across different prompts, even when the generated outputs differ.
To study these reliability challenges, we primarily focus code generation models. Such models complete programs from comments, descriptions of code functionality, or initial lines of code. Code generation is particularly amenable to study since it is objective: generated solutions unambiguously correct or incorrect. Yet it is also open-ended: the set of programs model could output arbitrarily large, the rate at which specific program is outputted is not very descriptive.
Many of the reliability challenges posed by code generation models, and open-ended systems broadly, also arise when studying qualitative failures in human decision making. These failures, called cognitive biases, are systematic ways in which humans deviate from rational judgment [Tversky) and Kahneman 1974 For example, Tversky and Kahneman find that humans inadequately adjust
Cooe for this paper is available at
biases
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Failure- Eliicitin Prompts
Model Outputs
Cognitive bias:
Poosible failure: Codex relles semantically imelevant/ parts of the prompt
humans change their answers based on irreevann content
def sum Sun and
concatenate:
..*
return (oin(s)
peint Bello world")
Concatenate appended to print ( ("Hello Morld")
del coneatenata(a)=
Concasenate all strinngs
Figure 1: Illustration of our experimental framework. We use a cognitive bias (framing effect) to inspire a potential code generation failure mode (relying on irrelevant information). We then transform inputs in way that suspect will elicit the failure mode (prepending sum). We evaluate whether the modifications lower accuracy, and if the output is an instance of the targeted failure mode.
estimates away from initial values, and disproportionately recall distinctive examples. To uncover cognitive biases, Tversky and Kahneman ask questions that are crafted to systematically reveal some qualitative irrationality. They uncover insights into human behavior from the diverse responses, without complete mechanistic insight into the minds that they aim to analyze.
In this work, we extend Tversky and Kahneman]s experimental methodology and results elicit failure modes of large code and language models, without relying on complete mechanistic insight into their behavior (Figure Mp. Given a potential failure mode (e.g. relying on irrelevant information in the input), we construct transformation over inputs that largely preserves semantics, but that we suspect will elicit the failure (e.g. prepending an irrelevant functionn. We first test the model is sensitive to the transformation, by measuring if it decreases accuracy. Then, we check that the model outputs have elements that are indicative of the targeted failure (e.g. copies the irrelevant function).
We draw on four different cognitive biases to hypothesize potential failures of OpenAl's Codex (Chen et al. 202T] and Salesforce's is CodeGen [Nijkamp al.. 2022). then apply framework to each. Our results indicate that these models often rely on irrelevant information when generating solutions, adjust solutions towards
solutions, are biased based on training-set frequencies, and reverts to computationally simpler problems when faced with a complex calculation. We also apply our framework to OpenAl's GPT-3 [Brown al. 2020], and show that it updates its predictions towards anchors, and predictably adjusts its responses based on the question framing.
Finally, we show that framework can uncover iigh-impact errors: errors that harmful and difficult to undo. Specifically, we use our framework to systematically generate prompts where Codex erroneously deletes files. Our results indicate that experimental methodology from cognitive science can help uncover failure modes of complex machine learning systems.
2 Related Work
Large language models. Recent work has developed large, capablc, autoregressive language models,
which predict future tokens from past tokens [Radford al. 2019 Wang and Komatsuzakij 2021
Brown et al. 2020 Chen et al.) 2021 Rae et al. [021]] These models can be used for open open-ended
generation tasks such as summariztiion [Stiennon et al. 2020 Ziegler al. 2019 Rothe et al., 2020). dialogue [Ram et al.
2018
Thoppilan et al. 2022). and long form question answering [Fan et al.)
2019], among others. Model -gnnerated code has been used to solve both programming and statistics
questions [Chen et al. 2021 Tang et al. 2021]
There is some existing work studying failures of large language models. Benchmarks that measure model performance on multiple choice questions (Wang et al.) 2019b a Hendrycks et al. 2021b).
mathematics [Hendrycks et al. 2021c Cobbe al.) 202 long- form question answering [ et al.
2021
Gabriel et al. 2021 Shuster et al. 2021 Krishna et al. 2021 and coding problems [Hendrycks)
et al.
2021a| Chen et al.
2021 reveal imputs that the model errs on, but not the kind of error makes.
Another line of work shows that test- based language models can internalize bias and stereotypes
[Sheng et al.) 2019, Nadeem et al., 2020 Groenwold et al. 2020 Blodgett et al., 2021 Gehman
et al. 2020). and proposes applying fairness measurements from cognitive social sciences to machine
learning systems [Jacobs and Wallach| 202T]. Some work adversarially prompts models to leak
training data [Crllini al. 2020 output specific content [Wallace et al. 2019 Carlini et al.

---

Ide Largest Lorgest_divisorre div int) => intr
def sumix, y)
s*e For given number find the largest
... Returns the sun of x and ... printt(Bellln world')
Irrelevant Preceding Function
Prompt
nunber smaller than that divides evenly largest divisor(15)
Canonical Solution
for in [aversed(range(n))) 1 mm
def concatanato
return
Returns the concatenation of all strings in s
def cbeck(candidate)::
Test Cases
assert candidate(3)
...
candidate[(49)==
print('Bello Beell world')
Figure 2: Left. Example of a HumanEval problem from Chen et al.) 2021 The problem contains prompt (blue), a canonical solution to the prompt (green), and a few test-cases (black). The prompt contains two components: function signature (first line), and a docstring (remaining lines). Right. Illustration of our framing experiment. The transformed prompt (everything above the black line) contains an irrelevant preceding function (IPF) prepended IPF contains randomly chosen prompt from HumanEval (purple) and a framing line (red). The output Codex generates (below the black line) matches the framing line. When we omit the random HumanEval prompt and the framing line (leaving only blue), Codex produces the correct output.
prompt from HumanEval (blue). The
2020 And a final line of work identifies additional potential failures of current and future machine learning systems [Bender et al. 2021 Bommasani et al. 2021 Weidinger et al. 202T
Cognitive biases. Tversky and Kahneman 1974 define human cognitive biases: systematic patterns of deviation from rational judgment. They observe that humans employ heuristics when computing probabilities assessing values, and that these heuristics lead to predictable errors. Follow-up work
has added to, refined, and validated the set of known cognitive biases [T versky and Kahneman [973]
198
Strack al. 1988 Kahneman and Frederick 2002 Windhager al. 2010 Meyer [014]
Some known failure modes of large language models resemble cognitive biases. Zhao et al. [2021] and Liu al. [22T show that the specific random samples used for few-shot learning can change
GPT-3's prediction on binary and multiple choice tasks. Similarly, Wallace et al. (019) show that
innocuous prompts can routinely generate toxic model output. Our framework builds on this work by (i) identifying the link to cognitive biases, (ii) focusing on open-ended generation, and (iii) leveraging
v'ersky and Kahnemanj experimental methodology elicit qualitative failure modes.
3
Code Generation Experiments
3.1 Models
We study two code models: OpenAl's Codex [Chen al. 2021], J, and Salesforce's CodeGen. (Nijkamp et al. 2022]. Both models are autoregressive given a sequence of previous tokens, they predict the next token. Practitioners query these code models with partial programs, docstrings, function signatures, and obtain completions as output.
Codex. We study OpenAl's Codex, large language model trained to generate code from docstrings [Chen et al. 202T]. We use the OpenAl API to query the "davinci-001" version of Codex, and use greedy decoding generate solutions. Details of this model architecture not public, but it is likely
similar to the largest model from Chen et al. 2021]: a 12B parameter version of GPT-3 [Bronn et al. 2020) that fine-tuned on GitHub instead of the CommonCrawl.
CodeGen. We additionally study the 6.2 billion parameter "mono" version of CodeGen, which is trained on text data and fine-tuned GitHub. Unlike Codex, the weights of CodeGen are publicly available so we rum inference locally. We use greedy decoding to generate solutions.
3.2 Benchmarks
In order to identify whether code models some failure mode, we need to generate prompts that elicit that failure. To do so, we systematically apply transformations to standard prompts. We use two benchmarks as sources of prompts to transform: HulanEval, and MathEquations.
3

---

Functional accuracy
Outputs framing line
Framing Line
Model
ORIGINAL
FRAMED
ORIGINAL
FRAMED
CODEX
32.9
2.4
1.4
91.7
raise Not.Implemented
CODEGEN
25.6
1.5
0.0
79.3
CODEX
32.9
3.0
9.7
92.7
pass
CODEGEN
25.6
2.1
0.0
78.7
CODEX
32.9
3.3
0.0
92.7
assert False
CODEGEN
25.6
4.2
0.1
72.6
CODEX
32.9
4.9
11.5
65.6
return False
CODEGEN
25.6
3.6
0.0
64.6
CODEX
32.9
10.6
0.0
62.2
print ("Hello world!")
CODEGEN 25.6
11.0
0.0
58.2
Table 1: Results of the framing experiments. We compare functional accuracy and the rate which framing line is outputted over HumanEval with (framed) and without (original) irrelevant preceding functions. We find that the irrelevant preceding functions lower functional accuracy across all framing lines for Codex and CodeGen. Moreover, we find that the outputted function often appears verbatim in the generated output, suggesting that both models rely irrelevant information in the prompt.
HumanEval. We use the HumanFval benchmark as a diverse source of "normal" prompts CChen et al. 202T]. HumanEval contains 164 programming problems, each of which includes a function signature and doestring. The docstring contains an English description of the desired functionality and few example input-oulput pairs. HumanEval also contains canonical solution for each program, which we use in Section 3.3.2 We give an example problem from HumanEval Figure
MathEquations. We also curate set of prompts of basic arithmetic functions. For example, we prompt Codex "Write function that sums the squares of its inputs", or "Write a function that sums its inputs called product _ullpppluu five". Further details are given in Sections 3.3.3 and 3.3.4]
3.3 Empirical results
In this section, we show how cognitive biases can (i) inspire hypotheses for potential failure modes, and (ii) help us design experiments to test these hypotheses. Our approach has three steps. First, we construct a transformation over prompts that largely preserves semantics, but that we suspect failure mode. Next, measure if code models
will elicit specific pognitive-bias-inspiee sensitive to the transformation, by measuring the decrease in accuracy. And finally, we check that the generated output has elements that indicative of the targeted failure mode. Our approach mirrors the high-level methodology from Tversky and Kahneman [974]; ; we empirically elicit specific failure modes using targeted prompts, without complete mechanistic insight into the system that we study.
We draw inspiration from four cognitive biases: the framing effect (Tvrrsky and Kahneman [188]]
Section B.3.1), anchoring (T (evrrsky and Kahneman [1744] Section 3.3.2). the availability heuristic (Tversky and Kahneman 1973|; Section 3.3.3 and attribute substitution KKhheeman and Frederick
2002
Section 3.3.4)
3.3.1
Inspiration: Framing effect
We first draw inspiration from the framing effect: predictable shifts in human responses when the same problem is framed in different ways Tversky and Kahneman [981]. In their study identifying the effect, Tversky and Kahneman [981 find that subjects favor certainly saving 200 people over saving 600 with probability 1/3, yet prefer losing 600 with probability 2/3 over certainly losing 400 (even though these are equivalent). At its core, the framing effect shows how humans can rely semantically irrelevant information when they make decisions.
Using the framing effect inspiration, we hypothesize that code generation models may generate solutions exclusively from irrelevant information in the prompt. To elicit this failure, we transform HumanEval prompts by prepending irrelevant preceding functions. Specifically, to generate irrelevant

---

def common (11, 12): ret se
Anchor Function
Full Prompt for var [11, 12]:
HumanEval Prompt
for el in 11: for var in [11, 12]: print(var)
if ell in var: ret return sorted Figure 3: Illustration of our anchoring experiment using a real example (expanded in Figure We construct the anchor function (left) by taking the function signature from the HumanEval prompt (blue), appending lines of the canonical solution (green), then adding anchoring lines (red). We construct the full prompt (center) by combining the anchor function, the original HumanEval prompt, and the first n lines of the canonical solution. The solution Codex generates (right) combines elements of canonical solution (checks condition and adds ret.), with the anchor function (for var loop).
(def common...)
First solution lines
preceding functions, we combine a random prompt from HumanEval with a framing line. We test five framing lines: raise MotImplemeetteError pass assert False, return False. and print ("Hello world!") We first check that prepending these irrelevant preceding functions decreases functional accuracy Next, to test if models relied on irrelevant information in the prompt, we measure how much more frequently the framing line appears verbatim in the generated output.
We report the results of our framing experiments in Table M We find that adding irrelevant preceding functions consistently lowers functional accuracy, by between 22.3 and 30.5 points for Codex, across the different framing lines tested. Moreover, both models frequeetly generate the framing line: 81 % of the time for Codex and 70.7% of time for CodeGen, compared to only 4.5% and 0.0% over untransformed prompts respectively. These results suggest that code generation models can erroneously rely on irrelevant information in the prompt in predictable ways, even in the extreme case when doing so contradicts the type specification in the funetion signature (return False) )
3.3.2 Inspiration: Anchoring
We next draw inspiration from anchoring: humans' tendency to insufficiently adjust their estimates away from initial values. For example, Tversky and Kahneman [974] find that subjects' median estimate for the fraction of African countries the UN shifts from 25% to 45% based on whether they were first asked if the fraction was greater or less than 10% and 65%, respectively. Anchoring captures how humans adjust to partial information, versus irrelevant information (framing effect).
Using anchoring as inspiration, we hypothesize that code generation models may adjust their output towards related solutions, when these solutions are included in the prompt. To elicit this failure, we prepend anchor functions to prompts: functions that are similar to a valid solution for a HvmanEval prompt, but contain some error. We first check that prepending these anchor functions decreases functional accuracy, as in Section 3.3.I Next, to test if models adjust their output towards related solutions, we check that the generated solution contains elements of the anchor function.
We aim to construct anchor functions that are similar to functions in HumanEval prompts and that compile, but incorrect. To do so, take a prefix of the canonical solution, then add additional anchor lines that produce an incorrect output. See Figure 5 for an example. We describe two types of anchor lines, and how we test their influence the generated solutions, in the following paragraphs.
Print-var anchor lines. We first study print-var anchor lines, which iterate over all variables in the function signature and print their values. For a function with inputs var1 and var2, the associated print-var anchor lines are:
for var in [var1, var2]: :
print (var)
To study the influence of the print-var anchor lines on the solution, we measure how often (i) just the first line (for loop), and (ii) just the second line (print statement) appear the generated solution.
Add-var anchor lines. We also study add-var anchor lines, which return the sum of all variables in the function signature (converted strings). For a function with inputs var1 and var2. the add-var anchor lines are:
Chen et al. 2021 we measure performance on HumanEval functionall accuracy: the fraction of programs that pass all of the test cases provided at the url: htops:///ithhu www./pppea/huuannnee..oeoeo
: Following Chen
5

---

Change functional accuracy with anchor functions
Anchor function uence on generated solutions
0.8
loop
Functional acc.
0.6
Frac solutions
Prints var
Codex
Exact copy
0.4
0.2
anchor function
Anchor function
0.8
No anchor function
CodeGen
Functiona acc.
0.6
Anchor function
Frac. solutions
0.4
0.2
Prints var
copy
0.0
3
6
8
0
1
2
8
Canonical solution prompted
Canonica solution ines prompted
Figure 4: Results of the print-var anchoring experiment. Left. We measure the functional accuracy of Codex (top) and CodeGen (bottom) with no anchor function prepended (baseline acc) and with print- var anchor function prepended (anchor acc), and find that prepending the anchor function consistently lowers accuracy. Right. We measure the influence of the anchor function on the generated solution by plotting the fraction of generated solutions that contain "for var in from the print-var anchor prompt (for var loop), the fraction of generated solutions that include "print (var) " (prints var), and the fraction of generated solutions that output the anchor function verbatim without additional content (exact copy), function of the number of canonical solution lines added to the prompt.
tmp str (var1) str (var2) return tmp
To study the influence of the add-var anchor lines on the solution, measure how often return tmp appears in the generated solution.
Print-var results. In Figure we show that prepending print-var anchor functions consistently lowers Codex and CodeGens' funetional accuracies across different number of prompted canonical solution lines. We vary the number of canonical solution lines to study prompts of different difficulties; the number solution lines increases, the number remaining lines models must produce decreases
We additionally find that elements of anchor function often appear in both models' outputs, suggesting that code generation models adjust their solutions towards related solutions. Figure d Codex generates for var in 32%-61 61 % of solutions when at least one line of the canonical solution is included, and generates print (var) in 26% 44% of solutions. CodeGen's 's behavior is qualitatively similar. Both models sometimes even incorporate the anchor lines into correct solutions; on Codex, the for var loop is used correct solution for 3%-11% % all outputs, while print (var) is used in correct solution for 1%-9% of outputs.
we see that
Control experiments. One concern might be that models just outputs the anchor function verbatim, as in Section I.3.I but we find that this does not explain the full results-both models include anchor lines many solutions that do not copy the anchor function verbatim. We also find that changing the name of the anchor function leads to only negligible changes; see Appendix A. for details.
Add-var results. We next consider results for add-var anchor lines. Full results for the add-var anchor prompts are presented Appendix and are qualitatively similar to the print-var results.
One again, we find that prepending the anchor function consistently lowers functional accuracy. Moreover, the outputted solutions often include an anchor line. For example, Codex and CodeGen generate return tmp in 26%-46% and 13%-79% of solutions respectively, depending how many canonical solution lines we prompt with. These results are not caused by models outputting the anchoring function verbatim: this only occurs between 7% and 12% of the time for Codex, and 4% and 12% for CodeGen. Overall, our findings suggest that code generation models can err by adjusting its output towards related solutions, when the solutions are included in the prompt.
We filter out programs whose entire canonical solution would appear in the prompt; see Appendix A.1

---

Write a function that squares the sum of its inputs
Write function that sums its inputs called product plus
def square sum (x, y): return x ** + ** 2
def product product_ppus__te 2(x, y) return x +
Figure 5: Left. Availability heuristic example where Codex mixes up the order of operations. The correct function signature (blue), square_sus matches the prompt. However, the incorrect function call (red) instead squares its inputs before summing them. The prompt is above the horizontal line, while the generated code is below. Right. Attribute substitution example where Codex relies on the function name to generate output. Codex correctly generates the desired function name (blue), but errs by using the function name instead of the prompt to generate the return statement (red).
3.3.3 Inspiration: Availability heuristic
We next draw inspiration from the availability heuristic: the tendency of humans evaluate how frequently an example occurs based on how easy it is to recall. For example, [Tversky and Kahneman 1973] find that humans tend incorrectly report that there are more first words that start with "T" and "k" than have third letter "r" and "k", because the former quickly come to mind.
Using the availability heuristic motivation, hypothesize that code generation models may em by outputting solutions to related prompts that appear more frequently in the training set. To elicit this failure, we start with prompts that apply a unary operation before binary operation (unary-first), then flip the order (binary-first), Programmers tend to apply unary operations first (e.g. when computing Euclidean distances or variances), so we conjecture that they appear more frequently on GitHub. We first check that flipping the order of operations decreases accuracy. Next, to test if code generation models instead outputs related prompts that occur more frequently in the training set, we measure whether code generation models instead output the unary-first solution.
We consider all 12 combinations of the binary operations sum, difference, and product, with unary operations square, cube, quadruple, and square root. Focusing on Codex we find that accuracy drops from 50% to 17% when flipping the order from unary-first binary- ffrst. Among combinations where flipping the order leads to error, we find that 75% of the binary-firs outputs are the unary-first solution. We exhibit one such error in Figure when prompted square the sum of its inputs, Codex generates the correct function name rsquare_aur ). but reverses the order of operations. Our results suggest that Codex can err by outputting solutions related, frequent prompts in the training
Control experiments. One worry is that the dip in performance due the instructional nature of our prompts. We rule this out by evaluating Codex on prompts where the docstring appears beneath the function signature and a definition rather than command, to more closely mimic some functions on GitHub. We obtain qualitatively similar results on these prompts, see Appendix A.4 for details.
3.3.4 Inspiration: Attribute substitution
Finally, we draw inspiration from attribute substitution: the human tendency to respond to a com- plicated question using simpler, related question [Kahneman and Frederick, 2002). For example, a professor when asked how likely a candidate to be tenured, may instead respond with how impressive they found their job talk.
Using attribute substitution inspiration, we hypothesize that Codex may use simple-bute but-incorrect heuristics to generate solutions. To elicit this failure, we add requests for conflicting function names to MathEquation prompts. For example, in Figure 5 we prompt Codex to write a program that sums its inputs called product_ppuu 2 We first check that adding conflicting function names decreases Codex's 's functional accuracy. Next, to test if Codex uses comple-but-incoree heuristics to generate solutions, we check whether the generate solution matches the function name.
We evaluate Codex using 90 MathEqqattion prompts where the desired solution and requested function name differ. To construct prompts, we begin with prompt that Codex originally solves
'We find that CodeGen often produces nonsensical solutions on the style of prompts used in Section 3.3.3 Section 3.3.4 and Section 3 we focus primarily Codex.
7

---

Name location
Correct
Matches function name
Other error
No name
100.0
0.0
Doestring
4.4
80.0
15.6
Function signature
4.4
70.0
25.6
Name first
51.7
43.7
Table 2: Results of the attribute substitution experiments. We report accuracy when we do not request a contradictory functtin name (no name), we request a function name in the docstring (docstring), in the function signature below the docstring (function signature), above the docstring (name first). Overall, we find that Codex frequently generates solutions based on the function name.
(sum, difference, or product), then append a request for a specific, contradictory function name (see Appendix A.4 for full implementation details).
We report our experimental results in Table 2 When we request a conflicting function name, Codex's accuracy drops from 100% to only 4.4%-4.6%. This finding holds whether we request the function name in the docstring, write it in the function signature below the docstring, or write the function the function. Moreover, for between 52% and 80% of prompts,
name over simple description Codex responds with the function specified in the function name. Our results indicate that Codex can err by using simple-but-in but-io---tt heuristics to generate solutions.
4 GPT-3 Results
In this section, we extend our study from Codex to GPT-3. To test GPT-3 for failure modes, we try to faithfully reproduce and extend the anchoring experiment of Jacowitz and Kahneman [1955 and framing effect experiment of Tversky and Kahneman [981].
Anchoring. As in Section 3.3.2 we study Section 3.3.2. we study anchoring: humans' tendency
to insufficiently adjust their estimates away from an initial value (Tvrrsky and Kahneman 1974]. We largely replicate the anchoring study presented in Jacowitz and Kahneman [1995] but test the "davinei-001" version of OpenAl's GPT-3 instead of humans.
In their original experiment, Cawieentt,,ccwzzzz Jacowitz and Kahneman asked students to estimate quantities such as the length of the Mississippi river in miles. They then asked new students estimate the same quantities, but first gave them a upper or lower bound on the true answer (e.g. the Mississippi river is longer than 700 miles), which they call anchors. They find that students tend underestimate the true quantity when prompted with the lower anchor, and overestimate it when prompted with the upper anchor.
We adapt the anchoring study from Jacowitz and Kahneman 1995 by finding the true answer for 14 of their 15 original question: then computing upper and lower anchors by increasing and decreasing the true answer by a fixed percentage p. See Appendix As an example, the actual answer is 2000 and p is 50%, the upper anchor is 3000 and the lower anchor is 1000. We this bound as anchor, that typical prompt might be:
B. I for full list of questions and true answers.
The length of the Mississippi river (in miles) is greater than 1000.
anchur
What is the length of the Mississippi River (in miles)? Answer:
To study anchoring in GPT-3, we measure how prepending the anchor changes GPT-3's estimate. We categorize four potential changes: the estimate does not change, the estimate shifts towards the anchor, the estimate shifts away from the anchor, and the estimate is gibberish. We report the results in Table 3 p (20%, 50%), ). We find that GPT-3 routinely updates its estimate when an anchor is prepended, and tends to shift the estimate towards the anchor. We also find that while GPT-3's updated estimate sometimes matches the anchor exactly (67% of the time). it also often lands between the anchor and the original prediction, mirroring the behavior of humans.
Our replication has few limitations. Like the original study our sample size small, we construct prompts with templates, and many of the outputs-on average 41 41%-are gibberish. Nevertheless, our results suggest that GPT-3 incorporates the anchor during estimation.
We omit question asking for the number of bars in Berkeley, CA, since the answer ambiguous
8

---

Anchor change to model output
p
No change
Towards anchor
Away from anchor
Gibberish
20%
10.7
28.6
10.7
50.0
50%
14.3
42.9
10.7
32.1
Table 3: Results of the adaptation of the anchoring study from Jacowitz and Kahneman GPT-3. We consider anchors that are 20% and 50% increases and decreases from the ground truth answer, and measure how often GPT-3's revised prediction does not change, shifts towards away from the anchor, or is gibberish, aggregated across lower and upper anchors.
1995 on
***
Deletion Error vs Number of Packages
Delete all files in an inputted directory that import all statsmodels, plotly, seaborn, seipy
***
Percentage
80
import as import
def selett_all(dieeee (directorr))
40
root, file ce.welk(darectoorr)
for in files:
Any package
First package
with ppann(sspath.jooniii if se.search("impors statsmodels", fp.read())) Σt}
f), 'r") as fp:
2
3
5
Number packages
Figure 6: Left. Example where Codex incorrectly deletes files. We prompt Codex to delete files containing all of statsmodels, plotly, seaborn, and scipy. Codex correctly iterates through all files in the inputted directory (blue). but then incorrectly deletes all files containing statsmodels (red), as attribute substitution suggests. Right. Plot describing the errors Codex makes function of the number of packages. We find that Codex often incorrectly deletes files if they contain any of the listed packages, and relies more on just the first package as the number of packages increases.
we study the framing effect: predictable shifts in human responses when the same problem framed in different ways. We largely replicate the framing exper- iment presented in T'versky and Kahneman [188]] we compare GPT-3's responses to two equivalent decisions: choosing either deterministically save (or let die) some fraction of population, to probabilistically save (let die) the whole population.
Framing effect. As in Section 3.3.1
We measure the rate at which GPT-3 chooses the probabilistic option across different population sizes and different fractions probabilities. See Section B.2 for full results. When using the probability in the original study, GPT-3 qualitatively mirrors humans: it chooses the probabilistic option far more frequently under the "not save" framing than under the "save framing". However, for higher probabilities, GPT-3 consistently chooses the probabilistic option for both framings; conjecture that humans could exhibit similar behavior in this regime, since the probabilistic option more certain. Overall, results suggest that GPT-3 selects different options based the framing, and could be a test-bed to identify qualitative human behaviors without running full human studies.
5 High-Impact Errors
We have shown how framework helps elicit failures of large language models. In this section, we use our framework to construct cases where Codex makes high-impact errors: harmful errors that hard to undo. Specifically, we construct prompts where Codex incorrectly deletes files.
As in Section 3.3.4] we draw inspiration from attribute substitution: the tendency of humans to respond to complex question with simpler, related question. Using attribute substitution motivation, hypothesize that Codex may simplify complex expressions such as conjunctions. Instead of checking all components of a conjunction at once, it might "give up" and consider subsets of the components individually (e.g. checking for A or V B instead of AA B). To elicit this failure, we prompt Codex to delete files containing specific sets of package imports; how often Codex generates a simpler output that erroneously deletes files, as well as how often produces the correct output. See Appendix for additional details.
Figure for example. We measure
We test for two types of simpler outputs: code deleting all files containing first package in the set (i.c. A instead of A A B), and code deleting all files containing any package in the set (i.e. A V B

---

instead of A A B). The latter operation is computationally simpler than checking if file contains all packages, since Codex can delete a file whenever single package in the set appears.
In Figure 6 we illustrate the breakdown of the errors Codex makes as a function of the number of package imports in the prompt. We find that Codex erroneously deletes files at least 80% of prompts when the number of package imports is at least three, despite producing correct output on 90% of prompts when the number of packages is at most two. Moreover, we find that Codex increasingly errs by using only the first package as the problem gets more challenging (i.e. the number of packages increases), as attribute substitution predicts.
Control experiments. To very that our findings generalize to different classes of realistic prompts, we test Codex prompts containing a descriptive doestring beneath the function signature libbliibrriiessse (directory). We observe qualitatively similar results, though we find more instances of low-impact errors; see Appendix C for details. Overall, results demonstrate how our framework can preemptively elicit high-impact errors, like erroneous deletions.
delete_ _atee_al__va__ aal_wwit___aee wil_awitt__llbrrrres
6 Discussion
In this work, we identify and test for classes of errors that open-ended generation systems can make, using cognitive biases as motivation. To do so, we generate hypotheses for potential qualitative failure modes, then construct transformations over prompts that elicit these failures. Our experiments uncover deficiencies of Codex, CodeGen, and GPT-3, and elicit high-impact errors that are challenging to undo. While we focus on a few specific failure modes, future work could apply our framework to uncover additional failures. Moreover, our framework queries systems black-box, so it could be used to quickly probe for errors in future systems as they are released.
Some of our results highlight how optimizing likelihood could be at odds with human intent. For example. over GitHub, programs may more often match their function signature than docstring (Section 3.3.3) or tend to complete to pass if the preceding function does (Section 3.3.1 Neverthe- less, our results elicit qualitative errors regardless of the "correct" behavior (i.e. even when what incorrect and correct flips), and demonstrate the importance of documenting qualitative failures.
The reliability challenges posed by the open-ended generation systems that study sometimes also apply to classifiers. Some classification errors can be more costly than others [Oakden-Rayne jet al. 2020, classifiers may use irrelevant information to make predictions (Sagawa al. 22200. and input-level transformations like universal adversarial triggers [Wallace et al. 2019 and distribution shifts [[Heddrycks and Dietterich 2019] induce errors. However, while classification errors may be succinctly summarized with a confusion matrix, generation errors cannot, since each output appears infrequently. To tame the large output space, our transformations must induce categories of errors that we can reliably measure. Despite this additional constraint, we are able to construct model-agnoster transformations: we do not use the training data, model parameters, or even output logits. Our success in this restricted setting demonstrates the comparative brittleness of completion systems.
We present method to systematically elicit errors from large language models. While believe our work is important to understand model behavior, bad actors could exploit the errors we reveal (e.g. by deleting files on systems with a Codex back-end). Nevertheless, we introduce new robustness challenges for developers and identify misuses of these models, which we feel supersedes this risk.
As a subroutine in experimental pipeline, we use cognitive biases as inspiration to identify potential failure modes. This is an example of using a reference system-a system that is analogous to the ML models study in some meaningful way-to generate insights into ML systems (Steinhardt 2022). We use humans as the reference, focusing specifically on their susceptibility to cognitive biases. Other references, such as complex systems or evolution, may uncover new errors and insights. Moreover, ML systems could additionally err in ways that known systems do not, so it will also be useful have intrinsic methods for characterizing model errors. Overall, our work underscores the need for more extensive testing of generative ML systems before their widespread deployment.
Acknowledgements
We thank the anonymous reviewers, Ruiqi Zhong, dean-Stanislas Denain, Aditi Raghunathan, Jessy Lin, and Lawrence Chan for feedback. This work was supported by NSF Award Grant no. 1804794.
10

---

References
Emily Bender, Timnit Gebru, Angelina McMillan- Majaar and Shmargaret Shmitchel. On the dan- gers of stochastic parrots: Can language models be too big? In ACM Conference Fairness, Accountability, and Transparency (FAccT), 2021.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. Stereotyping norwegian salmon: An inventory of pitfalls in faimess benchmark datasets. In Association for Computational Linguistics (ACL), 2021.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill. Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quiney Davis, Dorotrya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Ste- fano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa. Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie. Juan Car- los Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong. Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. arXiv preprint Xrx: 20t.::020...... 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprini arXiv:2005. 14165, 2020.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. arXiv preprint arXiv:2012.07805 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin. Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. arXiv preprint 210.:22077.03374 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10.1666. 2021.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli, ELI5: Long form question answering. In Association for Computational Linguistics (ACL), 2019.

---

Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. GO FIGURE: A meta evaluation of factuality in summarization. In Findings of the Association for Computational Linguistics (Findings of ACI.), 2021.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint PrXiv:2009.11460 2020.
Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. Investigating aurican-americaa vernacular english in transformer based text generation. In Empirical Methods in Natural Language Processing (EMNLP), 2020.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference Learning Representations (ICLR), 2019.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In Advances in Neural Information Processing Systems (NeurlPS), 202
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021b.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems (NeurIPS), 2021c
Abigail Z. Jacobs and Hanna Wallach. Measurement and fairness. In ACM Conference on Fairness, Accountability, and Trunsparency (FAceT), 2021.
Karen E. Jacowitz and Daniel Kahneman. Measures of anchoring in estimation tasks. Personality and Social Psychology Bulletin, 21(1T):161-1116 1995.
Daniel Kahneman and Shane Frederick. Representativenes. revisited: Attribute substitution in intuitive judgment. In Heuristics and Biases: The Psychology of Intuitive Judgement, pages 49-81. 2002.
Kalpesh Krishna, Aurko Roy, and Mohit lyyer. Hurdles to progress in long-form question answering. In North American Association for Computational Linguistics (NAACL), 2021.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprini arXiv:2 2021.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3. arXiv preprini arXiv:2101.068M4 2021.
David E. Meyer. Semantic priming well established. Science, 345(61966 2014.
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456 2020.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huam Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis. arXiv preprint arXiv:2203.13474, 2022.
Luke
Jared Dunnmon, Gustavo Cameirr, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM Conference on Health, Inference, and Learning, pages 151-159, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
12

---

Jack W. Rae. Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, Aslanides, Sarah Henderson, Roman Ring, Susannah Young. Eliza Rutherford. Tom Henni- gan, Jacob Menick, Albin Cassirer, Richard Powell, G. V. D. Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbi, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John F. J. Mellor, I. Higgins, Antonia Creswell, Nathan McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, D. Budden, Esme Sutherland, K. Simonyan, Michela Paganini, L. Sifre, Lena Martens, Xiang Lorraine Li, A. Kuncoro, Aida Nematzadeh, E. Gribovskaya, Domenic Donato, Angeliki Lazaridou, A. Mensch, Lespiau, Maria Tsimpoukelli, N. Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Tobias Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, I. Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury. Matthew Johnson, Blake A. Hechtman, Laura Weidinger, lason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem W. Ayoub, Jeff Stanway, L.. Bennett, D. Hassabis, K. Kavukcuoglu, and Geoffrey Irving. Sealing language models: Methods, analysis & insights from training gopher. arXiv, 2021.
Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, Eric King, Kate Bland, Amanda Wartick, Yi Pan, Han Song, Sk Jayadevan, Gene Hwang, and Art Pettigrue. Conversational ai: The science behind the alexa prize. arXiv preprint arXiv: 1000180000000 2018.
Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics (TACL), 8:264-280, 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations (ICLR), 2020.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as babysitter: On biases in language generation. In Empirical Methods in Natural Language Processing (EMNLP), 2019.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kicla, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567, 2021.
Jacob Steinhardt. Anchor weights for ML. Bounded Regret, 2022.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Fritz Strack, Leonard L. Martin, and Nobert Schwarz. Priming and communication: Social determi- nants of information use in judgments of life satisfaction. European Journal of Social Psychology, (8(5):4294443 1988.
Leonard Tang, Elizabeth Ke, Nikhil Singh, Nakul Verma, and Iddo Drori. Solving probability and statistics problems by program synthesis. arXiv preprint arXiv:2111.08277 2021.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang. Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Shung-Ching Chang, Igor Krivokon. Will Rusch, Mare Pickett, Kathleen Meier-Hell Hells--elllttem, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201. 088339, 2022.
Amos Tversky and Daniel Kahneman. Availability: heuristic for judging frequency and probability. Cognitive Psychology, 5(2):207-232, 1973.
13

---

Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):114-113311 1974.
Amos Tversky and Daniel Kahneman. The framing of decisions and the psychology of choice. Science, 211(4481):453 458, 1981.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing NLP. In Empirical Methods in Natural Language Processing (EMNLP),201)
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems (NeurIPS), 201 9a.
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman, GLUE: multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR), 2019b.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 billion parameter autoregressive language model, 2021.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng. Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving. and lason Gabriel. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.0433 2021.
Sonja Windhager, Florian Hutzler, Chuss-hristiin Carbon, Elisabeth Oberzaucher, Katrin Schaefer, Truls Thorstensen, Helmut Leder, and Karl Grammer. Laying eyes on headlights: Eye movements suggest facial features in cars. Collegium Antropologicum, 14(3):1075-110) 1080, 2010.
Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning (ICML), 2021.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 08593, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See discussion.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
discussion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or a URL)? [Yes] Included the supplement.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] We do not train new models, but Section 3.7 and Section we provide details about the models we evaluate.
14

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A] Our experiments consider the deterministic rollout of models on fixed prompis.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] We do not train models this work.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite GPT-3 and Codex, and give credit OpenAl.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le rre using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "wljPAKTZorD", "reviewer": "Reviewer_Nt74", "summary": "This paper explores the robustness of large language models through the lens of 4 human cognitive biases - framing, anchoring, attribute substitution, and availability. Results show that OpenAI's Codex program synthesis model is not robust to changes in prompts that are inspired by these 4 biases.", "questions": "**Major**\n* Figure 4 (right) - explain why “fraction of solutions” containing anchor function lines is so low when # of canonical solution lines prompted is 0\n* 3.2.4 - relationship to attribute substitution cognitive bias is a stretch. Not very convincing that confusing the model with conflicting function name vs. desired behavior is an example of model erring “by using simple-but-incorrect heuristics to generate solutions”. Recommend replacing this with examples from Section 5 and removing section 5, which doesn't seem to warrant a separate section\n* Would be nice to see more cognitive bias methods applied to GPT, since it shows diff/similarity between LM and human\n\n**Minor**\n* Footnote 2 - please share how many examples were filtered out\n* Figure 4 - clarify whether this is on humaneval only\n* Print-var results + add-var results sections - recommend rounding results to whole numbers. Tells same story, easier to digest\n* Table 3 - would be interesting to plot anchor values vs. prediction (e.g. expect to see as upper bound anchor increases, GPT-3 prediction increases) to clearly demonstrate anchoring effect", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* Well-written and easy to understand for the most part (especially introduction and some figures)\n* Categorizes a few types of errors for program synthesis models and develops interesting evaluations for them (i.e. functional accuracy + custom evaluations for each error type)\n* Experiments demonstrate that Codex, and likely other language model-based program synthesis models, are not robust to certain types of perturbations\n* Example of anchoring applied to GPT is especially interesting, since it shows a similarity between how language models (LMs) and humans behave \n\n**Weaknesses**\n* Paper seems to over-generalize contribution - e.g. claims to study \"failure modes of large language models\", but contributions are rather specific to prompt robustness. I would expect a paper with this general of a title and abstract to study more errors more comprehensively (e.g. anaphora resolution, entailment, long context, etc.). I recommend narrowing down the claims made in this paper\n* Contribution is relatively simple (changing prompts and measuring robustness). Could expand scope by increasing types of cognitive biases tested, or more experiments on GPT\n* Missing some key details for reproducibility and understanding\n  * Needs details on how functional accuracy is checked (e.g. environment, open source code)\n  * Figure 3 - unclear what is going on. Maybe add full example and/or more explanation in caption\n  * Figure 5 - clarify what colors mean, and what is input vs. generated\n  * Section 5 - what examples are used here, and how many? How were examples generated and evaluated?\n* Section 3.2.3 on Availability bias unsubstantiated\n  * L213-214 - “Our results suggest that Codex can err by outputting solutions to related, frequent prompts in the training set.”. Need to show evidence that unary operations applied before binary in training set to make this claim. Recommend sampling training set and counting unary-first vs. binary-first", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ovhXFBV6F3g", "reviewer": "Reviewer_mPku", "summary": "In this paper, the authors assess the quality and reliability of open ended systems such as GPT-3 and Codex from the perspective of cognitive biases. The authors primarily focus on the application of code generation and design examples that elicit commonly made errors by these systems for the task of code generation and completion. The authors primarily focus on the 4 different set of cognitive biases: (1) framing effect; (2) anchoring bias; (3) availability heuristic and (4) Attribute substitution.\n\nThe main contributions:\n- Creation of framework that contains hypothesis and prompts that extend existing cognitive bias methodology. The hypothesis and examples created can measure the model sensitivity to these transformations. The framework is also capable of discovering high impact errors and other types of errors made by the model.\n", "questions": "- it would be better if the authors could add more explanation about how the drop in performance related to cognitive biases. ", "limitations": "The authors address limitations of systems such as GPT-3 and Codex. However, there is no mention of limitations of designing prompts", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths & Weakness:\n- The paper studies an important problem of robustness of large scale LM's such as Codex and GPT-3 from the task of of code generation. Using cognitive bias as a way to measure robustness of these models is an interesting and novel approach.\n-  Results in the paper show that by manipulating exhibits different learning effects and thus resulting in significant drops in performance. However, claiming these to be exhibits of cognitive biases seems to be a bit of a stretch. Most of the errors discovered could be novel from the perspective of code generation but prior literature in the area of robustness and fairness and prompt engineering shown that these large scale model are susceptible to unwanted context added to prompts\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WstP3nHwpo0", "reviewer": "Reviewer_3Gtn", "summary": "This paper proposes to use human cognitive biases as inspiration for types of errors that large language models may make, as a means of finding and catching such errors. They run experiments on four such cognitive bias-inspired errors on code generation with Codex models. They also include two additional sets of smaller experiments, one with GPT-3 with a pre-existing human cognitive bias experiment, and another with Codex on code deletion as a means of emphasizing high-impact errors.", "questions": "- Most of my specific questions about the experiments are covered in the above section.\n- What decoding configuration was used for Codex/GPT-3? If temperature was used (i.e. non-greedy decoding), it should be reported and potentially run with multiple runs. The checklist mentions \"deterministic rollout\", but explicit clarification of the decoding configuration would be helpful. ", "limitations": "The authors have framed the work in terms of being helpful to catch errors of large language models, which I think is reasonable. The work does have experiments testing the propensity for models to indiscriminately delete files, but I believe the experiments are conducted within a reasonable and contained setting.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As a foreground, my view on the use of human cognitive biases as a motivation in this paper is that it is a useful and valid approach for coming up with potential error categories, but ultimately the results in the paper can be interpreted and have value independent of the associated human cognitive biases. Because of this, and my relative unfamiliarity with human cognitive bias work, I will focus my comments on evaluating the experiments primarily from the point of a view of a machine learning researcher.\n\nStrengths:\n\n- The paper is exceedingly clear, well-written and easy to follow. The examples of the prompt formats is extremely welcome and papers that do prompt-based experiments would do well to follow this example.\n- The experiments are well-structured and well-motivated. While I do have concerns regarding some of the experimental setup (described below), the logical flow from human cognitive biases to a proposed equivalent model experiment to the metrics being measured are generally clear, reasonable, and cover the questions a critical reader might have (e.g. also measuring copying of the irrelevant propmt).\n\nWeaknesses:\n\n- Experimental setup / evaluation. My major concern with this paper is the experimental design, and whether the experiments correspond to the research question being asked. I will discuss my concerns with each of the experimental setups here. This section includes both what I consider weaknesses as well as questions/additional results I would like to see.\n\n1) Framing.\n\nI have an issue with the choice of \"irrelevant preceding function\" and the interpretation of model behavior, in that I do not share the authors' interpretation that the prepended functions are irrelevant. For instance, given a prompt:\n\n```\ndef add(x, y):\n    \"\"\"Adds numbers\"\"\"\n    raise NotImplementedError()\n\ndef multiply(x, y):\n    \"\"\"Multiplies numbers\"\"\"\n```\n\nI *would* expect the model to output `raise NotImplementedError()`. This looks like a file that is work-in-progress, and to my mind the model is correctly extrapolating from the prompt. In the authors' words, I do not believe this is \"semantically irrelevant information\". A better example (and this overlaps somewhat with anchoring) would be something like:\n\n```\ndef divide(x, y):\n    \"\"\"Divides numbers\"\"\"\n    assert y != 0\n    return x / y\n\ndef multiply(x, y):\n    \"\"\"Multiplies numbers\"\"\"\n```\n\nIn this case, if the model outputs `assert y!= 0`, then we can conclude that the model is actually using the prepended function in an erroneous/unhelpful way. Put another way, if the experimenter wants the model to generate functional code, they should to prompt *with* functional code. This should be a pretext for all experimental involving functional accuracy.\n\n2) Anchoring.\n\nI think the setup and results in this experiment are clearer than for framing. One concern I have is with the confounding factor of the anchor function having the same name as the function to be completed. An experiment with a similar name instead (e.g. common1 vs common2) would clarify this result.\n\n3) Availability.\n\nThe authors conjecture without evidence that \"Programmers tend to apply unary operations first\", and use this as the basis for their conclusion that Codex has learned to output \"related prompts that occur more frequently in the training set\". From the results presented, we can take away that the model appears more biased toward unary-first solutions, but it seems difficult to draw conclusions relating to the training set without further evidence.\n\nI also have a milder concern regarding the formatting of \"docstrings\" in this and the attribute substitution experiments. Docstrings should go under the function signature and describe the function. The \"docstrings\" in these two experiments are provided above the function signature, and written in the form of an instruction \"Write a function that...\", which is not how docstrings are used in practice, and therefore should not be part of the expected behavior of the model.\n\n4) Attribute substitution.\n\nMy primary concern with this experimental setup is that it introduces contradictory function names and docstrings (see also the discussion above), and asserts that the model ought to take the dosctrings as the ground truth and that the function names are misleading. It is not clear to me that this has to be that case. I could conversely argue that in practice, documentation updates often lag behind code changes, and that the function name is a more reliable source-of-truth than docstrings. Under this interpretation, what the results in Table 2 show is that the model mostly \"correctly\" adheres to the function name and is only mildly affected by the \"incorrect\" docstring information. Even in this lens, I think the results are still valuable, because the conflicting information appears to introduce a set of \"Other errors\". I recommend the authors revisit and rethink the design and interpretation of this experiment.\n\n(Admittedly, this is confounded in the \"docstring\" sub-experiment, since the only source of information is the dosctring.)\n\n6) Deletion\n\nThis suffers from the same issue as the attribution substitution experiments, though not to the same degree (`delete_all` can be reasonable seen to be context dependent). That said, I wonder if the results would be different if the function were named more helpfully (`delete_all_with_libraries`).\n\nIn summary, while I think this approach for finding model errors is valid and worthwhile, the current experiments fall short of demonstrating the biases in a convincing manner. Addressing these issues would make a meaningful difference to my review.\n\n- Scale of the experiments. The experiments tend to be on a small scale with a small-ish number of examples. Particularly given the potential of models to be sensitive to the exact phrasing of prompts, this is a moderate but not large concern.\n- Lack of evaluation on open models. As the authors acknowledge, the Codex model is private and many details regarding the architecture and training are not public. This makes it difficult to ascertain, for instance, that the models have no had additionally curated data to avoid some of these errors (i.e. a more naively model may perform even worse than reported in the paper). While I acknowledge that this paper is primarily focused on Codex and duplicating experiments on an additional model with a separate setup is significant additional work for the authors, I think it is fair to say that experiments solely on a set of non-public models with many unknowns does hurt the scientific contribution of the paper and calls into question whether these results generalize. Even a small set of experiments on a similar more public model (e.g. CodeGen) that demonstrate equivalency of results would help assuage the reader.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Mjbna7E2xmL", "reviewer": "Reviewer_RQFZ", "summary": "Authors highlight the fragile nature of the GPT-3 style code generation models. Inspired from cognitive biases research, authors design different manual input transformations (prompt tuning) where they add some irrelevant information to the model. Using experiments with Codex model, they show that model's code generation performance drop significantly when they fed modified input to the model. It's worth highlighting that the proposed transformations are generally non-adversarial in nature.    ", "questions": "- One can argue that framing effect is by design and is not an unexpected behavior. Based on the GPT-3 paper's in-context learning experiments, we expect model to imitate the behavior of the functions provided in the context. Hence, when a preceding function throws `NotImplementedError`, it's probably reasonable for model to generate `NotImplementedError` as *one of the answers* (assuming we sample multiple answers with a temperature). \n- Which decoding algorithm are you using for generation? Is it greedy decoding or are you using some sampling algorithm? Further, for code generation tasks, a common practice is to use sampling and report pass@K metric as it reduces variance in the results. Have you observed the same accuracy drop with pass@K ?\n- Do you have any counterexamples i.e. transformations where you did not see any accuracy drop?  \n- \"Open-ended systems\" is a very broad term. Since this paper mainly focuses on the code generation task, I would recommend to use \"open-ended generation\" instead of using “open-ended systems”. \n\n**Typos**:\n- L41: that that \n- L111: that that \n- L140: prictable -> predictable \n- L298: able construct -> able to construct \n\n\nReferences: \n1. Li, Yujia et al. “Competition-Level Code Generation with AlphaCode.” ArXiv abs/2203.07814 (2022)", "limitations": "I don't see any major concerns related to negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Proposed transformations are very intuitive and are well-motivated by prior research in cognitive science. \n\nWeaknesses:\n\n- While the transformations described in the paper are intuitive, some of them are hard to scale. In particular, one has to design such transformations manually for a given programming language setup. Further, authors have not provided an easy to use list of transformations (or dataset version) which other researchers can reuse for future experiments. \n- Prior work [1] show that problem description, context are crucial to execution accuracy.  This paper's experiments also underline that execution accuracy highly depends upon the prompt (context) fed to the model and minor changes in the context can lead to big drop in execution accuracy. I am not sure that this paper offers new insights to the community.        \n- Authors crafted these prompts manually and show that all of these transformations lead to significant accuracy drop. It's not clear 1) how many different prompts they tried for a given experiment. 2) Have they reported results for all transformations or only for those transformations which lead to accuracy drop. \n- Paper is missing crucial details related to how the code was generated. In particular, it's not clear which decoding method (greedy, sampling with temperature, etc.) is used to generate the data. Without such crucial details, it's hard to replicate the results presented in the paper. \n   ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "J_A7AqY4vZR", "reviewer": "Reviewer_Ey5D", "summary": "The paper aims to evaluate generation models in text using qualitative categories - patterns that demonstrate systematic errors as opposed to individual instance errors. The main case-study is performed on OpanAI’s Codex, where the authors find that Codex does predictable mistakes which are dependent on (a) how the input prompt is framed (b) outputs are closer to anchors, or (c) biased towards instances that are similar to highly frequent training samples. \n\nThe paper also claims that the proposed framework, which uses the concept of human cognitive biases, to elicit hypotheses of errors, can unearth high-impact errors – which for Codex amounts to generations that can delete files.\n", "questions": "- The issues highlighted in Anchoring might be related to in-context prompting. [Recent work](https://arxiv.org/pdf/2202.12837.pdf) highlights how labels are the most crucial elements in the context. In the current scenario, it is possible that the model thinks of these anchor functions as the corresponding labels which it tries to incorporate into the solution. \nWhile I agree this behavior is not ideal, we probably should consider it as a feature and not a bug. As anchoring might have beneficial usage in controlling the output in zero- or few-shot. \nWith this in mind, probably it’s better to test if the issues of anchoring persist if the model is clarified which part of the context is a demonstration (the anchor part here), and which part is the query?\n\n- Some of the attacks seem to be highly out-of-distribution prompts - For example, calling a summing function as `product_plus_2` is probably an extreme example. As such, it would be more interesting if cognitive bias patterns can be used to find adversarial prompts whose differences are more imperceptible?\n", "limitations": "There is no clear section dedicated to the limitations of the current work - maybe the authors should think about situations where cognitive bias framework would fail to elicit errors.\nAlso, are there cases where output of certain prompt attacks can never be anticipated?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n- Essentially the paper attempts to understand how large language models are sensitive to the context provided - do they focus on irrelevant information or distractors? And can their failures be categorized into frequent modes. The novelty claimed here is that the paper analyzes these failures from the lens of cognitive biases.\n\n- The paper can be seen as an extension of Checklist (Ribeiro et al.). Here, the authors rely on cognitive bias literature to design adversarial prompts with expected outputs. \n\n- Unearthing high-impact errors is an important use-case.\n\nWeaknesses:\n\n- The nature of the prompts or the output tendencies (mimicking patterns from training corpora), etc. have been studied in the literature before on different applications. For example, checking for sensitivity to context and the noise present in them is a known approach. The contribution that this paper makes is probably in collating these testing strategies and clubbing them under the cognitive bias framework.\n\n- The connection to cognitive biases, while interesting, at times seem forced. Nevertheless, it is an interesting way to look at adversarial examples and might be beneficial as it summarizes some important patterns of failure. Particularly, patterns such as framing (noise in context), and anchoring are interesting as they have wide applicability. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept