Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors assess the quality and reliability of open ended systems such as GPT-3 and Codex from the perspective of cognitive biases. The authors primarily focus on the application of code generation and design examples that elicit commonly made errors by these systems for the task of code generation and completion. The authors primarily focus on the 4 different set of cognitive biases: (1) framing effect; (2) anchoring bias; (3) availability heuristic and (4) Attribute substitution.

The main contributions:
- Creation of framework that contains hypothesis and prompts that extend existing cognitive bias methodology. The hypothesis and examples created can measure the model sensitivity to these transformations. The framework is also capable of discovering high impact errors and other types of errors made by the model.

strengths_and_weaknesses: Strengths & Weakness:
- The paper studies an important problem of robustness of large scale LM's such as Codex and GPT-3 from the task of of code generation. Using cognitive bias as a way to measure robustness of these models is an interesting and novel approach.
-  Results in the paper show that by manipulating exhibits different learning effects and thus resulting in significant drops in performance. However, claiming these to be exhibits of cognitive biases seems to be a bit of a stretch. Most of the errors discovered could be novel from the perspective of code generation but prior literature in the area of robustness and fairness and prompt engineering shown that these large scale model are susceptible to unwanted context added to prompts


questions: - it would be better if the authors could add more explanation about how the drop in performance related to cognitive biases. 
limitations: The authors address limitations of systems such as GPT-3 and Codex. However, there is no mention of limitations of designing prompts
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper aims to evaluate generation models in text using qualitative categories - patterns that demonstrate systematic errors as opposed to individual instance errors. The main case-study is performed on OpanAI’s Codex, where the authors find that Codex does predictable mistakes which are dependent on (a) how the input prompt is framed (b) outputs are closer to anchors, or (c) biased towards instances that are similar to highly frequent training samples. 

The paper also claims that the proposed framework, which uses the concept of human cognitive biases, to elicit hypotheses of errors, can unearth high-impact errors – which for Codex amounts to generations that can delete files.

strengths_and_weaknesses: Strengths: 

- Essentially the paper attempts to understand how large language models are sensitive to the context provided - do they focus on irrelevant information or distractors? And can their failures be categorized into frequent modes. The novelty claimed here is that the paper analyzes these failures from the lens of cognitive biases.

- The paper can be seen as an extension of Checklist (Ribeiro et al.). Here, the authors rely on cognitive bias literature to design adversarial prompts with expected outputs. 

- Unearthing high-impact errors is an important use-case.

Weaknesses:

- The nature of the prompts or the output tendencies (mimicking patterns from training corpora), etc. have been studied in the literature before on different applications. For example, checking for sensitivity to context and the noise present in them is a known approach. The contribution that this paper makes is probably in collating these testing strategies and clubbing them under the cognitive bias framework.

- The connection to cognitive biases, while interesting, at times seem forced. Nevertheless, it is an interesting way to look at adversarial examples and might be beneficial as it summarizes some important patterns of failure. Particularly, patterns such as framing (noise in context), and anchoring are interesting as they have wide applicability. 

questions: - The issues highlighted in Anchoring might be related to in-context prompting. [Recent work](https://arxiv.org/pdf/2202.12837.pdf) highlights how labels are the most crucial elements in the context. In the current scenario, it is possible that the model thinks of these anchor functions as the corresponding labels which it tries to incorporate into the solution. 
While I agree this behavior is not ideal, we probably should consider it as a feature and not a bug. As anchoring might have beneficial usage in controlling the output in zero- or few-shot. 
With this in mind, probably it’s better to test if the issues of anchoring persist if the model is clarified which part of the context is a demonstration (the anchor part here), and which part is the query?

- Some of the attacks seem to be highly out-of-distribution prompts - For example, calling a summing function as `product_plus_2` is probably an extreme example. As such, it would be more interesting if cognitive bias patterns can be used to find adversarial prompts whose differences are more imperceptible?

limitations: There is no clear section dedicated to the limitations of the current work - maybe the authors should think about situations where cognitive bias framework would fail to elicit errors.
Also, are there cases where output of certain prompt attacks can never be anticipated?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper explores the robustness of large language models through the lens of 4 human cognitive biases - framing, anchoring, attribute substitution, and availability. Results show that OpenAI's Codex program synthesis model is not robust to changes in prompts that are inspired by these 4 biases.
strengths_and_weaknesses: **Strengths**
* Well-written and easy to understand for the most part (especially introduction and some figures)
* Categorizes a few types of errors for program synthesis models and develops interesting evaluations for them (i.e. functional accuracy + custom evaluations for each error type)
* Experiments demonstrate that Codex, and likely other language model-based program synthesis models, are not robust to certain types of perturbations
* Example of anchoring applied to GPT is especially interesting, since it shows a similarity between how language models (LMs) and humans behave 

**Weaknesses**
* Paper seems to over-generalize contribution - e.g. claims to study "failure modes of large language models", but contributions are rather specific to prompt robustness. I would expect a paper with this general of a title and abstract to study more errors more comprehensively (e.g. anaphora resolution, entailment, long context, etc.). I recommend narrowing down the claims made in this paper
* Contribution is relatively simple (changing prompts and measuring robustness). Could expand scope by increasing types of cognitive biases tested, or more experiments on GPT
* Missing some key details for reproducibility and understanding
  * Needs details on how functional accuracy is checked (e.g. environment, open source code)
  * Figure 3 - unclear what is going on. Maybe add full example and/or more explanation in caption
  * Figure 5 - clarify what colors mean, and what is input vs. generated
  * Section 5 - what examples are used here, and how many? How were examples generated and evaluated?
* Section 3.2.3 on Availability bias unsubstantiated
  * L213-214 - “Our results suggest that Codex can err by outputting solutions to related, frequent prompts in the training set.”. Need to show evidence that unary operations applied before binary in training set to make this claim. Recommend sampling training set and counting unary-first vs. binary-first
questions: **Major**
* Figure 4 (right) - explain why “fraction of solutions” containing anchor function lines is so low when # of canonical solution lines prompted is 0
* 3.2.4 - relationship to attribute substitution cognitive bias is a stretch. Not very convincing that confusing the model with conflicting function name vs. desired behavior is an example of model erring “by using simple-but-incorrect heuristics to generate solutions”. Recommend replacing this with examples from Section 5 and removing section 5, which doesn't seem to warrant a separate section
* Would be nice to see more cognitive bias methods applied to GPT, since it shows diff/similarity between LM and human

**Minor**
* Footnote 2 - please share how many examples were filtered out
* Figure 4 - clarify whether this is on humaneval only
* Print-var results + add-var results sections - recommend rounding results to whole numbers. Tells same story, easier to digest
* Table 3 - would be interesting to plot anchor values vs. prediction (e.g. expect to see as upper bound anchor increases, GPT-3 prediction increases) to clearly demonstrate anchoring effect
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes to use human cognitive biases as inspiration for types of errors that large language models may make, as a means of finding and catching such errors. They run experiments on four such cognitive bias-inspired errors on code generation with Codex models. They also include two additional sets of smaller experiments, one with GPT-3 with a pre-existing human cognitive bias experiment, and another with Codex on code deletion as a means of emphasizing high-impact errors.
strengths_and_weaknesses: As a foreground, my view on the use of human cognitive biases as a motivation in this paper is that it is a useful and valid approach for coming up with potential error categories, but ultimately the results in the paper can be interpreted and have value independent of the associated human cognitive biases. Because of this, and my relative unfamiliarity with human cognitive bias work, I will focus my comments on evaluating the experiments primarily from the point of a view of a machine learning researcher.

Strengths:

- The paper is exceedingly clear, well-written and easy to follow. The examples of the prompt formats is extremely welcome and papers that do prompt-based experiments would do well to follow this example.
- The experiments are well-structured and well-motivated. While I do have concerns regarding some of the experimental setup (described below), the logical flow from human cognitive biases to a proposed equivalent model experiment to the metrics being measured are generally clear, reasonable, and cover the questions a critical reader might have (e.g. also measuring copying of the irrelevant propmt).

Weaknesses:

- Experimental setup / evaluation. My major concern with this paper is the experimental design, and whether the experiments correspond to the research question being asked. I will discuss my concerns with each of the experimental setups here. This section includes both what I consider weaknesses as well as questions/additional results I would like to see.

1) Framing.

I have an issue with the choice of "irrelevant preceding function" and the interpretation of model behavior, in that I do not share the authors' interpretation that the prepended functions are irrelevant. For instance, given a prompt:

```
def add(x, y):
    """Adds numbers"""
    raise NotImplementedError()

def multiply(x, y):
    """Multiplies numbers"""
```

I *would* expect the model to output `raise NotImplementedError()`. This looks like a file that is work-in-progress, and to my mind the model is correctly extrapolating from the prompt. In the authors' words, I do not believe this is "semantically irrelevant information". A better example (and this overlaps somewhat with anchoring) would be something like:

```
def divide(x, y):
    """Divides numbers"""
    assert y != 0
    return x / y

def multiply(x, y):
    """Multiplies numbers"""
```

In this case, if the model outputs `assert y!= 0`, then we can conclude that the model is actually using the prepended function in an erroneous/unhelpful way. Put another way, if the experimenter wants the model to generate functional code, they should to prompt *with* functional code. This should be a pretext for all experimental involving functional accuracy.

2) Anchoring.

I think the setup and results in this experiment are clearer than for framing. One concern I have is with the confounding factor of the anchor function having the same name as the function to be completed. An experiment with a similar name instead (e.g. common1 vs common2) would clarify this result.

3) Availability.

The authors conjecture without evidence that "Programmers tend to apply unary operations first", and use this as the basis for their conclusion that Codex has learned to output "related prompts that occur more frequently in the training set". From the results presented, we can take away that the model appears more biased toward unary-first solutions, but it seems difficult to draw conclusions relating to the training set without further evidence.

I also have a milder concern regarding the formatting of "docstrings" in this and the attribute substitution experiments. Docstrings should go under the function signature and describe the function. The "docstrings" in these two experiments are provided above the function signature, and written in the form of an instruction "Write a function that...", which is not how docstrings are used in practice, and therefore should not be part of the expected behavior of the model.

4) Attribute substitution.

My primary concern with this experimental setup is that it introduces contradictory function names and docstrings (see also the discussion above), and asserts that the model ought to take the dosctrings as the ground truth and that the function names are misleading. It is not clear to me that this has to be that case. I could conversely argue that in practice, documentation updates often lag behind code changes, and that the function name is a more reliable source-of-truth than docstrings. Under this interpretation, what the results in Table 2 show is that the model mostly "correctly" adheres to the function name and is only mildly affected by the "incorrect" docstring information. Even in this lens, I think the results are still valuable, because the conflicting information appears to introduce a set of "Other errors". I recommend the authors revisit and rethink the design and interpretation of this experiment.

(Admittedly, this is confounded in the "docstring" sub-experiment, since the only source of information is the dosctring.)

6) Deletion

This suffers from the same issue as the attribution substitution experiments, though not to the same degree (`delete_all` can be reasonable seen to be context dependent). That said, I wonder if the results would be different if the function were named more helpfully (`delete_all_with_libraries`).

In summary, while I think this approach for finding model errors is valid and worthwhile, the current experiments fall short of demonstrating the biases in a convincing manner. Addressing these issues would make a meaningful difference to my review.

- Scale of the experiments. The experiments tend to be on a small scale with a small-ish number of examples. Particularly given the potential of models to be sensitive to the exact phrasing of prompts, this is a moderate but not large concern.
- Lack of evaluation on open models. As the authors acknowledge, the Codex model is private and many details regarding the architecture and training are not public. This makes it difficult to ascertain, for instance, that the models have no had additionally curated data to avoid some of these errors (i.e. a more naively model may perform even worse than reported in the paper). While I acknowledge that this paper is primarily focused on Codex and duplicating experiments on an additional model with a separate setup is significant additional work for the authors, I think it is fair to say that experiments solely on a set of non-public models with many unknowns does hurt the scientific contribution of the paper and calls into question whether these results generalize. Even a small set of experiments on a similar more public model (e.g. CodeGen) that demonstrate equivalency of results would help assuage the reader.
questions: - Most of my specific questions about the experiments are covered in the above section.
- What decoding configuration was used for Codex/GPT-3? If temperature was used (i.e. non-greedy decoding), it should be reported and potentially run with multiple runs. The checklist mentions "deterministic rollout", but explicit clarification of the decoding configuration would be helpful. 
limitations: The authors have framed the work in terms of being helpful to catch errors of large language models, which I think is reasonable. The work does have experiments testing the propensity for models to indiscriminately delete files, but I believe the experiments are conducted within a reasonable and contained setting.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Authors highlight the fragile nature of the GPT-3 style code generation models. Inspired from cognitive biases research, authors design different manual input transformations (prompt tuning) where they add some irrelevant information to the model. Using experiments with Codex model, they show that model's code generation performance drop significantly when they fed modified input to the model. It's worth highlighting that the proposed transformations are generally non-adversarial in nature.    
strengths_and_weaknesses: Strengths:

- Proposed transformations are very intuitive and are well-motivated by prior research in cognitive science. 

Weaknesses:

- While the transformations described in the paper are intuitive, some of them are hard to scale. In particular, one has to design such transformations manually for a given programming language setup. Further, authors have not provided an easy to use list of transformations (or dataset version) which other researchers can reuse for future experiments. 
- Prior work [1] show that problem description, context are crucial to execution accuracy.  This paper's experiments also underline that execution accuracy highly depends upon the prompt (context) fed to the model and minor changes in the context can lead to big drop in execution accuracy. I am not sure that this paper offers new insights to the community.        
- Authors crafted these prompts manually and show that all of these transformations lead to significant accuracy drop. It's not clear 1) how many different prompts they tried for a given experiment. 2) Have they reported results for all transformations or only for those transformations which lead to accuracy drop. 
- Paper is missing crucial details related to how the code was generated. In particular, it's not clear which decoding method (greedy, sampling with temperature, etc.) is used to generate the data. Without such crucial details, it's hard to replicate the results presented in the paper. 
   
questions: - One can argue that framing effect is by design and is not an unexpected behavior. Based on the GPT-3 paper's in-context learning experiments, we expect model to imitate the behavior of the functions provided in the context. Hence, when a preceding function throws `NotImplementedError`, it's probably reasonable for model to generate `NotImplementedError` as *one of the answers* (assuming we sample multiple answers with a temperature). 
- Which decoding algorithm are you using for generation? Is it greedy decoding or are you using some sampling algorithm? Further, for code generation tasks, a common practice is to use sampling and report pass@K metric as it reduces variance in the results. Have you observed the same accuracy drop with pass@K ?
- Do you have any counterexamples i.e. transformations where you did not see any accuracy drop?  
- "Open-ended systems" is a very broad term. Since this paper mainly focuses on the code generation task, I would recommend to use "open-ended generation" instead of using “open-ended systems”. 

**Typos**:
- L41: that that 
- L111: that that 
- L140: prictable -> predictable 
- L298: able construct -> able to construct 


References: 
1. Li, Yujia et al. “Competition-Level Code Generation with AlphaCode.” ArXiv abs/2203.07814 (2022)
limitations: I don't see any major concerns related to negative societal impact.
review_title: null

Meta-review: metareview: This paper aims to qualitatively categorize errors by large language models for generation tasks (i.e., summarization, program synthesis), drawing inspiration from 4 human cognitive biases: framing effect, anchoring bias, availability heuristic and attribute substitution. The work includes case studies on Open AI's Codex and GPT-3, and demonstrates that Codex makes predictable mistakes based on the framing of the input prompt, outputs that are closer to anchors and instances that are similar to frequent training examples. The paper uses the proposed framework to elicit high-impact errors. The paper is interesting and is clearly presented, the experiments are well-designed. The rebuttal includes additional experiments suggested by the reviewers with different prompts and additional models.
confidence: Less certain
Decision: Accept