Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the effects of GPT3 on generalization, bias, uncertainty, and knowledge discovery.
strength_and_weaknesses: Strengths
- The four facets considered are important and timely given the popularity and impact of GPT3. 
- The paper is coherent and thorough.

Weaknesses
- It is hard to follow the conclusions. It would be a useful endeavor to have a summary of the main takeaways from each facet.
- It would be nice to have a list of open questions that result from this work.
clarity,_quality,_novelty_and_reproducibility: Clarity: High
Quality: High
Novelty: High
Reproducibility: Adequate, but it would be nice to know how best to reproduce the results here in.
summary_of_the_review: This is an interesting paper that looks into how reliable GPT3 is in various settings. The takeaways will be well-received by our community.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors explore reliability in GPT3, specifically in four areas: generalizability, fairness/bias, uncertainty calibration, and knowledge updates. They develop prompting methods for improving accuracy in these areas.
strength_and_weaknesses: Strengths:
- In general, it's great that these ares are getting more attention, especially in LLMs.
- The deeper analysis on prompt engineering for these specific bias use cases seems very useful (e.g., prompting with balanced demographic groups leading to more fair outputs).
- Section 5.2 (Memorization vs Updating) experimental setup was really interesting and well-designed. This felt like it could be expanded into a larger paper, with more exploration on why types of information the model memorizes (and what it has trouble forgetting)

Weaknesses:
- It feels like this paper over-claimed its novelty. In the abstract, they say that "existing research focus on models’ accuracy on standard benchmarks and largely ignore their reliability", however the GPT-3 paper has multiple pages on bias analysis, as do the Gopher paper, PaLM paper, etc. 
- Similarly, there is a lot of very relevant related work in Appendix A which should be in the actual paper. For example, in Section 3 (Social Bias and Fairness) it should be made more clear that similar analyses (e.g., WinoGender) were indeed run in the original GPT3 paper, and that the main contribution in this paper is "how these biases change under different prompting schemes in the few-shot setting" (quoted from the appendix, but not clear in the paper itself, at least to me). 
- Typos:
   - Page 7, "the 2-shot accuracy is 5.8 points worse than the 64-shot is better calibrated"
   - Page 3, "coreference resolution question answering" (needs an "and"?)
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, and seems very reproducible. Like I mentioned, most of the novelty is in the prompting methods to make specific reliability methods work better.
summary_of_the_review: If this paper's contributions were phrased as "Others have studied reliability in LLMs. We propose a range of prompting strategies to make this work even better", I would be more supportive. As it is, it seems more like the authors are presenting the main analysis of reliability in LLMs, which feels like an overstatement.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper systematically studies the reliability of GPT-3 from four key facets: generalizability, fairness, calibration, and factuality. With proper prompting strategies, GPT-3 can outperform some supervised baselines like RoBERTa and BERT on these four facets. Reliability is one important aspect of deep learning models and experiment results are comprehensive for these facets. This work shows the capabilities of large language models and sheds new insights into the reliability of prompting large language models.

strength_and_weaknesses: Strength: 
(i) Reliability is an important perspective for the deep learning community. This paper investigates the reliability of the famous model GPT-3. Comprehensive experiments are performed from four different aspects: (1) generalizability to distribution shifts; (2) social biases and fairness; (3) uncertainty calibration; and (4) factual correctness.
(ii) The experiment is comprehensive in general. For instance, for generalizability, the authors consider domain shift, perturbations and spurious correlation, adversarial attacks completely. These perspectives are common facts regarding the robustness of the model.
(iii) The conclusion is interesting for large language models, which can inspire the community further and is helpful for model deployment. For example, prompts can update memorized knowledge, which is a meaningful characteristic.

Weaknesses: 
(i) In figure 1, the authors claim that "prompt generalizes out-of-distribution" and "prompt can update memorized knowledge". However, I think other facts are influencing the final results like the scale of the model and the number of data. Can you perform an ablation study to verify that?
(ii) Since the whole paper uses empirical methods to study the reliability of the model, the number of baselines is somehow limited. For example, in Table 1, the authors only compare GPT-3 with one supervised model RoBERTa. Generalizability is a hot topic in the deep learning area and can the authors show more supervised baselines here?
(iii) I also doubt the fairness of the experiment. Since the training strategies and the number of parameters will also influence the final experiment results, it should be better to list the experiment details and the number of parameters in each experiment. 
(iv) There are mainly two differences between LLMs and other baselines. One is the scale of the model, and another is the training setting (number of data, training strategies). However, I think the architecture of different models is also important for reliability. For example, can you provide experimental results for other types of architectures like CNN?
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well-organized and clear. 
Quality: The paper focuses on empirical results for reliability and the experimental results are extensive.
Novelty: Since the paper performs practical results only, the novelty is somehow weak in my view.
Reproducibility: There is no URL (e.g., github page) for reproducibility.
summary_of_the_review: This paper studies the reliability of large language models - GPT-3 empirically and provides some interesting conclusions. Reliability is one important topic in the deep learning field and the experiment is comprehensive in this paper. Even though there exists some unfairness for further verification, their conclusions are meaningful to the deep learning community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper analyzes the "reliability" of large language models (LLM) with prompt engineering or in-context learning.  Authors consider four "facets" of reliability: (1) robustness to domain shift, (2) neutrality w.r.t. social biases (3) uncertainty calibration and (4) ability to update the learned knowledge. For each facet, authors study how LLM prompt-tuning / in-context learning compares against traditional fine-tuning of models such as RoBERTa, T5, DeBERTa, etc. For some facets, authors compare several prompting strategies in terms of their reliability. Aside from this general line of thought, the paper also contains a plethora of other results in each of the four respective "facets".
strength_and_weaknesses: ### Strengths

1. The paper studies multiple highly impactful issues; As LLM prompting is gaining new applications, it is important to understand whether or not we can "trust" it - and how to make it more trustworthy.
2. Authors report a metric ton of evaluations. If the paper's worth was measured by the "total weight" of all contributions, this one would be among the best.
3. Authors make considerable effort to make the paper easier to follow. The main sections ("facets") follow a consistent structure. Each has a short summary at the end.
4. The paper is well-written and reasonably "tidy" -- especially if we take into account the density of information.
5. Authors clearly state the limitations of their work, both in the main paper and in the dedicated limitations section.

### Weaknesses

In general, it feels like the paper attempts to squeeze too much information into a limited space. Each of the four "facets" is an established research area with its own complexity, caveats, and prior work. It feels like the paper tries to be both an overview of these facets *and* an independent analysis of each one. In each specific section (facet), the paper has area-specific flaws: inconsistent evaluation, incomplete ablation, missing some task-specific caveats


1. missing comparisons and related work in individual facets, e.g. GPT-3 social / ethical issues [1,2,3], GPT-3 uncertainty[4], GPT-3 domain adaptation[5], knowledge editing[6], retrieval-augmented question answering [7, 8].
   * for instance, in Section 5 (knowledge updating), it is unclear how the retrieval-based editing compares with [6]  -- and which of the results are novel compared to [7, 8], and how does the proposed calibration strategy compare with [4].
   * please do not consider [1-8] as an exhaustive list of related work. I am not an expert in most of these facets. My point is: since a non-expert (me) can name multiple missing baselines, authors are likely to find more relevant work if they deliberately search for it.

2. As an overview of reliability: the choice of four specific facets feels arbitrary: e.g. it is unclear why we focus on generalizability and not, say, robustness to adversarial [9,10,11], or prioritize mitigating social bias over avoiding toxicity[2,12].

4. (Most) results are presented as-is, with no analysis. For instance, in Section 2, GPT-3 has a smaller generalization gap than a fine-tuned MLM model -- but what exactly causes this behavior?
  * Example A: Is this because of prompting v.s. fine-tuning? -> What if we use prompting for the MLM model?
  * Example B: Is this because of language model's training objective?  -> what if we compare similarly-sized CLM and MLM, e.g. T5-11B vs BLOOM/BLOOM-13B -- and try prompt tuning in the baseline model
  * Example C: Is this because of the training data? -> What if we compare CLMs and MLMs trained on similar *public* datasets?
  * Note: it is not a weakness that authors did not perform every possible analysis - but it would help the paper


5. (Some) claims are overly broad, e.g. "Language model probability and self-consistency frequency can produce better calibration than a supervised DPR-BERT model, specially on OOD test sets" -- this claim requires testing multiple language models in different setups.

6. It is not always clear how authors selected the baseline models for each task, e.g. S2 only has roberta, S3 adds deberta, S4 only bert, S5 is only T5

7. In T7, authors evaluate uncertainty calibration through selective prediction (e.g. accuracy on top-10% most confident). However, it is difficult to assess: what portion of the selective accuracy gain is due to better uncertainty calibration, and what portion is simply a more accurate base model. Prior art typically uses additional uncertainty metrics to decouple these two effects.



[1] https://aclanthology.org/2021.nuse-1.5/

[2] https://cdn.openai.com/palms.pdf

[3] https://aclanthology.org/2021.acl-long.416/

[4] https://arxiv.org/abs/2205.14334

[5] https://arxiv.org/abs/2112.08786

[6] https://arxiv.org/abs/2110.11309

[7] https://arxiv.org/abs/2005.11401

[8] https://arxiv.org/abs/2206.06520

[9] https://simonwillison.net/2022/Sep/12/prompt-injection/

[10] https://arxiv.org/abs/1908.07125

[11] https://arxiv.org/abs/2209.02128

[12] https://arxiv.org/abs/2112.04359






clarity,_quality,_novelty_and_reproducibility: 

**Clarity & quality:** the paper is well written. Few small typos (below) do not affect the overall quality. If I had to cram 4 research papers in 9 pages, I would probably do much worse.

**Novelty:** the paper has a large number of assorted finding in the four facets of reliability. To the best of my knowledge, at least some of them are not covered by related work, at least in their specific formulation with LLM prompting. Unfortunately, I am not an expert in all four research directions present in the paper and may have missed something (hence my confidence score).


### Minor stuff

> P1:  and their evaluation methods may not feasible on the much larger GPT-3 model.

May not BE feasible. Also, if you decide to keep the current direction of the paper, please explain why they are not feasible (possibly later in the paper).

> S3:   Random shuffling achieves the lowest bias gaps (...) Interestingly, putting either Pro-Bias or Anti-Bias examples at the end leads to much larger bias gaps. This contradicts the recency bias …

[mostly nitpicking] This contradiction was not clear to me. If you randomly order sentences, then, assuming recency bias, the most recent sentence is equally likely to be biased either way, and I thought it would average to zero. Perhaps it would be best to explain this contradiction in more detail.

summary_of_the_review: 

This paper combines 4 related (but separate) short research papers and an overview. To the best of my knowledge, all four research directions could be upgraded to a standalone short/full paper, through the addition of baselines, missing related work, fixing some task-specific issues and providing further justification of the already claimed results. At least 2 of 4 papers (and likely all 4) would stand a good chance to be accepted here or on *ACL / EMNLP-type conferences. The overview part has its own merit: should authors modify it to add relevant prior work and focus on analyzing existing results, it would be a valuable resource to beginners.

However, in the current form, each of the four research directions is only given a couple of pages, lacking background and task-specific nuances - and adding this information would be difficult because of the size limit, and deferring it to the appendix would make the paper difficult to follow. I would consider one of two strategies:

* splitting the paper: giving each research case at least a short standalone paper -- where you will have space to better support your main claims.
* reformatting as a meta-analysis: focus on summarizing existing work and pinpoint open problems in model reliability; possibly consider more facets, or provide a better justification for the four aspects currently in the paper.

To clarify: that is what I would have done. I will be glad if authors prove me wrong by finding a more elegant solution to these issues.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was a study on the gpt-3 api, namely code-davincin-002. 

The paper analyzes the api across multiple facets. 

Reviewers were generally supportive of this paper and the prompting changing the reliability of gpt-3 is pretty interesting.


justification_for_why_not_higher_score: It's interesting but not that interesting. It's not bad but not bad only. 
justification_for_why_not_lower_score: I think this paper has interesting findings that would be interesting.
Decision: Accept: poster