Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: The paper discusses the problem of evaluating a piece-wise function using large language models. It presents a phenomena where adding biased examples actually decreases the function evaluation accuracy.
strengths_and_weaknesses: Strengths

* This work investigates an interesting problem in problem-solving/reasoning capabilities of LLMs. Domains where LLMs are not very accurate or non-intuitive behaviors are very important to find and this paper does a good job of presenting such a domain.

* The experiments regarding easy/hard problems and adding more negative examples in the appendix are also interesting. While I understand, they may be beyond the scope of the size requirements for this workshop, a way to concisely present these results together will make the paper stronger.

Improvement scopes

* I am not convince about the conclusion regarding balanced and biased prompts. For example, one case of biased, as presented in the paper, is 0 examples for one, and none for the other. What would happen if we had 2 examples for one and 4 for the other etc?
* This is not really a weakness of the paper, but rather talks about the fast pace of LLM growth. By the time I read the paper, GPT-3 is already not the best Open-AI model. Additionally, there exist models like instructGPT which allows different types of prompt engineering to circumvent such problems. A more complete work, while still in the scope of this workshop, could compare more models, more ways to define piece-wise functions etc.
suggested_changes: Figure 1 should also include OPT/GPT3 without examples, similar to what was shown in Figure 3.

In Appendix D.2, I assume the authors meant to write "much stronger inverse scaling when *biased* examples are provided."
review_title: null

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: The paper proposes an evaluation of LLM with few-shot examples that are misrepresentative of the target input and shows that in the context of integer-to-integer piecewise functions, the performance inversely scales with model size.
strengths_and_weaknesses: Strengths:
1. The task is well-defined.
2. The experiments with both OPT and GPT models support the claim.

Weakness: 
1. It is unclear whether the results are due to the model's misunderstanding of the task definition or failure to understand which function to use. 

Question: Will chain-of-thought style reasoning where the model first identifies which piecewise function to use and then generate the output might scale differently? 
suggested_changes: A lot of details are included in the appendix which is crucial to understand the experimental setup. I think it would be super helpful to include one example prompt (B.4) in the main paper.
review_title: null

Decision: Invite to present