Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: This paper tries to evaluate the impact of prompt engineering and calibration techniques on smaller LLMs at reasoning tasks. Through their experimentations, they demonstrate that each strategy favours certain model and their joint effect is usually negative.
strengths_and_weaknesses: Strengths:
* The authors in this paper try to understand the impact of calibration and prompt engineering on small LLMs and use a set of commonsense reasoning benchmarks for that evaluation.
*  The authors demonstrate the idea of calibration and prompt engineering really well and concisely in their figure, which can easily be understood by the reader.

Weakness:
* The paper tries to evaluate the impact of calibration and prompt engineering, but no conclusive takeaways are provided in the paper. These strategies do not seem to follow a certain pattern, in that case, the authors should have tried to understand why a certain approach works on a certain dataset.
suggested_changes: * The authors can try to dive deep into understanding the reasons for the performance of different models. An in-depth analysis will bring more nuance to this paper.
review_title: null

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: The authors investigate the applicability of prompt engineering and calibration techniques on smaller language models (up to 3B parameters) for zero-shot multiple choice commonsense reasoning tasks. Based on experiments from five commonsense reasoning benchmarks, the results reveal that each strategy benefits certain models, but their combined effects are mostly negative.
strengths_and_weaknesses: Strengths:

* The paper clearly states the motivation, the specific experiments used for testing, and provides comprehensive experiment results that are reproducible. 
* The experiments from multiple models clearly support the conclusions made on how joint effects are mostly negative and the best approach is model specific. 
* The paper is well written with a clear flow of thoughts from the beginning to the end. 

Weaknesses: There are no apparent weaknesses, I think analysis of small models are still relevant despite what a lot of people believe. 


suggested_changes: It's probably easier for the reader if the url to the code is included within in the first two pages of the paper instead of the appendix.
review_title: null

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: This paper investigates whether prompt engineering and calibration help smaller language models in multiple choice commonsense reasoning. The key findings are that each strategy favors some language models and their joint effects are mostly negative.
strengths_and_weaknesses: Strengths:

1. This work studies the interesting topic of how to make small models benefit from advanced techniques like prompt engineering and calibration, as they help LLMs to perform reasoning. The topic would be of interest to the community.

2. Comprehensive experiments are conducted on five popular multiple-choice commonsense benchmarks. The results are convincing.

Weaknesses:

1. Some descriptions need more clarity. For example, In Figure 1 caption, "Unlike previous works, we do not bind options to symbols, like (A)." Why did the authors make this choice instead of following previous works? How about the influence on the final performance?

2. According to the results presented in Tables 1, Table 4, and Table 5, the two approaches have different effects on the performance. The authors thus concluded that "each strategy favors some language model"; however, the conclusion is too vague to give insights to readers. A more in-depth analysis of why the models can benefit from different strategies would make this work more interesting. I give some examples below to show my thoughts on interpreting the results. They are not blames but are suggested analyses in the future study. 
For example, (i) we see that CA (which is ZS with calibration) generally works well with GPT-2. Does this indicate that CA is more useful for auto-regressive models?; (ii) PE becomes more effective with larger Flan-T5 models. This aligns with the existing findings of emergent ability as larger models have a better ability for in-context learning; (iii) ZS performs the best in vanilla T5 models compared with the other prompting strategies, which tends to indicate that vanilla T5 models can hardly learn from the prompts. 


suggested_changes: Please see the weakness part above. The paper can be improved with better clarity and a more in-depth analysis of why different techniques favor different models.
review_title: null

Meta-review: metareview: Overall, this work makes valuable attempts to evaluate the impact of prompt engineering and calibration techniques on smaller LLMs at reasoning tasks. The topic would be of interest to the community. 

Pros: The paper is well-written and easy to read. The idea of calibration and prompt engineering is clearly demonstrated. The experimental results are convincing. 

Cons: Two reviewers mentioned a lack of conclusive takeaways provided in the paper. Reviewer PTuR provided detailed suggestions for the authors to interpret the results.


confidence: 4: The area chair is confident but not absolutely certain
Decision: Invite to archive