Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: This paper proposes a case study of Chain-of-Thoughts (CoT) prompting in Large Language Models (LLMs) for batch data in streaming settings. The authors investigated the correctness and the depth of CoT in batch settings.
strengths_and_weaknesses: Weakness
1. "we substitute ground-truth rationales that produce the correct answer with
Zero-Shot-CoT that produces the wrong answer."

How did the author design the wrong answer and the wrong rationales? 
How did the wrong answers deviate from the right answers?

(The wrong answer generation can be viewed from a relevant NLP task perspective-- automated multiple choices questions (MCQ) generation. Generating good distractors for MCQ is a challenging and authors first provide heuristics to generate difficult distractors)

2. The authors did not share the source codes or sample prompts.

Strength
The Shallow vs Deep CoT is promising. The Shallow CoT demonstrated better accuracy than the Deep CoT in several batches for all four datasets.
suggested_changes: 1. Please provide how the wrong answers were generated. 
2. Please provide sample wrong answer prompts.
review_title: null

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: The paper proposes an analysis of chain-of-thoughts for Language Models. Correct and deep chains of thoughts are replaced with wrong and shallow ones, showing a limited loss in performance.
strengths_and_weaknesses: STRENGTHS
==========
I enjoy this kind of analysis, where intuitive principles and general beliefs are questioned. I think the paper touches on important topics, given the recent advancements in Large Language Models.

WEAKNESSES
==========
- I enjoyed the introduction even if I am not an NLP expert, but adding a bit more context and application of the proposed method would be good. Why do we want to replace correct with incorrect CoT? From my understanding, this could lead to faster predictions. If it is the case, can we measure such a gain? I think answering these questions would strengthen the paper's contribution.
- Of course, it is outside the paper's scope, but it is natural to wonder if the findings hold regardless of the language and the underlying model. Maybe this aspect can be listed in the work limitations/future works.

Minors:
- y-axis of the plots does not start from 0. I see that in this way, differences are clear, but it also takes a moment for the reader to interpret the results correctly. I would suggest highlighting this fact by starting the y-axis from 0 and breaking the line (e.g., with a dashed line), or by replacing the bar plots with tables.
- The text refers to Figure 1(a) and Figure 1(b), which is, of course, clear ((a) is left and (b) is right), but it might be good to add these labels to the graphs
suggested_changes: To improve the submission, I suggest the following changes:
1) Discuss the applicative case, and if possible, quantify the advantage (if any) of Shallow and Wrong CoTs.
2) Mention limitations and future works, pointing out which could be possible to achieve starting from the findings of the paper
3) Modify the graphical items as mentioned above
review_title: null

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: This paper presents a case study that examines the impact of rationale correctness and depth on the performance of CoT in a streaming batch setting. To assess correctness, the authors replace the gold rationales with incorrect ones produced by Zero-Shot-CoT. To assess depth, the authors use the number of \n to differentiate between deep CoT and shallow CoT. 
strengths_and_weaknesses: Strengths 

1. Present a case study on how to construct prompts in a streaming batch setting which is useful for other research in this area.

2. The experiments are thoroughly conducted on various datasets.

Weaknesses

1. Some experimental details are missing, and the presentation can be further improved.  Due to space limitations, it is understandable, but I would like to see some explanation on the following aspects:

(1) In section 4.1, how to ensure that more than 50% of the rationales are incorrect? 

(2)	How to manually partition the dataset into batches? Do you do it randomly?

(3)	In section 4.2, how to partition an original batch into deep CoT and shallow CoT? If so, I do not think the comparison is fair, since both Zero-shot-CoT and Auto-CoT are experimented on the full original batch. 

(4)	In figure 1(right), according to the paper “Automatic chain of thought prompting in large language models”, in the first batch, the Auto-CoT and Zero-Shot-CoT should obtain equal accuracy since they are actually the same in the first batch. Why figure 1 shows a quite different case?

2. The claim is not very well supported.

The paper mentioned that ”Additionally, promptings that consist of shorter rationales exhibit superior performance compared to those with lengthier rationales.”  However, I speculate that for more difficult questions, the rationale may be longer due to the need for additional reasoning steps. Therefore, how can the authors prove that the impact on overall accuracy is due to the depth of the rationale rather than the level of difficulty of the question itself?

suggested_changes: 1. Some necessary explanations can be added. Please see the details in the Strengths And Weaknesses section.

2. Section 4.2 can be further improved to make the experimental design more reasonable and solid.
review_title: null

Meta-review: metareview: All reviewers agree that this work studies a promising direction. The analysis also questions intuitive principles and general beliefs that will be of great interest to the NLP community. The writing quality is generally good. 

The major concern is the clarity of the experimental setting. The authors may consider improving the paper following the reviewers' suggestions.


confidence: 5: The area chair is absolutely certain
Decision: Invite to present