Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: In this submission, the authors apply chain-of-thought prompting for medical diagnostics. They evaluate on DDXPlus, where they achieve a significant improvement in accuracy.
strengths_and_weaknesses: Strengths:
- Automatic diagnosis is an important and intestine field of study, and important to increase accuracy as much as possible
- The experiments are a good indication that chain-of-thought prompting can have a significantly impact in reasoning in medical diagnosis
- Prompts are available in the appendix
- clear presentation

Weaknesses:
- The experiments are limited. It is not clear if the chain-of-thought prompting is the sole reason for the improvement or the choice of examples.
- It is not clear what “we invite a physician” means. Have they been paid? Is it an author?

suggested_changes: An experiment showing that the claim (of applying cot is useful) is robust when considering a different set of examples is missing.
A little more information on the evaluation is needed (see weaknesses above).
review_title: null

final_score: rating: High Impact (HI): a submission which meets the reviewing criteria and is predicted to make an impact on the field
review_content: summary_of_contributions: This paper proposes a new framework for improving the performance of dialogue systems in automatic diagnosis (AD), which requires knowledge-intensive, multi-step diagnostic reasoning. The authors show that their conversation system with DR-CoT prompting performs 15% better on diagnostic accuracy than the conventional prompt.
strengths_and_weaknesses: STRENGTHS
The paper proposes a novel approach to improving the performance of dialogue systems in automatic diagnosis by extending the Chain of Thoughts (CoT) approach to medical reasoning and incorporating a Diagnostic Reasoning (DR-CoT).
The authors propose a few-shot LLM-based dialogue system for AD, which in principle has an unbounded diagnosis label set.
The authors evaluate their proposed framework using a language-model-role-playing evaluation framework that simulates realistic patient-doctor interactions.

WEAKNESS
The paper should provide a more detailed explanation of the evaluation metrics used for measuring the performance of the dialogue system with DR-CoT prompting
suggested_changes: I suggest that, rather than putting key explanations into the appendix making the reader have to go to the bottom to get the broad clear concept, it can be summarized in the result and conclusion section while potentially reducing the wordings under the introduction section. This, I feel is because the author is constrained to a two-page limit. It can be better
review_title: null

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: This paper extends chain-of-thought (CoT) prompting to medical reasoning for automatic diagnosis. The method is simple but effective, by mimicking the process of forming differential diagnosis (DDx) in patient-doctor interaction. It also proposes a reliable role-play framework for automatic evaluation, and shows that DR-CoT performs significantly better than standard CoT on this testbed.
strengths_and_weaknesses: Strengths

1. Extend CoT prompting of Large Language Models to a new but important area, automatic diagnosis beyond conventional arithmetic, commonsense, and symbolic reasoning tasks.

2. This method is inspired by real patient-doctor interaction, which is explainable, and effectively leverages various abilities of LLMs like retrieval and summarization.

3. The method is robust since the improvements on out-domain test sets are more significant.  

4. The language-model-role-play interactive evaluation framework is quite novel. The idea can be expanded to a wider range of application scenarios, and the author also shows its reliability through expert assessment.

Weaknesses

It’s better to compare DR-COT with some other baseline models on DDXPlus to see if it is good enough. As I see, the accuracy is only 20-30, which seems to be dangerous for a medical diagnosis. 

suggested_changes: 1. In Figure 1, the meaning of the x-axis, i.e., # of dialogue turns, is vague. More details are needed to be explained.

2. Better to put Appendix C into the main body.
review_title: null

Meta-review: metareview: This paper proposes a novel approach based on chain-of-thought prompting for automatic diagnosis, which is an important research field. The paper is generally clearly written. Most of the claims are well supported. The key concern is the lack of analysis on the robustness of CoT in the concerned task.
confidence: 5: The area chair is absolutely certain
Decision: Invite to present