Thinking before Choosing: Human Logic-Based Reasoning Network for Multiple-Choice QA
Authors: Xin Guan, Biwei Cao, Qingqing Gao, Bo Liu, Jiuxin Cao


Reviewer paper summaries: Summary:
This paper deals with the task of multiple-choice question answering. They propose a multi-stage framework for the task which consists of three stages. In the first stage, the employ a generative model which generates an elaboration of the question and posits a potential answer. It also generates a judgement for each choice and if it is deemed in correct, what should be a correct answer. In the second stage, they apply multi-head co-attention to compare the answers with each other. In the final stage, they use the embeddings from stage 2 to make the final prediction. They perform  experiments on multiple dataset and show that their model outperforms the chosen baselines significantly. They choose T5-Large as baselines, and their model uses T5-Large in the first stage. They fine-tune the LM in stage 1 using the gold answer. Further they perform ablation analysis to understand the impact of each stage on the final performance. Next, they present a CoT style prompting approach based on their pipeline design and show that it outperforms two other prompting approaches including CoT.


Contributions:
1) Interesting breakdown of stage-wise approach to MCQA where understanding the question, choices, comparing the choices and then making a final judgement are separated.
2) Empirical evaluation of the proposed approach on 5 datasets compared to multiple baselines.
3) Ablation analysis, interesting prompting approach that builds on top of the proposed breakdown of the task.
Summary of strengths: Strengths:
1) Three stage breakdown of MCQA task is intuitive and aligns with human approach to the MCQA task. It is interesting to see a prompting approach built on top of this outperform existing prompting approaches (although they are old and rudimentary).
2) Empirical results show that the proposed model outperforms the chosen baselines on the multiple datasets.
3) Presented ablation study shows that each of the stages contributes to the final performance. Further studies to use the approach to develop prompting approaches for existing LLMs seems useful as well.
Summary of weaknesses: Weaknesses:
1) Unexpected baseline results: Presented results of baselines are significantly lower than existing state of the art results on the datasets. For example, a much smaller model such as RoBERTa (125M-400M parameters) (https://arxiv.org/abs/1909.11764) achieves close to ~85% on ARC-easy dataset and ~68% for ARC-challenge dataset. In comparison, presented results in this work are 70.6 and 49.2 respectively. This seems way lower than what is possible with the baseline (T5-large). Significant justification of why this is the case is required. Else, it is hard to understand why there is such a discrepancy in the baseline performance. This also raises questions about whether the gains hold even on a well-trained baseline. Is the baseline T5 model fine-tuned  or used as is? I looked at appendix C1 but it still doesn’t answer this. 
2) Misalignment between motivation and approach: L101-115 in the Introduction states that the motivation behind the stage 1 is to analyze the question and choices. The training approach described in sec 3 doesn’t align with it at all. The training data only consists of gold answers stated directly rather than any analysis or rationale of why this choice should be the correct one. There are no attempts to elicit such information from the pre-trained LLM which was used in this stage. Rather, it was fine-tuned to generate the gold answer itself. This seems counter-intuitive.’
3) While it is quite interesting that their prompting approach outperforms CoT and PS-CoT, it is not clear why. Further presentation of case study results which show why the improvement occurs and what the current prompting approach is able to do better than those approaches is needed. Is that specific to the MCQA datasets that are chosen? Or is the proposed approach a stronger and more generalizable approach? These questions are unanswered in the current version of the paper.
Comments, suggestions and typos: -
Overall assessment scores: 2