SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step
  Reasoning
Authors: name: Ning Miao; name: Yee Whye Teh; name: Tom Rainforth
Published: 2023-08-01 10:31:36

Abstract: The recent progress in large language models (LLMs), especially the invention
of chain-of-thoughts (CoT) prompting, makes it possible to solve reasoning
problems. However, even the strongest LLMs are still struggling with more
complicated problems that require non-linear thinking and multi-step reasoning.
In this work, we explore whether LLMs have the ability to recognize their own
errors, without resorting to external resources. In particular, we investigate
whether they can be used to identify individual errors within a step-by-step
reasoning. To this end, we propose a zero-shot verification scheme to recognize
such errors. We then use this verification scheme to improve question-answering
performance, by using it to perform weighted voting on different generated
answers. We test the method on three math datasets-GSM8K, MathQA, and MATH-and
find that it successfully recognizes errors and, in turn, increases final
predictive performance.