A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains (2024)

Abstract: Prompting language models to provide stepby-step answers (e.g., “Chain-of-Thought”) is the prominent approach for complex reasoning tasks, where more accurate reasoning chains typically improve downstream task performance. Recent literature discusses automatic methods to verify reasoning steps to evaluate and improve their correctness. However, no fine-grained step-level datasets are available to enable thorough evaluation of such verification methods, hindering progress in this direction. We introduce REVEAL: Reasoning Verification Evaluation, a new dataset to benchmark automatic verifiers of complex Chain-ofThought reasoning in open-domain question answering settings. REVEAL includes comprehensive labels for the relevance, attribution to evidence passages, and logical correctness of each reasoning step in a language model’s answer, across a wide variety of datasets and state-of-the-art language models. Available at reveal-dataset.github.io.

heading: 1 Introduction
text: Complex reasoning tasks involve answering questions that require multiple steps of reasoning (Welbl et al., 2018; Talmor and Berant, 2018). Addressing these questions may require open-domain knowledge (Geva et al., 2021), mathematical reasoning (Cobbe et al., 2021; Hendrycks et al., 2021), logic (Dalvi et al., 2021), and so on. Reasoning chains— breaking the task into multiple steps explicitly—is useful for improving performance in such tasks, with LMs demonstrating better performance when encouraged to generate the reasoning chain behind their answer (Lampinen et al., 2022; Zelikman et al., 2022; Hu et al., 2023), commonly implemented via Chain-of-Thought (CoT) prompting (Wei et al., 2023, see Fig. 1 for an example).
Evaluation in such settings is traditionally limited to evaluating only whether the final answer is correct (Chowdhery et al., 2022; Wei et al., 2023;
Wang et al., 2023). However, correct reasoning chains have been shown to be correlated with better final answers (Jung et al., 2022), with recent literature proposing automatic methods for verifying the quality of the reasoning chains themselves along various axes such as informativeness, relevance, factuality and logical correctness (Golovneva et al., 2023; Press et al., 2023; Opitz and Frank, 2021; Leiter et al., 2022). While such verification methods are a promising direction for improving reasoning in LLMs, it is not clear how to evaluate them due to the lack of high-quality, step-level annotated data, and collecting such data was shown to be difficult (in terms of reaching high inter-annotator agreement) and costly (Golovneva et al., 2023).
We present REVEAL (Reasoning Verification Evaluation), an evaluation benchmark for complex reasoning verifiers.1 REVEAL covers a diverse set of reasoning skills, complexity levels, and knowl-
1At huggingface.co/datasets/google/reveal. We adopt practices by Jacovi et al. (2023a) against data contamination and request that any future redistribution or usage of the data respects the same constraints.
1 4615
edge domains. It contains 704 unique questions from 4 popular QA datasets, and 1,002 CoT answers generated by 3 language models, consisting of 3,360 CoT steps in total.
Each step is first labeled for relevance with respect to the final answer, and then whether the step is an attribution step (introduces factual knowledge which can be attributed to a source), a logical step (introduces logical inference from previous steps) or both. For attribution steps, we collect labels for correctness to retrieved Wikipedia paragraphs given as evidence (with full support, partial support, contradiction, or no-support as labels). For logical steps, we label for logical correctness. Each label includes free-text justifications written by the annotators. An illustrative instance from the dataset is shown in Figures 3 and 4. We split the dataset into REVEAL-Eval, the main evaluation benchmark containing high inter-annotator-agreement labels, and REVEAL-Open, a smaller set of interesting borderline cases with open labels due to low interannotator agreement. In §5 we describe the dataset, and report fine-grained analyses of non-attributable steps in REVEAL-Eval (i.e., evidence that supports or contradicts them was not found) and of disagreement categories in REVEAL-Open.
REVEAL supports versatile evaluation settings, for example: (1) Attribution steps, along with their evidence, can serve as a high-quality Natural Language Inference (NLI, Dagan et al., 2005; Bowman et al., 2015) benchmark in a setting of factchecking LM outputs (Gao et al., 2023; Zhang et al., 2023); (2) CoT verifiers can be evaluated
at the level of individual steps, or (3) at the level of full CoT answers; (4) Each label in the data contains five free-text justifications (one per annotator), which can accommodate research around the generation of explanations and justifications, or be used to understand nuance in borderline cases.
As we focus on the evaluation of step-level validation in complex reasoning, in §6 we report the performance of multiple up-to-date verification baselines, leveraging NLI classifiers, GPT-3 and PaLM 2, showing much room for improvement in current state-of-the-art solutions. In particular, verifiers struggle at classifying whether a step conveys correct logical inference from previous steps.
In summary, this work includes the following contributions: (I) A protocol for step-by-step verification of reasoning chains (§2); (II) An annotation schema to reliably execute the protocol with human annotators (§3); (III) A new benchmark dataset for evaluating automatic reasoning chain verifiers (§4, §5); (IV) Detailed analyses of challenges in retrieving evidence to knowledge claims in reasoning and documentation of disagreements in the data (§5); (V) A study on the challenges for current verifiers (§6). These contributions advance the research on verification of reasoning chains and methods for correctly reasoning about complex questions.
2

heading: 2 Formalism for Verification of Reasoning Chains
text: In this section, we formalize the task of verifying reasoning chains for attribution and logical correctness and discuss its evaluation.
We consider a reasoning chain as a sequence of n steps r = s1, ..., sn, where si is a claim generated conditionally on the steps preceding it s1, ..., si−1. We focus on reasoning chains that are generated to answer a given question q and have a CoT format, that is, where the last step sn includes the final answer to q and every step is a standalone sentence.
Given vocabulary Z, a verifier V : Z|q| ×Z|r| → [0, 1] receives a question q and a reasoning chain r conditioned on q, and outputs a score for the correctness of r as an answer to q. “Correctness” can be defined in various factors, such as factuality, grammaticality, and coherence. We focus on two key factors—correctness with respect to world knowledge, i.e., that the answer is derived based on grounded facts, and logical correctness, where reasoning steps are inferred with logically correct inference. We consider the world knowledge that supports the claims in r as some evidence e, external to the reasoning chain. For our purposes, we separate the retrieval of e from the verification process. Therefore, we consider a verifier V : Z|q| × Z|r| × Z|e| → [0, 1] that also receives as input evidence e to ground the correctness of r.
Full chain vs. step-level verification. At highlevel, a verifier is a system which receives as input a question and a reasoning chain answer (in our case, a CoT answer), and outputs whether the entire reasoning chain is correct—this is the more prevalent approach (§7). A more fine-grained approach is to evaluate each reasoning step for correctness separately, where the reasoning chain is correct if every step is correct (Li et al., 2023). We adopt the step-level approach, as it provides the ability to evaluate full-chain verifiers, the ability to detect the exact point of failure in reasoning chains, and the ability to distinguish between different types of errors. Step-level detection can additionally help with detecting cases of “snowballing of hallucinations” from earlier steps (Zhang et al., 2023).
Correctness of an individual step. Fig. 2 details our methodology for defining the correctness of an individual step, which we expand on below.
(1) Step relevance. Each step is “relevant” or “irrelevant” to answer the question. Irrelevant
steps do not invalidate the chain’s correctness. (2) Step type. The correctness of a step can be defined in three ways: Whether it is entailed from approved world knowledge (“attribution step”), entailed from previous steps (“logical step”), or both (“both”). We define attribution steps in language adopted from fact-verification (Konstantinovskiy et al., 2018; Guo et al., 2022): If the step “contains knowledge that should be verified against an external source.”
(3) Step attribution to external source. An attribution step is correct if it is “fully attributable” to a given source, meaning “strictly according to the given source, all of the information in the claim is correct.” Steps are otherwise “contradictory” or “partially attributable” to the source, or “unsupported” by the source. This categorization mimics the Attribution to Identified Sources formalization (AIS, Rashkin et al., 2021). (4) Step logical correctness. Each logical step is “correct” if it can be logically inferred from the previous steps, otherwise it is “incorrect”. The correctness of logical steps that follow incorrect logical steps is undefined.
(3 & 4) Hybrid steps. When steps contain both world knowledge and logical inference, we assume that all external knowledge is fully attributable for the purpose of logical correctness, and that all logical inferences from previous steps are correct for the purpose of attribution correctness.

heading: 3 Annotation Schema
text: We introduce a labeling schema for reasoning chain verification, according to the formalization in §2.
3
The schema prioritizes annotation quality, ease of use and scalability. Such an implementation requires making some design choices, which we explain and justify below. The annotation interfaces and annotation questionnaire are available in §A.
We observe that the overall annotation process of a reasoning chain involves different types of information and requires various skills. For example, to determine if a reasoning step can be attributed to a given evidence paragraph, it is essential to read the paragraph in detail and judge its relation to the step without any additional context. In contrast, to validate if a reasoning step is logically correct, it is required to think critically whether some information is missing or incorrect in the overall reasoning process while leveraging the annotators’ background knowledge. The two modes of thinking (logic and attribution) can interfere with each other, where focusing on one at a time is less cognitively-demanding.
To accommodate the complexity of the verification task, we therefore split it into two annotation tasks: (1) A full-chain task that considers the logical transitions between steps, while assuming that any specified facts are factual; (2) a step-wise task that checks if a given step is attributable to or contradicts some given external knowledge.
Note that the two tasks are designed to be independent, such that different annotators can annotate logical correctness independently from attribution, and vice versa. This allows for parallelization of the annotation process, reduces cognitive load, and provides a more robust set of labels that is less dependent on specific annotators. For example, some annotators were assigned to only one of the tasks, according to feedback during pilot phases.
Justifications. In addition to the verification labels, in both tasks we ask annotators to provide a free-text justification for their choices. This information helps to monitor the annotation process, provide additional valuable insight on the reasoning behind ratings—as it is possible to make the same decisions in these tasks for different reasons—and additionally serves as a method of interacting with the annotators to provide and receive feedback.

heading: 3.1 Task 1: Relevance, Type, and Logic
text: Given a complex reasoning question and a CoT answer, the first task involves annotating the chain step-by-step, labeling the following information for each step: relevance, step type and logical correct-
ness, according to the classes described in §2. The logical correctness of a step in the chain is evaluated with respect to the steps preceding it and the question. Importantly, this is done without considering the correctness of attribution steps that introduce external knowledge—i.e., all external knowledge is assumed to be correct in this task, as the focus is on the logical derivations. Consider for example the following reasoning chain:
Question: Which population is bigger, California or Texas?
Answer: Texas population is 39.53m.(1) California population is 39.24m.(2) Thus, Texas has a bigger population than California.(3) So the answer is Texas.(4)
When evaluating logical correctness, step 3 would be considered correct, even though the cited number for the population of Texas is incorrect.

heading: 3.2 Task 2: Relevance and Attribution
text: The second task considers the correctness of facts stated in the reasoning chain and therefore is applied only to steps labeled as attribution steps (in the first task). The task involves verifying every step against a set of evidence paragraphs, where the annotator should indicate for a given evidence paragraph if the step is fully attributable (supported), partially attributable, contradicts or is unsupported. The annotation of a step ends when either a fullysupporting or contradicting evidence is found, or the maximum number of paragraphs is labeled. In this work, we limit this number to three, as evidence support dramatically decreases with additional evidence (Fig. 5 right) and to reduce cost.
Relevance is annotated in this task as well, to account for more specific phrasing under the lens of attribution: An attribution step is marked relevant here if it introduces information that is helpful to answer the question. When writing justifications to their ratings, annotators were encouraged to quote specific parts of the CoT or evidence as needed.

heading: 4 Data Collection Process
text: In this section, we describe the full process for constructing REVEAL, using our annotation schema. The process is divided into two phases: Collecting and generating data for annotation (steps 1–2) and the annotation process (step 3).
Step 1: Reasoning Chain Generation. We use four open-domain complex-reasoning QA datasets to elicit model-generated reasoning chains:
4
1. STRATEGYQA (Geva et al., 2021): Yes/no questions that require a diverse set of reasoning skills and applying implicit knowledge.
2. MUSIQUE (Trivedi et al., 2022): Multi-hop reasoning questions with free-text entity answers, generated based on paragraphs from Wikipedia. For our dataset, we consider the subset of 2-hop questions, as we found qualitatively that the 3- hop and 4-hop questions are often unnatural and hard to understand.
3. SPORTS UNDERSTANDING (Srivastava et al., 2023): Yes/no questions that require reasoning about knowledge on sports players, leagues, and sports maneuvers.
4. FERMI (Kalyan et al., 2021): Estimation questions with numerical answers that require both knowledge and reasoning to answer. The questions are designed to have no clear gold answers—they are exercises of common-sense reasoning and numerical estimation, e.g., “How much water does a school use in a week?”
Examples from each source are provided in Tab. 5 (§A). This diverse set of tasks provides a variety of question types, knowledge and reasoning requirements, and answer formats (binary, numerical and free-text entities). The questions were sampled randomly from the evaluation sets of each dataset and evenly split across the four datasets.
We use three different LMs to generate CoT answers: Flan-PaLM-540B (Chung et al., 2022), GPT-3 (text-davinci-003, Brown et al., 2020) and Flan-UL2-20B (Tay et al., 2023). We prioritized a variety in the models, with two large-size highperforming models with different pretraining data, and one smaller-size, weaker model, in order to gather a variety of CoT answers. CoT prompt demonstrations were written according to standard practices (Wei et al., 2023), with in-domain examples for each dataset taken from the dataset’s training set, and designed to be simple and informative. A subset of the questions from each dataset (onefourth) were answered by all three models with three separate CoT answers, to serve as a base for analyses that require multiple answers per question, while the rest were answered by each model once in equal proportions (one-fourth each). Our goal in this methodology was to maximize the flexibility of the dataset and potential for analyses and evaluations, for as many LMs and verifiers as possible. Specifically, we aimed to collect a sufficient variety of answers, both correct and incorrect, from a
practical and realistic distribution.
Step 2: Evidence Retrieval. For attribution verification, we use Wikipedia as an external knowledge source and retrieve three paragraphs for each attribution step. StrategyQA, MuSiQue, and Sports Understanding are well-supported by Wikipedia, while Fermi is explicitly designed to be difficult to support, giving a variety of cases. To promote the retrieval of supporting paragraphs, we mix dense retrieval and lexical-based retrieval, fetching two paragraphs with GTR (Ni et al., 2021) and one with BM25 (Zaragoza et al., 2004). In addition, retrieval is done using decontextualized versions of the CoT steps, generated by the decontextualization model of Choi et al. (2021) which replaces co-referenced pronouns with explicit entity mentions.
Step 3: Annotation. We use a pool of 13 Englishspeaking annotators, and collect 5 annotations per question and its corresponding CoT answer, for each of our two tasks (see §3). A small portion of the labels (approx. 5%) were annotated by the authors of this work (3 annotations per sub-task), to fill any gaps in examples that were not fully annotated for technical reasons.
Justifications and Quality Validation. Each annotation in REVEAL is accompanied by a free-text justification written by its annotator. Those justifications serve two goals: they allow us to easily monitor the annotation process by leaning into the annotators thought process and provide them with meaningful feedback (Nangia et al., 2021), and serve as a valuable resource on their own which we leave to be explored in future work. We maintained a high level of quality for these justifications, such that they do not collapse to templated answers or provide insufficient information. Examples of these justifications are given in Fig. 4.
5
Unsupported 38.6%
Fully Attributable
43.8%
Partially Attributable 6.1%
Contradictory 11.5%
Attribution labels for CoT steps
Unsupported 65.4%
Fully Attributable 24.7%
Partially Attributable 3.4%
Contradictory 6.5%
Attribution labels for step-evidence pairs
Not found 38.6%
1 53.2% 2
5.4%
3 2.8%
Required number of evidence passages to find support or contradiction
Figure 5: Statistics for the attribution task in REVEAL-Eval (§5). Left: Label distributions for each step or stepevidence pair. Right: Statistics for the number of retrieved evidence passages for each attribution step until a supporting or contradicting evidence was found (up to 3).
In addition, we performed three pilot rounds for annotator selection, annotator training, and task improvement, all to maintain a high quality bar for the collected dataset (the pilot annotations were discarded from the final dataset). Finally, we split the dataset into two subsets, REVEAL-Eval and REVEAL-Open, where the latter contains low-confidence labels that received lower inter-annotator agreement (see §5).

heading: 5 REVEAL
text: We ran the data collection protocol described in the previous section to obtain our dataset, REVEAL. In terms of inter-annotator agreement, we report a Krippendorf’s α of 0.49 for attribution steps and 0.46 for logical steps.
We split REVEAL into two subsets, as we observe that some reasoning chains are very challenging to annotate because of ambiguity or other factors (analyzed in §5.2). Any CoT answer with at least one step that has low inter-annotator agreement is treated as an indecisive case, which applies to 18% of the CoT answers. A step is considered to have “low inter-annotator agreement” if less than three annotators agree on any single label for attribution or logical correctness (for example, an attribution step labeled as “partially attributable” by two annotators, “fully attributable” by one annotator, and “contradictory” by two annotators). We release these examples separately in a data subset called REVEAL-Open, as they are valuable instances of difficult or borderline cases, and the rest of the examples in a subset called REVEAL-Eval. Since the annotations in REVEAL-Open are indecisive, in §6 we evaluate only on REVEAL-Eval.
Tab. 1 details statistics on the two data splits, showing that REVEAL-Eval has several hundreds to thousands of examples for each setting to support reliable evaluation. Instances are approximately distributed uniformly across source datasets and
LM generators. We release all five annotations for each label in the dataset, including anonymized annotator identifiers for each label, to support future annotation methodology research and reproducibility (Sandri et al., 2023).
Considering the label distributions, 98.6% of steps are relevant to answer the question, and 87.5% of logic steps are logically correct. Fig. 5 (left) shows label distributions for attribution steps and step–evidence pairs (note that partial attribution is considered as “not enough info” in standard NLI formalization). While 43.8% of attribution steps were fully attributable, this is still relatively low, which may point at attribution as a main obstacle in reasoning. However, we do note that unsupported steps are not necessarily incorrect, as lack of support may stem from wrong claims, but also from issues in retrieving relevant supporting evidence. We analyze this further in §5.1.
Fig. 5 (right) shows the distribution for the required number of evidence paragraphs for finding support or contradiction for each step: In the majority of cases where support was found, it was already present in the first retrieved paragraph.
Fig. 6 shows distributions for full chain-level attribution and correctness for each subset of the data. In terms of specific types of errors in the CoTs: 77.3% of CoTs in REVEAL-Eval have a step which is not fully attributable, while 18.5% have a
6
step which is logically incorrect.

heading: 5.1 Analysis of Unsupported Claims
text: A large fraction of the labeled steps was found unsupported by the retrieved evidence (38.6%, Fig. 5). To understand this, we conduct a qualitative analysis of 40 unsupported steps (10 from each source dataset, randomly sampled). For each step, we first verify whether the step is factually correct. For correct steps, we analyzed whether the issue was due to irrelevant retrieved evidence, or due to other reasons. See §A for full details.
Of the 40 unsupported steps, 19 were indeed not factual. Of the remaining 21: In 13 steps the retrieved evidence was relevant, but additional reasoning and world knowledge was required to verify them (e.g., which teams are part of the spanish “LaLiga”, or that “ACM” stands for “Association for Computing Machinery”). In 6 steps the retrieved evidence was irrelevant. In 1 step there were two claims within the given step, but only one was supported by the evidence, and in 1 step the evidence was irrelevant due to insufficient decontextualization (“Brown” ref. “Jaylen Brown”).
From this analysis, we estimate that roughly half of the unsupported cases stem from using imperfect retrieval. Additional challenges such as imperfect decontextualization and requiring to reason over multiple evidence passages can also slightly contribute to failing to support factually correct claims.

heading: 5.2 Analysis of REVEAL-Open
text: REVEAL-Open consists of questions paired with answers in which at least one of the steps has low inter-annotator agreement, defined as a case where no more than two (out of five) annotators agree on a single label for that step. Tab. 2 summarizes our analysis of the low-agreement cases. We inspected each of these steps (including the underlying data, as well as annotators’ individual labels and written feedback) and report a qualitative description of the complications that may have contributed to the low agreement. On the basis of these qualitative notes, we devise 13 broader categories and manually assign one or more categories to each step.
The most frequent complication categories for attribution steps are world knowledge or general inference, rating category definition/criterion, and specialized knowledge and insufficient nuance/hedging tied. For logical steps, the most frequent categories are calculation or unit issue, inconsistent/unclear reference or standard, and invalid
inference in a previous step. We conjecture that improvements to the annotation instructions could help to minimize some of the categories, despite our efforts to make revisions during the pilot phases to achieve this effect. In particular, additional guidance can assist decisions on when and how to apply world knowledge, and clarify distinction between attribution labels. Recruiting annotators with appropriate domain knowledge for the main subjects of a given dataset (e.g., sports or sciences) could also reduce the number of disagreements related to both the content and the standards of comparison or equivalence.

heading: 6 Experiments
text: We use REVEAL to evaluate existing methods for CoT verification, on step-level verification (§6.1) and CoT-level verification (§6.2).
Verifiers. We use Flan-UL2 (20B), Flan-PaLM (540B), PaLM-2-L (Anil et al., 2023) and GPT-3 (text-davinci-003) for LM baselines in few-shot prompting settings (prompt templates detailed in §B). In addition, we use two specialized baselines: a T5-based model with 11B parameters trained on a mixture of NLI datasets (Honovich et al., 2022), and FacTool—a GPT-3-based fact-checking pipeline (Chern et al., 2023).
For the prompting-based baselines, we use the label generated by the LM as the predicted class (which always matched to one of the task labels in our experiments). The NLI classifier receives as input a premise and hypothesis and predicts a score between 0 and 1 that indicates if the hypothesis is entailed by the premise or not. FacTool returns a binary factuality classification label, and we use it
7
only for the attribution task.
Evaluation. As few-shot evaluation is noisy (Perez et al., 2021; Jacovi et al., 2023b), we average the predictions of the LM baselines over 5 different 8-shot prompts, sampled in random order from 13 demonstrations for each sub-task, which we wrote for this purpose (the number of demonstrations is trimmed if it exceeds a given model’s context length). All sets of demonstrations were class-balanced and balanced across source datasets and labels. Prompt templates are available in §B. For FacTool, we insert the appropriate evidence from the dataset, instead of allowing FacTool to retrieve evidence separately, for compatibility with the attribution labels. We release the prompt demonstrations alongside the dataset.

heading: 6.1 Step-level Verification
text: Given the unbalanced class distribution, we report macro-F1 performance in Tab. 3 (class F1 results are in §B). Results for the step relevance task (classifying if a step is relevant for answering the question) are omitted, as all the models collapsed to the majority baseline, leading to an F1 of near-0.
Step attribution. In this task, the model receives as input a decontextualized version of a CoT step and a Wikipedia evidence paragraph. There are two variants: 2-class—classifying whether the step is entailed or not by the evidence; 3-class— classifying between full entailment, contradiction, or not enough info. The best-performing model in 2-class was the T5-based classifier, which is much smaller than the other models although fine-tuned on large NLI data. 3-class performance follows scaling laws, which shows that detecting contradictions is a difficult problem.
Step logic. This task is to classify if a given step is logically entailed by the previous steps and the

heading: Baseline Single decision Pipeline
text: question. All prompting-based baselines were biased towards a “logically correct” classification, achieving high F1 on the “correct” class (>85%) and low F1 on the “incorrect” class (~33-47%). The T5 NLI baseline shows a bias towards the “incorrect” class, but performs significantly worse overall. This indicates that NLI fine-tuning with simple factual claims does not generalize to out-ofdomain complex logical structures.
Step type. The task here is to classify whether a CoT step is an attribution step, logic step, or both, given the input question and its full answer. All models struggled with this task, achieving macroF1 below 65%.

heading: 6.2 CoT-level Verification
text: This task involves verifying whether the CoT correctly justifies the answer or not—without necessarily specifying which step contains which type of error. Results are in Tab. 4. We report on two implementations of each LM baseline: (1) Pipeline implementations are decisions by combining the LM’s decision on each step separately according to the step-level task predictions in §6.1. (2) Singledecision implementations are by prompting the LM to classify (in few-shot) whether a CoT is a correct answer to a question or not, given the question, CoT answer, and evidence paragraphs. The class balance for this task is 80:20 (incorrect CoTs being the majority class).
When observing macro-F1 performance, the pipeline variants significantly outperform singledecision in all cases. The gap appears to stem from a significantly higher F1 on the incorrect class (35- 55% vs. 80-87%), while the correct class receives low F1 for both variants (30-56%). Overall, all baselines struggled with the high-level task of CoT verification, showing significant room for improve-
8
ment on REVEAL and on CoT verification.

heading: 7 Related Work
text: While CoT prompting (Wei et al., 2023) has originally emerged as a simple prompting technique to increase performance of the “final” answer to the question, it has subsequently been interpreted as a part of the answer to be verified for correctness itself (Golovneva et al., 2023). As part of this narrative, various methods have been proposed to verify reasoning chains, either for LM evaluation (Welleck et al., 2022), improvement (Chen et al., 2023) or for training (Lightman et al., 2023). To our knowledge, no current work provides a fully labeled benchmark of reasoning chain correctness in information-seeking tasks.
Recent work on verification of reasoning chains primarily focuses on methods of verification and metrics: Golovneva et al. (2023) propose a suite of metrics for CoT quality, including knowledge attribution and logical correctness (both formalized as instances of NLI), grammaticality, informativeness, and so on. They provide a set of examples in math QA, closed-domain QA, and NLI, annotated in various quality metrics. Prasad et al. (2023) propose evaluating for informativeness and logical correctness in math QA and closed-domain QA settings. Ott et al. (2023) collect a variety of existing gold-reference human-written CoTs from existing datasets, and provide a web-interface annotation tool for annotating CoTs for errors in logical inference, factual knowledge, verbosity and reading comprehension. Lightman et al. (2023) provide a large dataset of math QA with step-level annotations for mathematical correctness of CoTs by one LM, primarily for training.

heading: 8 Conclusion
text: We design a methodology for human verification of reasoning chains, and employ it to annotate a dataset of CoT-format reasoning chains generated by three LMs in open-domain complex-reasoning QA tasks. The dataset, REVEAL, can be used to benchmark automatic verifiers of LM reasoning. Our work advances the research towards LMs that can provide correct and attributable reasoning behind their decisions. We find that CoTs generated by LMs are often not fully correct, and that automatic verifiers struggle to verify them appropriately. In particular, between attribution and logic types of correctness, CoT-generating LMs struggle more
with attribution, but in contrast, CoT verifiers struggle more with verifying logical correctness.

heading: 9 Limitations
text: Evidence retrieval. Our work focuses on evaluation of verifiers that receive evidence as input—i.e., in the task of attribution to a given source, rather than the task of fact-checking where evidence retrieval is explicitly part of the task. As such, we can only evaluate verifiers that also operate on specific given evidence, rather than fact-chekers that perform evidence retrieval themselves. Additionally, for some of the knowledge claims in the dataset labeled as “unsupported”, it is possible that there exists evidence that will support or contradict them, which our retriever did not surface. We note, however, that the goal of the dataset is to simply collect a sufficiently wide variety of cases on which to evaluate verifiers, rather than to evaluate the CoT itself using an “ideal” retriever, and the labels in the dataset are well-defined for the specific evidence passages used.
Fermi and Wikipedia evidence. We retrieve evidence against Wikipedia, in the interest of having clean and reliable evidence, and due to the fact that StrategyQA, MuSiQue and Sports Understanding are sufficiently addressed by Wikipedia knowledge. However, Fermi is a dataset which is explicitly designed to require knowledge that is difficult to attribute or measure, and require difficult inference for both knowledge and logic claims. As such, Fermi is by design not well-addressed by Wikipedia retrieval. We chose Fermi as one of our four datasets in order to have sufficient variety in the data for claims that are very difficult to support.
Reasoning chains in formats other than Chainof-Thought. Our dataset uses CoT as a writing style for reasoning chains, due to its popularity and sentence structure that allows convenient step separation. It is of course possible to reason about complex questions in other ways, such as using sentences that combine multiple facts together, or claims that combine knowledge and logic together, and so on. Such examples can be considered a more difficult case for reasoning verification, as they will potentially require an additional solution for extracting atomic claims from the reasoning chain. CoT-format chains, such as in our dataset, do not have this requirement.
9

heading: Acknowledgments
text: We thank Eran Ofek for his assistance in providing expert annotations, and to Idan Szpektor and Avi Caciularu for providing helpful feedback throughout the project. We are grateful to the annotators who provided labeling judgments and useful feedback throughout the annotation process.

heading: A REVEAL: Extended Details
text: Here we provide additional details for the collected dataset and the collection process.

heading: A.1 Source Data Collection
text: Tab. 5 shows examples from each of the four source datasets, alongside example answers and their correctness. These examples are shown here for illustration, and were used as part of the pilot annotation phases, so they are not examples from the final REVEAL dataset.
To generate CoT answers, we constructed a separate prompt for each dataset, using examples from its training set with verified (correct) CoTs. StrategyQA, MuSiQue and Fermi provide gold-reference solutions (in non-CoT format) that we used to write the prompt CoT demonstrations. For Sports Understanding, we wrote the CoT demonstrations given the gold answer from the dataset. The CoT prompts used 6 demonstrations (question and CoT answer pairs) each.
CoT answers were split into sentences using NLTK’s sentence tokenizer (Bird et al., 2009), and each sentence was considered one reasoning step. For purposes of retrieval they were decontextualized with the decontextualization model by Choi et al. (2021). The evidence paragraphs were retrieved from a 2021 image of Wikipedia. The evidence paragraphs in the dataset have on average 103 words, and the longest paragraph has 582 words.

heading: A.2 Annotation Questionnaire
text: We split the annotation into two tasks, one focused on the logic annotation (including relevance, step type and logical correctness ratings), and the other focused on the attribution annotations (including relevance and step-evidence attribution). The annotation interfaces are shown in Fig. 8.
Task 1 is implemented and phrased as follows:
1. Relevance: Is step i relevant to answering the question? (A) Yes, it is relevant. The information in this step might be helpful to answer the question. (B) No, it’s not relevant. The information in this step is not helpful to answer the question.
2. Type: Does step i bring new information or describes a logical step? (A) Attribution step. The step brings new information to help answer the question. (B) Logic step. The step
Reveal Dataset - Source Task Split
only makes logical inferences based on the question or previous steps. (C) Both. The step introduces new information and makes a logical inference.
3. Logic: Considering only the logical inference done in step i, is it consistent with the previous steps? (A) Yes, correct inference. The inferences that the step makes based on the question or previous steps is correct. (B) No, incorrect inference. The inferences that the step makes based on the question or previous steps is incorrect.
4. Justification: Provide a justification for your ratings.
Task 2 is implemented and phrased as follows:
1. Relevance: Should claim i be attributed? (A) Yes, it’s necessary to attribute the claim. The claim has information that needs to be verified. (B) No, it’s unnecessary to attribute the claim. The claim doesn’t have any information that requires verification.
2. Attribution: To what extent can the information in claim i be verified by evidence j? (A) Fully. Strictly according to the cited evidence, the claim is correct. (B) Partially. All important information in the claim is supported (no contradiction), but some information is
15

heading: Source Data Question Answer Error
text: unsupported. (C) Contradictory. Some information in the claim is contradicted by the cited evidence. (D) Unsupported. Some important information (or more) in the claim is not supported by the cited evidence (but no contradiction).
3. Justification: Provide a justification for your attribution rating. What parts of the claim are and are not supported by the evidence? Feel free to paste parts of the response and the source document as evidence.

heading: A.3 Additional Statistics
text: Distributions across source tasks and models are shown in Fig. 7. To allow for a method of comparing multiple CoT answers to the same question, there are 161 questions which were answered by all three models in the dataset. Of them, 96 have all three CoT answers in REVEAL-Eval, while the remainder 65 questions have one or more CoT answer in REVEAL-Open.
Each step’s annotation took between 300 to 600 seconds, with some answers having 5 or more steps,
and with 5 annotations per step—leading to an expensive annotation process.

heading: A.4 Analyses
text: Tab. 10 contains the full analysis for the properties of unsupported attribution steps.

heading: B Experiments and Analyses

heading: B.1 Experiment Details
text: In the few-shot evaluation settings, prompts were constructed by randomly selecting 8 demonstrations from class-balanced sets of 13 demonstrations. Prompts were trimmed to the biggest number of demonstrations which, combined with the query, were under the context length for the given LM.
The demonstrations were taken from the training sets of the source datasets, or from the examples used for the pilot annotation phase (which were discarded from the final REVEAL dataset).
The dataset used to train the T5-based classification baseline is a compilation of multiple datasets: MNLI (Williams et al., 2018), SNLI (Bowman et al., 2015), FEVER (Thorne et al., 2018), Sci-
16
Tail (Khot et al., 2018) and VitaminC (Schuster et al., 2021).

heading: B.2 Additional Results
text: Per-class F1 metrics in all settings are shown in Tables 6 to 8.

heading: B.3 Prompt Templates
text: We describe here the template structures we used for our few-shot verification prompts. We note that we made an effort to test multiple templates and settings, to make sure that an adequate effort was allocated to implementing baselines that are as strong as possible.

heading: Attribution Task
text: Premise: [evidence paragraph]
Hypothesis: [CoT step]
Output: {Entailment, Not Entailment}

heading: Logic Task
text: Premise: [previous CoT steps]
Hypothesis: [CoT step]
Output: {Correct, Incorrect}

heading: Step Type Task
text: Question: [question]
Answer: [full CoT]
Step: [CoT step]
Step type: {Attribution, Logic, Both}

heading: Relevance Task
text: Question: [question]
Answer: [full CoT]
Step: [CoT step]
Is this step relevant to the answer? {Yes, No}
CoT Correctness task
Baseline F1 (Logic Task)
Baseline F1 (Full Correctness)
Evidence i: [evidence paragraph]
Question: [question]
Answer: [full CoT]
The answer is: {Correct, Incorrect} All evidences are added to the demonstration for i in the number of evidences (i.e., the number of attribution steps in the CoT).
17

heading: Complexity
text: Category
% steps in category Description
18

heading: Evidence

heading: In this task you will evaluate the quality of the system reasoning steps given a question. Refer to the full instructions with rating examples.
text: 19

heading: Dataset Step Factually
text: 20

heading: Future Work
text: We release all five annotations for each label in the dataset, including anonymized annotator identifiers for each label, to support future annotation methodology research and reproducibility (Sandri et al. , 2023). Considering the label distributions, 98. 6% of steps are relevant to answer the question, and 87. 5% of logic steps are logically correct. Fig. 5 (left) shows label distributions for attribution steps and step–evidence pairs (note that partial attribution is considered as “not enough info” in standard NLI formalization). While 43. 8% of attribution steps were fully attributable, this is still relatively low, which may point at attribution as a main obstacle in reasoning. However, we do note that unsupported steps are not necessarily incorrect, as lack of support may stem from wrong claims, but also from issues in retrieving relevant supporting evidence. We analyze this further in §5. 1. Fig. 5 (right) shows the distribution for the required number of evidence paragraphs for finding support or contradiction for each step: In the majority of cases where support was found, it was already present in the first retrieved paragraph. Fig. 6 shows distributions for full chain-level attribution and correctness for each subset of the data. In terms of specific types of errors in the CoTs: 77. 3% of CoTs in REVEAL-Eval have a step which is not fully attributable, while 18. 5% have a
6
step which is logically incorrect