ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning
Authors: Olga Golovneva
Moya Peng Chen
Spencer Poff
Martin Corredor
Luke Zettlemoyer
Maryam Fazel-Zarandi
Asli Celikyilmaz
Venue: ICLR 2023 notable top 25% (2023)
Keywords: step-by-step reasoning
evaluation

Abstract: Large language models show improved downstream task performance when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness (independent of the final answer) is difficult without reliable methods for automatic evaluation. We simply do not know how often the stated reasoning steps actually support the final end task predictions. In this work, we present ROSCOE, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics. To evaluate ROSCOE against baseline metrics, we design a typology of reasoning errors and collect synthetic and human evaluation scores on commonly used reasoning datasets. In contrast with existing metrics, ROSCOE can measure semantic consistency, logicality, informativeness, fluency, and factuality — among other traits — by leveraging properties of step-by-step rationales. We empirically verify the strength of our metrics on five human annotated and six programmatically perturbed diagnostics datasets - covering a diverse set of tasks that require reasoning skills and show that ROSCOE can consistently outperform baseline metrics.

Published as a conference paper ICLR 2023
ROSCOE: A SUITE OF METRICS FOR SCORING STEP-BY- STEP REASONING
Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, Asli Celikyilmaz Meta AI Research
{ (olggol, mpchen, spoff, mcorredor, 1sz, maryamfazel, aslic) Emeta. com
ABSTRACT
Large language models show improved downstream task performance when prompted to generate step-by-step reasoning to justify their final answers (Nye et al., 2021; Wei et al., 2022). These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness (independent of the final answer) is difficult without reliable methods for automatic evaluation. We simply do know how often the stated reasoning steps actually support the final end task predictions. In this work, we present ROSCOE, suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics. To evaluate ROSCOE against baseline metrics, we design a typology of reasoning errors and collect synthetic and human evaluation scores on commonly used reasoning datasets. In contrast with existing metrics, ROSCCE can measure semantic consistency, logicality, informativeness, fluency, and factuality - among other traits by leveraging properties of step-by-step rationales. We empirically verify the strength of our metrics on five human annotated and six programmatically perturbed diagnostics datasets- covering diverse set of tasks that require reasoning skills and show that ROSCOE can consistently outperform baseline metrics."
INTRODUCTION
Scaling language models has improved state-of-the-art performance nearly every NLP benchmark (Brown al., 2020), with large language models (LLMs) performing impressively as few-shot learners (Brown et al., 2020). Despite these achievements, even the largest of these models still struggle with tasks including math word problems (Hendrycks et al., 2021), symbolic manipulation (Rytting & Wingate, 202 I), and commonsense reasoning (West et al., 2022). Recent work has shown that prompting (Wei et al., 2022; Wang et al., 2022) or fine-tuning (Lampinen et al., 2022) LLMS to generate step-by-step rationales can lead to improvements on reasoning tasks. Some of these include small-scale analysis of specific types within steeep rationales (Lewkowycz et al., 2022; Chowdhery et al., 2022), as shown in Table 1. However, existing works primarily focus on end-task performance. Although text generation evaluation metrics sometimes offer fine-grained quality evaluations (e.g., adequacy, fluency) against human scores (Opitz & Frank, 2021; Leiter et al., 2022), these metries generally treat the output as whole, and many of these generative metrics operate on tasks such as summarization or machine-translaie rather than reasoning.
'Code can be found at roscoc. Annotated datascts be downloaded from http:///www....oo..//////////a/a projects/rcscce/ http://ccccceaaotaaaiiioooo amnotaiinns...pppp....

---

Published as a conference paper at ICLR 2023
In this paper, we present ROSCOE, suite of inter- pretable and fine- ine-grained step-by-step generation eval- uation metries to address the above gaps. Rather than providing one score that only evaluates the generated text on the overall, ROSCOE encapsulates fine-grained metrics under four perspectives: (1) semantic alignmenl defines to what extend the generated reasoning is coher- ent, and grounded with the source context; (2) logical inference evaluates if the generated reasoning steps are consistent within itself and checks for logical fallacies; (3) semantic similarity quantifies the degree of similar- ity between the generated reasoning and the context or between intermediate steps to capture hallucinations or repetitions; and (4) language coherence evaluates if the whole chain flows naturally.
Table I: Step-by-step reasoning for logical inference task generated from 2-shot GPT-3 dext-davinci-0022 (Brown et al., 2020). Error types are highlighted: logical or factual issues, partially correct, bomewhat-relevant Full 2 s-tt exemplars are in Appendix B.
Shelley is from Virginia, but is visiting that city with that famous market where they throw the fish. Going home next Thesday!
Context:
Question: Is it likely that Shelly will be the Pacific Occan this weekend?
GPT-3 Step-by-ster
reasoning:
No, it is not likely that Shelley will be near the Pacific
Oczan this weekend.
Shelley from Virginia, which is on the East Coast of the United States.
The Pacific Oceun is the West Coust.
Shelley is visiting a city with a famous market, which
could be anywhere in the world.
To evaluate ROSCOE against existing metrics, we devise a taxonomy of reasoning errors for multi-step genera- tions and use it to create synthetic data and collect human evaluations on commonly used reasoning datasets. Our taxonomy and annotated datasets help us gain deeper insights into the causes reasoning inconsistencies and weaknesses of LLMs. We evaluate ROSCOE with 18 fine-gre -ee-ggrined metrics under the above four perspectives. ROSCOE demonstrates performance gains against baseline evaluation metrics on all tasks that require reasoning over context. Additional sensitivity analysis shows that ROSCOE is more robust when dealing with tasks that require logical and arithmetic reasoning.
Tuesday is her travel day, she will likely not be
traveling this weekend.
Contributions. (1) We propose a new taxonomy for reasoning errors, and use it for collecting human annotations and creating synthetic datasets. (2) Using our taxonomy, we propose new suite of metrics that focus on sequence and step level analysis of step-by-step reasoning. (3) We present extensive comparative analysis on 11 datasets of varied complex reasoning problems demonstrating the strengths of each metrie, especially in terms of interpretability relative to baselines, and considerations for use.
2 RELATED WORK
Evaluating Explanations. Free-form natural Language (NL) explanations of model decisions should enable accurate representation of the reasoning process and degree of plausibility (Danilevsky et al., 2020; Jacovi & Goldberg, 2021; Jacovi et al., 2021). A qualitative assessment of NL explanations with correctness labels collected from human judges was presented in (Camburu al., 2018). Recent work has also investigated automatic metrics for natural language generation (NLG) evaluation including word overlap or embedding based similarly with human written explanations (Clinciu et al., 2021). Though fast and coft-effective, automatic metrics for NLG are not equipped to measure the logical inconsistencies or information gain with thinking steps (Reiter, 2019; Celikyilmaz et al., 2020). Explanations have also been evaluated by collecting datasets, and running correlation analysis to investigate the degree to which an automatic metric correlates with human judgements of clarity, relevance and informativeness (Leiter et al., 2022; Welleck et al., Althoug reliable, human evaluation is an expensive, domain specific, and time-consuming process. In comparison, ROSCOE provides generic automatic evaluation procedures that are domain and task specific.
Automatic Metrics. Many NLG evaluation metrics exist in the literature including ones based on: n-gram match (Lin, 2004), regression (Sellam et al., 2020), embedding proximity (Zhang et al., 2020), paraphrasing (Thompson & Post, 2020), generation an evaluator (Yuan al., 2021); information alignment (Deng et al., 2021): I); among others. Although these metrics are easy to use, they evaluate the alignment of two texts as a whole and are not designed to assess individual reasoning steps. The closest metrics to ours are CTC (Deng
2

---

Published as a conference paper ICLR 2023
Table 2: Taxonomy of Step-by-Step Reasoning Errors. Full list of the error types with examples illustrated in Table 10.
Error Type
Definition
Grammar
Faulty, unconventional, controversial grammar usage
Factuality
Information about an object (i.e. quantity. churacteristics) or a named entity doesn't match with the input context.
Hallucination
Information not provided
statement and is irrelevant wrong
Redundancy
Explanation contains redundant information. which even though might be factual. is not required answer the question Step paraphruses information already mentioned in previous reasoming steps
Repetition
Missing step
The content the generated reasoning incomplete and lacks required information produce the correct answer.
Coherency
Steps contradict each other or do not follow cohesive story Model lacks relations that should be known from general world (e.g.. "all ducks are birds")
Commonsense
Arithmetic
Ertor in math calculations
et al., 2021) and BARTScore (Yuan et al., 2021), as both introduce a set of interpretable metrics to evaluate the similarity between two texts. However, ROSCOE is unique in providing fine-grained interpretations of reasoning steps, determining contradictions, and identifying ordering issues in the reasoning narrative.
Self-Consistency with LLMs. Recent work on improving LLMs performance on complex reasoning tasks uses an ensemble strategy called self-consistency (Wang et al., 2022). This method samples a diverse set of reasoning paths from a language model via reasoning traces prompting and returns the most consistent final answer in the set. Other work evaluates the diversity of reasoning path (Li et al., 2022), or the consistency of an inference step (Creswell et al., 2022) or finetune LLMs (Zelikman et al., 2022) to improve on difficult NLP tasks. In contrast to these works, we present suit of metries that focus on determining the type of the error (e.g., commonsense or logical inconsistency) in a reasoning path, if one exists.
3 REASONING ERROR TAXONOMY AND DATASETS CONSTRUCTION
Problem Formulation. Our goal is to score step-by-step rationales generated by a language model. We assume that the model is given source context {s1, ... ST) of T"sentences indicating a problem statement followed by question and is prompted to generate step-by-step reasoning (Nye et al., 2021). We refer this hypothesis h {h1,-.. fyy of N-steps, including final answer as the last step. We do not assume availability of gold step-by-ster reasoning references r {ri,-.. , .xk of K-steps.
Taxonomy. We propose new taxonomy of generic reasoning errors for language problem solving. We first conduct manual preliminary analysis on different types of LLMs reasoning errors using five Human judged datasets described below. Based on our analysis, we identified nine error types centered on the overall reasoning chain (i.e., the quality of the step-by-step thinking, including consistency with the context and commonsense reasoning). Our taxonomy also includes fine-graiie reee-graind errors marking inconsistency of a reasoning step with the previous steps, whether each step contributes to the final decision, and overall logical inference or fluency issues. The definition error types is in Table 2, and Table 10 provides examples.
Datasets and Annotations. To evaluate ROSCCE, we select datasets covering diverse set of tasks that require reasoning skills (e.g., logical, arithmetic, and commonsense reasoning tasks). We separate these datasets into two: (1) Diagnostics datasets that contain gold standard step-wise reasoning chains, where synthetically perturb some of the reasoning steps to introduce different generation errors (e.g., missing step, mathematical error, etc.); (2) Human judged datasets with model generated step-by-step reasoning outputs where the reasoning error evaluations are solicited from expert judges. We investigate these in $5.
4 REASONING SCORER: ROSCOE
We present our fine-grained metrics under four perspectives: semantic alignment, semantic similarity, logical inference and language coherence. Each metric is bounded within [0, 1], where indicates the perfect score
3

---

Published as a conference paper at ICLR 2023
and 0 corresponds to failure. A metric is reference-free or unsupervised when it uses the source and hypothesis (h- + s), while reference-based or supervised when evaluated between hypothesis and reference (h- - r).
4.1 SEMANTIC ALIGNMENT METRICS (ROSCOE-SA)
At the core of the ROSCOE semantic alignment metrics is the reasoning alignment vector from the N-step hypothesis h to the source & of length T: r-align(h (h- + 8) = {(11, 022 ,ON): }. where each alignment value a; = r-align(h; + 8) [] + max (cos(hi; s,)1 E [0, 1] the normalized cosine similarity between hypothesis step and most similar sentence in a context, and explicitly measures the grounding of the step-wise reasoning with respect to the source text (illustrated in App. D, Fig. 3). We estimate the alignment vector r-align(h (h + x) by matching source text and the reasoning chains on the embeddings of tokens and individual reasoning steps. A similar information aligmment score is introduced in CTC (Deng et al., 2021) to measure the confidence that the information of the i-th source document token 8j is grounded by hypothesis token h4. Our reasoning alignment is different in that we measure if a hypothesized reasoning step h; supports the
source context s. Our proposed metrics are summarized in Table 3.
Table 3: Semantic alignment metrics (ROSCOE-SA)
Score Faithfulnesx-Stpp
Description
level score is based on the alignment from hypothesis steps to source sentences, and calcu- lated as the mean reasoning alignment score over the steps of reasoning (see illustration in Appendix D. Figure 3): (1/N) Eie raalign(k, -+ s). Faithfulness measures if the model misinterpreted the problem statement, or the reusoning chain vague, inrelevant, misuses information.
(h - s)
Palthfulness-Toker
We extend step-level embeddings of the Paithfulnoss-Seep by measuring similaritics between the token embeddings: (I/(N M)) E [r-align((h, (b, - + Eje: r-align" (aliggn""""((,,,, (fr.) - x]], as shown in App. D, Fig. 3. M, is the number of tokens step hs M E/ Ad, is the total number tokens the reasoning chain, Aug is the jth token in step, and r-align bke the alignment vector from tokens step Ny all tokens in
(h 4)
Informativenesx-Sepp (Info-Step) (h ++ a)
Measures how information present in the source used in the reasoning steps: ((1/T) E r-align(s, A) + (1/N) Em= r-align(h, + a)l Info-stop gives higher score to reasoning steps that are well-grounded with respoct to the source, and identifies the degree of information from source that is covered by the generated hypothesis, A lower Step score corresponds to the reasoning steps that not related the source sentences or have provided in the context.
information
Repetition-Ton Token (h, hn)
To identity repeated, or paraphrased steps, we look at the token alignment scores between all steps in the hypothesis chain: 1 - max mAX 1-0--1 (1/M) En r-align (h,) - A,)]. For each pair of sentences, we look the mean token alignment, and find those sentences that maximize this alignment score. In other words, Repetition Token will punish chains where there are at least two stops with high overlap token embeddings.
Hallucination
aligmment score to identily steps that are both not related to the context and not the reference chain (lo avoid punishing for possibly relevant commansense knowledge): - mmxxst. ([1 - r-align(h - and (-) the element-wis product.
To lind irrelevant reasoning steps,
(h + (s)
*] [1 r-align(h - r.)). Here. 1 an onl-ones
Redundancy (h- + r)
To find chains that contain information that is not required to solve the problem (i.e., redundant steps), we identify those hypothesis steps that are least aligned with the reference stops: min.-1 N r-align(h;; - r). This score punishes chains
with steps that are not required for the correct solution.
Semantic SSverage-Step ((r.)) 6)
score can be viewed as a measure how casily gold reference could be generated by hypothesis. compares step level grounding of the hypothesis with respect the source, and the gold reference grounding: ((I/T) D r-align(r, (1/N) E rallgnn(h, s)]. where | indicates absolute value.
Reasoning Alignment (h r)
The most straightffrward way evaluate the correctness of the hypothesis chain is compare the degree of the overlap between the hypothesis and the reference. One way of doing that to measure the reasoning alignment betwoon them:
(I/N) D r-align(h, -> r).
Commonsense
stated in the source, but required to solve the problem such general world knowledge (e.g." "velocity is distance divided by time". "I foot is 12 inches". "all ducks are birds", etc.). We detect such information by extracting steps in the reference reasoning that are grounded by the source text:
Measures if hypothesäs lacks steps that
(r- ())
max;-1...c ([1 r-align(r - 1x] [1 - r-align(( - s)]).
Missing Step (r-
To identify steps that are missing from the hypothesis hut could required to solve the problem, we look at the alignment
between reference and the hypathesis, similar to Redundancy. However, here we go through each step in the reference, and
check if there is similar step the hypothesis: min x (r-align(rr-
*Semantic alignment refers to determination of relations between concepts with the same similar intended meaning (Agirre et al., 2013).
4

---

Published as a conference paper ICLR 2023
4.2 SEMANTIC SIMILARITY METRICS (ROSCOE-SS)
Semantic similarity metries quantify the degree of semantic equivalence between pieces of text. As opposed to the ROSCOE- SA metrics, ROSCOE-S5 SS considers text as a whole, rather than relying on text units comparisons. We propose the following metrics summarized in Table 4.
Table 4: Semantic similarity metries (ROSCOE-S3).
Score
Description
Informativeness-Cro Chain (Info-Chain) (h-+a)
Similar to Info-Step, this metric quantifies the degree of agreement between the hypothesis chain and the source and calculated as [1 + coel co a)|/2. We embed reasoning chain and source context as a whole, as opposed to using step-wisc introduced Table
embeddings in *-Step types
Sepetition-Step (hg++h.)
Measares repetition-related emors on the step level by checking if paraphrases information already mentioned in the previous steps: (1 mAX max,mi-s--1 1e0e- A,]1;/2 /2. Unlike Ropetition- Token, which orderless and compares individual tokens in pairs of steps, Repetition Step considers step embeddings similarity and is more rohust to changing contexts.
Semantic Coverage- Chain (r + h)
Reflects the overall degree of similarity between the reference and hypothesis chains, comparing reference and hypothesis embeadinggs as a whole: cos(r, c2
4.3 LOGICAL INFERENCE METRICS (ROSCOE-LI)
Logical inference metrics (Table 5) measure logical errors between pieces of text. We use an NLI model that was trained to classify sypothesis-conter pairs into entailment, neutral, and contradiction classes (Laurer et al., 2022) to infer the contradiction probability Poontr-
Table 5: Logical inference metries (ROSCOE-LI).
Score Self-Conssstncyy
Description
Measures logical entailment emrrrs wäthin the reasoning steps: maKiun.. N maxx by). This metric will punish chains where there a pair steps that are likely contradict each oiher.
(bgethg)
Concoe-on (h ++ s)
Measures logical eatailment errors between any generated reasoning h. and the souree context mak N max T Poonte Pfi,teAf, (f,,,,). s,). Specifically, for each reasoning step we measure the probability that contra- diets any sentence the context. We lake the maximum probability of contratiction over all steps, folkowing the logic that a contradiction anywhere in the reasoning chain signals failure of the overall argument.
4.4 LANGUAGE COHERENCE METRICS (ROSCOE-IA) LC)
To evaluate language coherence (Table 6), we use perplexity PPL as scored by the GPT2-Large model (Radford et al., 2019), and English grammatical acceptability Param scored by the classifier model from Krishna et al. (2020). Both models were used as-is with no finetuning. Table 6: Language coherence metrics (ROSCOE- -CC).
Score
Description
Perplexity-Chain (f)
Average perplexity of all tokens in the generated reasoning steps: I/PPL((). The context used to score each token the previous tokens in the current and all previous steps. Steps are joined with space character. To keep the range and orientation consistent with the other scores we invert the perplexity.
Perplexity-Step (h.))
Average perplexity of all tokens in the generated reasoning steps, where the context used to each token only the step: I/((/N) EM, PPL(A.)). To keep the range and orientation consistent with the
previous tokens within the other scores we invert the perplexity.
Granunar (hu)
Probablity of grammatical acceptability of each step, averaged over all steps: (1/N) ( N P (Aa).
5 EXPERIMENTAL SETUP
Diagnostics Datasets. We construct first category of labeled datasets by generating perturbations - i.e., deterministic modifications on half of the reference reasoning steps and assign binary labels based on whether not chain has been perturbed. We select seven language understanding and entailment datasets
5

---

Published as a conference paper at ICLR 2023
that require complex problem solving skills, and have reference step-by-step explanations: Entailment-Bani (deductive reasoning) (Dalvi et al., 2021). ProofWriter (logical reasoning) (Tafjord et al., 2021): 1); three arithmetic reasoning datasets MATH (Hendryeks et al., 2021), ASDIV (Miao al., 2020) and AQUA (Liang et al., 2018); EQASC (explanations for commonsense question answering) (Aggarwal et al., 2021), and StrategyQA (question answering with implicit reasoning strategies) (Geva et al., 2021) (see dataset details in App. 1. Using our taxonomy, we introduce 12 error perturbation rules and apply on these datasets to construct our diagnostics datasets (see details in App. E.3).
Human Judged Datasets. We select our second category of datasets from commonly used complex reasoning tasks: GSM8K (arithmetic reasoning) (Cobbe et al., 2021), DROP (discrete reasoning) (Dua et al., 2019), ESNLI (deductive and commonsense reasoning) (Camburu et al., 2018). COSMOS-QA (commonsense reasoning) (Huang et al., 2019) and SemEVAL (Ostermann et al., 2018) (commonsense reasoning). Wei et al. (2022) provide model generated chain of thought reasoning steps for GSM8K. We used chains produced by the 175b_verification model to annotate for reasoning errors. For other datasets, we prompt GPT-3 LLM (Brown et al., 2020) with few-shot in-context examples to obtain step-by-step reasoning sequences (see examples in App. E.2). We use the eπror types in our taxonomy in Table 2 as human evaluation perspectives of reasoning errors where we solicit five expert annotators The data collection interface provided judges with the source text (e.g., source and a question, or hypothesis, premise, and a question if they entail) and associated reasoning text clearly separated into individual steps. Judges were asked to rate the chain as a whole (e.g., on overall quality) as well each individual step (e.g.. commonsense errors, contradicts with the previous steps). App. Table 16 summarizes the distribution of error types annotated by the judges. See App. F for details.
ROSCOE Training. To obtain reasoning step embeddings, we finetune SimCSE (Gao et al., 2021), a supervised sentence similarity model extending the RoBERTa word embedding model (Liu et al., 2019) on multi-step reasoning datasets we listed in 55 (see details in Table 11)". SimCSE is a contrastive learning model that is trained triplets of reference reasoning steps, positive and hard-negative hypothesis reasoning steps to minimize the cross-entropy objective with in-batch negatives. For contrastive learning, we use the context and reference reasoning steps as positive sample (s, r), and context and perturbed reference steps (s, h) as hard-negative pairs. For finetuning, we embed source context and hypothesis chain as a whole, without splitting into steps. With the finetuned model we embed each individual step, as well as a reasoning chain as a whole. We use the pretrained checkpoint of supervised SimCSE model sop-sincce-roberte initialize our model, and further train it for five epochs on our synthetic train data (details in App. G). We also compare ROSCOE scores calculated against sup- pup-simcse-rus sincce-roberta-bass SimCSE model, and all-mpnet-hase-V sentence embedding model (Reimers & Gurevych, 2019) to understand metrics sensitivity to the embedding method.
to
Baseline Metrics. We use text generation evaluation metrics as baseline metrics and comprehensively examine the ones outlined in $2, which are: n-gram match based metrics including ROUGE-1, ROUGE-2, and ROUGE-L (Lin, 2004); pre-trained scores including BLEURT (Sellam et al., 2020), PRISM (Thompson & Post, 2020), BERTScore (Zhang et al., 2020), BARTScore using the Faithfulness (s - h) direction for factuality and relevance, and its finetuned variant BARTTScoreCCNN-PP
BARTScore+ (Yuan et al., 2021); and information alignment metrics of CTC, CTC-Relevancy and CTC-Consistency. We also include BARTScore-P, which we obtain by finetuneing BART (Lewis et al., 2020) on the same reasoning datasets we use for finetuning our SimCSE embedding models. Most of our ROSCOE metrics are constructed reference- free. We also have metrics that use reference reasoning steps which we examine against human judgements. We use the official code for each metric.
'We chose expert annotators over crowd-sourcing, because our annotation task is cognitively challenging and requires fine-grained annotation. Finee-tuned model is available https: haallbleehhttppssgggggggggeeeee htttss//waggggggaaaaccceceeeoeee 5tt-rrrrrtt-t----rrtteet base
6

---

Published as a conference paper at ICLR 2023
Meta Evaluation. We use Somers' D (Somers, 1962), which measures the ordinal association between two measured quantities, to meta-evaluate each scorer against synthetic and human scores. We prefer Somers' D over more commonly used Kendall's т or Kendall's 7-b, because it is better in handling the ties of a biased random variable (Agresti, 2010, Section 7.1.5), which imposes upper bound on the possible values Kendall's 's r--b) can take. For each score Y considered, our correlations are built against the biased random variable X E [0. 1]. represented by the perturbation error presence indicator and evaluated using XX = T (
6 EXPERIMENTAL RESULTS
Controlled Experiments with Diagnostics Datasets. Table shows Somers' D correlation for metrics mea- sured reference-free on six different datasets and compares baselines ROSCOE-* aggregated categories cal- culated with finetuned embeddings: ROSCOE-SA ROSCOE- ss, ROSCOE- LI, ROSCOE Results also include ROSCOE metries with all-mipet-base--2 (ROSCOE sa', ROSCOE-SS' and cesiincs--rbbrrtee base (ROSCOE-SA* ROSCOE-SS sentence embedding models. Correlations for ProofWriter are taken on its depth-5 subset. We report highest correlation scores across perturbations within each dataset. The breakdown of all ROSCOE metrics is in App. Table 18.
We observe that: (1) ROSCOE can out- perform all other reference-free methods on all six diagnostic datasets, (2) the gains for more pronounced in four out of six diagnostics datasets, which sug- gests that ROSCOE can capture hallucinations and repetitions in step-wise reasoning. On Proofwriter, our scorers show lower corre- lations, because as shown in Table E. the context is list of facts and rules and the reasoning steps can include unordered fact and rule combinations, but still correct an- swer can be deduced. This makes it challeng- ing for ROSCOE to evaluate the steps in se- quence. Overall, the correlations of the base- line metries are much lower than ROSCOE, because the baseline metrics are designed to capture the semantic or lexical overlap be- tween a reference and hypothesis and it is harder to detect logical consistency without a golden reference text. ROSCOF, is specifically focused on reference-free settings, and can gauge each individual step against the source and other generated steps. In fact, our met- ries also work well against the baselines in the reference-based setting (comparing against reference reasoning steps). In App. Table 19 we present correlations when metrics are measured as reference-based. We also observe that finetuning SimCSE gives highest improvements on the ASDIV dataset. ASDIV is 1 1-step reasoning dataset (see App. Ta- ble 12), where step is represented by an equation with one of the arithmetic perturbations added. We
Table 7: Somers' D correlation of different metries on six Diag- nosties datasets. Metries measured reference-free on (s. h). We take the maximum score over different perturbations. The two highest correlations for each dataset are bolded and underlined, re- spectively. Correlations that are not significant (p-value 2 0.05) are omitted when aggregating, and "_" denotes absence of any signifi- cant correlation. Breakdown of all baseline and ROSCOE metrics is shown in App. H.I, Table 18.
ROSCOE-
EntBank Math AQUA ProofWr.
EQASC
ASDIV
ROUGE-L BLEURT
0.365
0.156
0.264
0.106
0.315
0.269
0,257
0.148
0.252
0,024
0.447
BERTScore
0,380
0.124
0,220
0.117
0.462
0.322
BARTScore
0.358
0.185
0.317
0,081
0.415
BARTScore+
0.315
0.164
0.251
0.054
0.297
BARTScore-P
0.186
0.128
0.215
0.011
0.276
PRISM
0.453
0,208
0.191
0.235
0.436
CTC Relev.
0.258
0.188
0.217
0.394
0,485
0.382
CTC Consist.
0.282
0.157
0.513
0.270
0.396
ROSCOE-SA
0.919
0.939
0.971
0.763
1.000
0.879
ROSCOR-SX
0.913
0.936
0.972
0.771
1.000
0.198
ROSCOE- SA?
0.919
0.939
0.971
0.732
1.000
0.515
ROSCOE-8 88
0.955
0.924
0.982
0.624
1.000
0.857
ROSCOE-SS!
0.909
0.932
0.982
0.631
1.000
0.280
ROSCOR-SS*
0.901
0.949
0.991
0.621
1.000
0.289
ROSCOB-LI
0.917
0.331
0,424
0.289
0.793
0.771
ROSCOR-ICC
0.604
0.392
0.359
0.788
0.859
0.485
'We use SciPy (Virtanen et al., 2020) to calculate correlations and obtain p-values from hypothesis test where the null hypothesis absence of association.
7

---

Published as a conference paper at ICLR 2023
hypothesize that including these patterns in finetuning helped the model to better learn relationships be- tween context and equations, and resulted in higher scores. On EQASC dataset, Repetition" scores are able to catch all duplicated steps in chain, i.e., we can separate perturbed and non con-peetrrbed chains based on the given threshold value for the Repetition scores, and achieve perfect correlation scores (App. Table 20). To understand if finetuning actually helps to improve scoring, we compare non-aggregated metrics (see details in App. Table 18). We observe, that finetuning indeed helps to improve ROSCOE: on average across datasets, all correlations except Repetition scores improve (up to 0.556 on Informativeness Chain), with mean Repetition-Tokes Token not changing, and mean Repetition-Step degrading by 0.005. We speculate that since we finetune the model using reasoning chains and context as a whole, it helps to better capture step-by-step rationales, while possibly degrading on word and sentence-level semantics.
Meta-Evaluations on Human Judge- ment Datasets. Table 8 reports a sum- mary of meta-evaluation of ROSCOE met- ries comparing against baselines on hu- man judged datasets. The correlations are measured based on the presence of a particular error from Table 2 and we report the highest correlation across all enror types within each dataset. We ob- serve that: (1) on all tasks, ROSCOE met- ries outperform all other baselines when evaluated as reference-frer
Table 8: Somers' D correlations of metries with human judgement. We report the maximum over the error types in Table All metrics are measured reference-free on (s. h). The highest two correlations each column are bolded and underlined, respectively. Correlations that are not significant (p-value 2 0.05) are omitted when aggregating, and "." denotes absence of any significant correlation. Breakdown of all baseline and ROSCOE metries shown App. H.2.
DROP
GSM8K
ESNLI
COSMOS
SemEVAL
Rouge-
0.278
0.252
-0.411
-0.788
BLEURT
0.328
0.256
0,541
0.218
-0.356
BERTScore
0.275
0.235
0,590
-0.420
-0.295
BARTScore
-0.835
-0,546
0.549
-0.544
(2) overall, ROSCOE yields considerably better corre- lations, which indicates that step-by-step reasoning generations can be more effec- tively evaluated with ROSCCE. In gen- eral, most correlations with human judge- ments are moderate when compared to the synthetic correlation scores, indicat- ing that step-by-step reasoning evaluation among the cognitively hard tasks for neural models (Deutsch et al., 2022). In- terpretable metries such as ROSCOE can
BARTScore+
-0.665
0.482
-0.186
BARTScore-P
-0.642
0.255
-0.207
PRISM
-0.733
-0.455
0,580
-0.376
CTC-Relevance
0.333
-0.371
0.334
-0.349
CTC-Consistency
0.462
-0.174
0.647
0.275
-0,301
ROSCOR-SA
0.578
0.392
0.521
0.555
0.337
ROSCOB-SA!
0.790
0.500
0.799
0.638
0,485
ROSCOB-SA*
0.578
0.392
0.599
0.555
0.337
ROSCOE-SS
0.824
0.514
0,530
0.593
0.411
ROSCOB-SS!
0.791
0.471
0.507
0.642
0.508
ROSCOE S5
0.799
0.638
0.531
0.658
0,535
ROSCOB LI
0.584
0.345
0.531
0.444
0.372
RCSCOB-LC
0.205
-0.184
0.447
-0.212
0.517
provide better information about model's reasoning skills, thus future work should improve such metrics on aligning with human judgments. In App. H.2, we show fine-grained experimental analysis per each human labeled dataset. Specific examples showcasing ROSCOE scoring abilities are summarized in Table 40.
7 ANALYSIS
How sensitive are ROSCOE metrics against level of errors? To evaluate how well metric values match human assessment of reasoning, we measure sensitivity to the level of errors. We perturb sentences the MATH (arithmetic) and EntailmentBank (deductive reasoning) diagnostic datasets (similar to & 5) and inject different levels of errors into the reasoning text. Using randomly selected perturbation types, we construct up to maximum of 3 perturbations per instance. We measure the correlation (Somers' D) between the reasoning inconsistency level 1, 2, 3 of the reasoning steps (i.e., the number of injected errors) and the metric score. Fig. illustrates the results averaged over different perturbations.
We expect the metrics correlate with humans better when the level of errors is high. Both semantic alignment of the reasoning ROSCOE-SA and the semantic similarity metrics ROSCOE-SS show consistent behavior
8

---

Published as a conference paper ICLR 2023
on both datasets, while baseline metrics fluctuate with low correlations. Baseline metrics perform better on EntailmentBank, On MATH, ROSCOE-LC and the baseline metrics show minimal impact, which can be that some of the perturbations applied on the MATH dataset (e.g., RandomOperation, or ShuffleNumbers) are harder to detect with language model based (BARTScore) and NLI model based (ROSCOE LC)) metrics.
What does ROSCOE illuminate about scores across errors and tasks? For an ideal scorer based on ease of use, it would be possible to pick a set of fixed thresholds that had error discrimination power across datasets. However, we show that this datasct-agnostic ideal is currently not possible and an issue endemic across scores, including baselines. We study which metrics correlate strongly with which perturbations, with focus of con- sistency across datasets. From this, we plot the interquartile ranges for strongly correlated metric and perturbation pairs. We show sample of these in Fig. 2, though find that the trends generally hold across metrics and perturbations (see Fig 6). We note that within a given dataset, scores are well separated: the perturbed version of a dataset for a given score and perturbation type shows little interquartile overlap with the original version. However, this does not hold across datasets e.g., in (Score: Info- Chain, Perturbation: Repetition), if one were to set a detective threshold for the Repetition perturbation based off EntBank (around 0.95), it would mark almost all values of EQASC as perturbed, even non-perturbed sam- ples. This shows the challenge of using metrics for classification without calibration for drifts in both mean and variance across datasets, even a metric generally correlates well with detecting given emrrr
ROSCOE-SS DE-SA -ARTS
ROSCOF-J
ROUGE-L
ROSCDE-LC
PRISM
1.00
0.75
0.50
0.25
0.00
1.00
EntBank
0.75
0.50
0.25
0.00
Level of Error
Figure Sensitivity of selected metrics on Somers' D by injecting levels of error into reasoning steps.
Score: Info Chain. Porturbation: Repettion
Score: Parthnuincos Step. Perturt stion Negate Stop
Score: BERTScore, Perturbation: Rep Mon
(ProotVe.. perturbes)
|Prottsr origina0
(Math, perfurbeed)
(Math, originan
(EntBank, perturbed)
jtrtftank, origina-
(EQASC, perturbes))
origina0
JAQUA, perfurbee))
(AQUA, originan
0.6
0.6
-0.4
-0.2
orggnal
Figure 2: Box-and-whisker plots of interquartile ranges of scores, for perturbations and reference-free metries with strong Somers' values. Scores are split by dataset and perturbation use. While interquartile ranges separate well by perturbation use within single dataset, there is overlap across datasets. This shows the drift of neural scores across datasets and applies to both ROSCOE (left, center) and strong baselines (right).
8 CONCLUSION
In this paper, we introduce ROSCOE, a new suite of interpretable, unsupervised metrics that enables evaluation of step-by-step reasoning generations of LMs when no golden reference generation exists. We present a taxonomy of reasoning emrors used to generate and evaluate our metrics. Experimental results, from evaluating on both synthetic and human-labelee datasets exhibiting multiple types reasoning (commonsense,, arithmetic, and logical inference, etc.), demonstrate superior performance compared to prior semantic and lexical similarly based baseline metrics for text generation. Our analysis shows improved capability in evaluation of reasoning exhibiting nuances, such as factual and logical errors in step-wise decisions.
9

---

Published as a conference paper ICLR 2023
ETHICS STATEMENT
Explainability builds transparency and trust for users, eases bug-fixing and shortens improvement cycles for metric designers, and will be required by law/regulations for Al systems be applied to large-scale, high-stakes domains. In this context, we hope our work will catalyze efforts on the topic of explainable evaluation metries for language model rationale generations. We should mention that our evaluation metries do not monitor the explanations from integrity or bias perspectives. Our work also uses five human expert annotators and in the annotation process, annotators need to rate the model generated candidate rationals. While the model-genetee generated explanations can produce potentially unsafe content, the datasets for annotations include domains related to logical and arithmetic concepts and general commonsense knowledge. The anecdotal consensus was that the generations were safe and didn't 't include biased statements.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our empirical results, we will open source our code to Github, which will contain: instructions for installing the virtual environment, data preprocessing, all score generation and correlation scripts (both for ROSCOE and baselines), and trained embedding models. Detailed explanation of all the finetuned models and metrics given in the main paper well as in the Appendices. We will also release all the diagnostic and human judgment datasets used in our experiments.
REFERENCES
Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. Explanations for CommonsenseQA: New Dataset and Models. 2021.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. *SEM 2013 shared task:
Semantic textual similarity, In Second Joint Conference on Lexical and Computational Semantics ( (*SEM),
Volume Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pp-
32-43, Atlanta, Georgia, USA, June 2013. Association for Computational Linguistics. URL https:
//aclanthology.org... wtt.prrr////1...../
Alan Agresti. Analysis of ordinal categorical data, volume 656. John Wiley & Sons, 2010.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pp. 632-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10.18653/v1/D15. 1075. URL https: http///aaaannnoo.o///////// D15-1075.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. 33: 33:1877-1901, 2020. URL
Camburu. Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Natural language inference with natural language explanations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Gareett (eds.), Advances in Neural Information Processing Systems 3/, pp. 9539- 9549. Curran Associates, Inc., 2018. URL http:://pppppeeeeeee --a natural -
10

---

Published as a conference paper ICLR 2023
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. Evaluation of text generation: survey. CoRR, abs/2006. 14799, 2020. URL
abs abbs/200..1479.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
Miruna-Adriana Clinciu, Arash Eshghi, and Helen Hastie. A study of automatic metries for the evaluation of natural language explanations. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pp. 2376-2387, Online, April 2021, Association for Computational I. Linguistics. doi: 10.18653/v1/2021 .eclle///222.eaallllcao .aw.:mm....mmmmmmmm200000 202 URL https:// ac wanthology.orggooooooooooooo heayyyorgg2221... 2021. waw.l...lcll.cmmo.............l. mma n020..
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
Antonia Creswell, Murray Shanahan, and Irina Higgnns Selection-inference: Exploiting large language mod- els for interpretable logical reasoning. arXiv, 2022. URL atttps://aatt /ttp:////////x/a///aa/// abs//bs////// 220500000.....
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, and Peter Clark. Explaining answers with entailment trees. EMNLP, 2021.
Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. A survey of the state of explainable AI for natural language processing. In Proceedings of the 1st Conference of the Asia- Paci-Paciiic Chapter of the Association for Computational Linguistics and the 10th International Joint Conference Natural Language Processing, pp. 447-459 Suzhou, China, December 2020. Association for Computational Linguistics. URL ht / aaaaacaannt aclanthology.orrg
Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. Compression, transduction, and creation: A unified framework for evaluating natural language generation. In EMNLP, 2021. URL https: hac//aclaante wwwaannnoooooorrro 20/2222211222 1.mpp...mppp.....mm main. 5..5999 599.pdf.
Daniel Deutsch, Rotem Dror, and Dan Roth. Re-examining system-level correlations of automatic summa- rization evaluation metrics. arXiv preprint arXivv2204.10216. 2022.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 2068-2262200000 2378, Minneapolis. Minnesota, June 2019. Association for Computational Linguistics. doi: //acl anthol wwwphholllyyoogg//e 9- 1.46.
URL https:
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint /04.082, 2021.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
11

---

Published as a conference paper ICLR 2023
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Machine reading comprehen- sion with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP), pp. 2391-2401, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 18653// 1243. URL https : heatttss//aclaatthollloooo aolaantooooooooooooooooooo /tr//r////// 110-111--3 1243.
Alon Jacovi and Yoav Goldberg. Aligning faithful interpretations with their social attribution. vol- ume 9, pp. 294-310, Cambridge, MA, 2021. MIT Press. doi: 10.1162/tucla_a000 a 00367. URL https: / //accaaanntt anchonnthlllo hottcchooogy..oo wow.looooyyoo22....ooo wttp./orr//22..acl1111.
Alon Jacovi, Swabha Swayamdipta, Shauli Ravfogel, Yanai Elazar, Yejin Choi, and Yoav Goldberg. Con- trastive explanations for model interpretability. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1597-1611, Online and Punta Cana, Dominican Republic, November 2021, Association for Computational Linguistics. doi: 10. 18653/v1/2021 .emnlp-mmmmeee malpp.-mannn. 120. URL https:::p//.... /
Kalpesh Krishna, John Wicting, and Mohit lyyer. Reformulating unsupervised style transfer as paraphrase generation. In Empirical Methods in Natural Language Processing, 2020.
Andrew K Lampinen, Nicholas Roy, Ishita Dasgupta, Stephanie Cy Chan, Allison Tam, James Mcclelland, Chen Yan, Adam Santoro, Neil C Rabinowitz, Jane Wang, and Felix Hill. Tell me why! Explanations support learning relational and causal structure. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11868-1181 11890. PMLR, 17-23 Jul 2022. URL https: / ppwppediongggooooooo ngs.ml r.press///ee... v 62 / amp nen 2t.mm...
Moritz Laurer, Wouter van Atteveldt, Andreu Casas, and Kasper Welbers. Less annotating, more classifying- addressing the data scarcity issue of supervised machine learning with deep transfer learning and bert-nli. 2022.
Christoph Leiter, Piyawat Lertvittayakumjorr, Marina Fomicheva, Wei Zhao, Yang Gao, and Steffen Eger. Towards explainable evaluation metrics for natural language generation. CoRR, 0tss/2033.11333) 2022. httts::: /ttp//iiioogo./////////////////t
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising soquenceto-seqqenne pre-training for natural language generation, translation, and comprehension. lion for Computational Linguistics, pp. 7871-7880, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.186633w1/2020.cco acinm.....mmin.....m main. 703. URL https: hatp://s//ccaatttoooooooor/////0 ma main.703.
Proceedings of the 58th Annual Meeting of the Associa-
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206. 148558 2022.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners. arXiv, 2022. URL https:/ http::::s///axx.oo////////////// 2206.02336.
Chao-Chun Liang, Yu-Shiang Wong, Yi-Chung Lin, and Keh- Yih Su. A meaning-based statistical English math word problem solver. pp. 652-662, June 2018. doi: 1/653/11/1181100 URL https:
/ //aclaannnn waw.laaahooooooorgggoaooo
12

---

Published as a conference paper ICLR 2023
Chin- Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 84, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https: //aaclatttolooy..rrg///.3
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint orXiv:/97.116600 2019.
Shen-yun Miao, Chao-Chun Liang, and Keh- Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. pp. 975-984, 2020.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 2021.
Juri Opitz and Anette Frank. Towards a decomposable metric for explainable evaluation of text generation from AMR. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1504-1518, Online, April 2021. Association for Compu- tational Linguistics. doi: 10.18653/v1220211 eacl- m2i...129. URL https: http///aaaaaaaaaaoorrr/..
202l.eacl-main..
Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. Semeval-2018 task 11: Machine comprehension using commonsense knowledge. In *SEMEVAL, 2018. URL https:
1119.ppff
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Nils Reimers and Iryna Gurevych. Sentence-bert Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2019. URL https: a/aaaiiiiii aww.i..oigorgo/.//o............a abs 908.10084.
Ehud Reiter. Natural language generation challenges for explainable AL. In Proceedings of the 1st Workshop on Interactive Natural Language Technology for Explainable Artificial Intelligence (NI4XAI 2019), pp- 3-7. Association for Computational Linguistics, 2019. doi: 10. 00.18865//1//111-84000 URL https: / /aaclannhooogy.orrg www.l..orgg/ll..e. - 8402.
Christopher Michael Rytting and David Wingate. Leveraging the inductive bias of large language models for abstract textual reasoning. 2021. URL https:/ / www.reeiee.wweeeeeeeeee forum? id=urueR033mm
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT:
Learning robust metrics for text generation. In
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892, Online, July 2020. Association for Computational Linguistics. doi: 10. 00011853//112222.ac--o 704. URL
https:// acl anthol wwwhhhllgy.org/////eee 202c mm -m....m0mnn.
Robert H Somers. A new asymmetric measure of association for ordinal variables. American sociological review, pp. 799-811, 1962.
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3621-3634, Online, August 2021. Association for Computational Linguistics. htt.://w//222...nnnnnaaaacll/l/
doi: 10. 00.11653/v1/2021 fin.ings-acl.317.. URL https: /t//aclaaa antho www.aantoooooooooo log.oorg/2221...... findings- ac1.317.
13

---

Published as conference paper ICLR 2023
Brian Thompson and Matt Post. Automatic machine translation evaluation in many languages via zero-
shot paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 90-121, Online, November 2020. Association for Computational Linguisties. doi: 1..18653/v1/222 http220002..mmmmaaiiii.. URL https::///// http://aaaaacooollll.looo.ce 200.0/mm1222.m0..pp maatnnlpppmi........ main.8.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournaeea,, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, lan Henriksen, A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272 2020. doi: 10. 91000/44992-01-0666660
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. /171, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou, Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. 2022.
Sean Welleck, Jiacheng Liu, Ximing Lu, Hannaneh Hajishirzi, and Yejin Choi. Naturalprover: Grounded mathematical proof generation with language models. 2022.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: from general language models to commonsense models. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4602-4625. Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653W1/20222maad0 maw..naammmama3441.. URL https: / http://aaaanntlll...rrrro
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text text gen- eration. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 27263-27277. Cur- ran Associates, Inc., 2021. URL httpt::///// 4422655fec23300011222eee3333888 Papez..d..
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning. arXiv, 2022.
Tianyi Zhang, Varsha Kishore, Felix Wu* Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https: / //wpenreviw.nne./// htwprrrrreee..ete.ooorrrrr.eeeee http://rrmmmmttkeeeeeee.e
14

---

Published as conference paper ICLR 2023
APPENDIX
A LIMITATIONS
Our study is the first initial step that investigates the evaluation of the step-by-step reasoning produced by large language models. Our taxonomy (in Table 2) covers several reasoning errors and we designed metrics to evaluate a spectrum of criteria including the ones in the taxonomy. Even though we cannot say we cover all possible reasoning errors, metrics generic enough, work natural language rationales, and consider the alignment with the input context and the generated explanation. Nevertheless, we believe our study can spur others investigate different reasoning errors and use our code and datasets templates extend further.
Due the extensive analysis needed to thoroughly test and communicate the ability of our proposed metries to capture reasoning errors, we decided to leave some follow-up questions, such as the application of these metries for improving downstream task performance, for future exploration.
SHOT PROMPTING EXAMPLES (CONT. FROM $ 1)
Below the 2-shot example used to generate the explanations from GPT-3 as we show in the Fig. 1.
I will explain how to make logical inferences:
John wakes the sound of his children giggling outside the door. They bust and say "Happy Father's Day!" with plate full of food. 2022 has been a great year.
Context:
Question: Can we infer that John was born before the inauguration of Barack Obama? Explanation: We can infer that John is father because of "his children". We can infer that the current year is 2022. Barack Obama was inaugurated in 2009, Ohama was inaugurated. John's children old enough speak, has been about 13 years since John born after Ohama's inauguration, he would have been younger than 10 when he had children. So, can infer that John was born before Obama's 's inauguration. The answer is "yes".
Context: Sam sees piano in the subway stations. Laughing, his friends prod him to play, saying "Remember how amazing your rendition of Chopsticks was at the Christmas party?"
Question: Can infer that John majored in piano Juilliard?
Explanation: Juilliard is top school for music, so someone who majored piano Juilliard would be expert pianist. "Chopsticks" is very simple song to play on the piano. friends were therefore likely being sarcastic when they told him to play, which would imply that Sam was not an expert piano player. So it is unlikely that Sam majored in piano at Juilliard. The answer is "no".
Table 9 illustrates sample if in-context examples we used as prompts obtain step-wise explanations.
15

---

Published as a conference paper ICLR 2023
Table 9: Example prompts used to generate step-by-step explanations using GPT-3.
(Used for FSNLI dataset)
Explain if premise entails hypothesis.
Premise: Tomato sauce (also known as Ncapolitan sauce, and referred to in Italy as Salsa di pomodoro) refers to any of a very large number of sauces made primarily from tomatoes, usually to be served as part of a dish (rather than as a condiment). Tomato sauces are common for meat and vegetables, but they are perhaps best known as sauces for pasta dishes.
Hypothesis: Tomato sauce only rarely used top pasta.
Explanation: Sulsa pomodoro is made of peppers. The
No.
Kirk Johnson (born 1960) is an American scientist, author, curator, and museum administrator, and is currently serving as Director of Smithsonian's 's National Museum of Natural History. Dr. Johnson is the host of the PBS Nova series, Making North America, which three-part series that describes the shaping of North America, which aired on November 4, and 2015.
Premise:
Hypothesis: johnson hosted Making North America when turned 50. Explanation: Johnson was born in 1960 and hosted Making North America in 2015, he was 55, not 50. The answer is No.
Premise: person rides his bicycle in the sand beside the ocean.
Hypothesis: A person is on beach. Explanation: person rides beside the ocean means is beach if there is sand beside the ocean. The answer is Yes.
(used for SEMEVAL dataset)
Explain if premise entails hypothesis.
Premise: went into my bedroom and flipped the light switch. Oh, see that the ceiling lamp not turning on. It must be that the light bulb needs replacement. go through my closct and find and place it my pocket. also get my stepladder and place under the lamp. I make sure the light switch is the off position. I climb up the ladder and unscrew the old light bulb. I place the old bulb in my pocket and take out the new one. I then screw in the new bulb. climb down the stepladder and place it back into the closet. I then throw out the old bulb into the recycling bin. I go back to my bedroom and turn on the light switch. I am happy to see that there again light my room. Which room did the light go out in? Hypothesis: Bedroom
light bulb that will fit this lamp
Label: Correct
Explanation: The answer is bedroom, because the light went out in the bedroom. The person replaced the light bulb in the bedroom.
The weather was so nice today that decided to have barbecue. called up some of my friends invile them over. set up some chairs and tables outside for my friends to sit around. I then took out some hot dogs from my fridge and brought them over my harbecue. My harbecue
Premise:
gas barhecue, so all had to do was press the knob and turn it to ignite it. Once gut a little fire going I put my hut dogs the grill over it to cook. I closed the lid and left my hot dogs to cook for about ten minutes. After ten minutes decided to check on them to see they were done. They looked like they were cooked, so I put them all on a plate to serve to my friends. went to my kitchen and got hot dog buns, ketchup, mustard, condiments on the table my friends could help themselves. We atc the hot dogs and relaxed around the table. Why are we having the barbecue, to celehrate occasion, or just because we want to? Hypothesis: Just because they want to. Label: Correct
relish to serve on top the hot dogs. laid out of the fuod and
Explanation: The answer is correct, because the weather was nice. They decided to have a barbecue. There is no mention of any special occasion.
Premise: Another Nurse can in to look at me, took one look at my rash and said OMG and ran out of the room At that point I started crying. regretted coming to the hospital alone and started wheezing again. Then the doctor came in, her eyes were all bugged out What will I do after the doctor examines me? Hypothesis: will talk with my friends. Label: Incorrect
Explanation: The premise is about the nurse's reaction the patient's rash, and how the patient feels. The hypothesis does not follow up from this premise correctly. Instead, it talks about talking with friends.
16

---

Published as conference paper ICLR 2023
C TAXONOMY OF REASONING ERRORS (CONT. FROM $ 3)
To gain deeper insights into the types of reasoning errors introduced by LLMs while explaining their decisions, we propose a new taxonomy of generic reasoning errors for language problem solving. Specifically, we sampled from the training portions of the logical inference and commonsense reasoning datasets, and prompted GPT-3 with reasoning explanations using prompts similar to App. B. We used task specific in- domain examples for prompting- We also analyzed model generated explanations shared in Wei et al. (2022). We then manually looked into each explanation and identified potential errors that are inconsistent with the source, question the prompt and within the reasoning chain. Some tasks require model classify the logical relationship between premise and a hypothesis. others are question and answering tasks, We adjusted context and prompts according to the type of the task.
Our reasoning error taxonomy summarized in Table 10. contains types of errors concerning an overall chain or an individual step. Specifically, the chain-level coarse soarse-graieed evaluations of the overall reasoning chain deals with overall quality of the step-by-step thinking, coherence, consistency of the explanation within itself, and consistency with the context, etc. On the other hand the step-level tra-ggained evaluations focus on the consistency of reasoning step with the previous steps, if step conveys new and supporting information over the previous steps, factuality or logical inference issues. We use these error categories to construct diagnostics datasets with perturbed errors as well as human judged datasets of reasoning errors. In the taxonomy, we indicate *-step level errors to differentiate from the chain level error types.
17

---

Published as a conference paper at ICLR 2023
Table 10: Taxonomy of Step-by-Step Reasoning Errors. Errors used for perturbations in constructing the diagnostic datasets (Diag.) and for human annotation (Human) of the model generated reasoning chains are also marked. (Cont. from Table 2.
Error Type
Description
Example
Diag. Human
Grammar
Faulty, unconventional, or contro- versial grammar usage
Reference: He chews pieces gum a day so over 30 days he will chew 4030= =4 =4=30=120> 120 picces of gum. Model Expl: Hc cats 4 pieces of gum day and lasts 30 days
so he cats 4+30 = a4*30=120 120m 120 pieces of gum.
Factuality
Information about object (i.e. quan- tity, characteristies) a personal named entity does not match infor- mation provided in the question
Context: Anakin eaught 10 fish. Model Expl: Loesin caught 10 fish.
Hallucination
Information not provided the problem statement and is irrelevant wrong
Context: The busketball team went to the steakhouse eat dinner. The first player atc 6-ounce steak. The second player ale beef tips, containing 8 beef tips, each an ounce in size, The third player ate one-pound steak, And the fourth and fifth players ordered vegetarian meals, In total, how many ounces meat were consumed by the team? Model Expl: The fourth and fifth playets ordered vegetarian meals, for total of 2 ounces of meat.
Redundancy
Explanation contains redundant in- formation, which even though
Context: Bob is cold. If something is quiet then is blue. All blue things round. If something round and smart then it cold. Question: Bob cold.
might be factual, is not required answer the question
Model Expl: If something is round and not smart then it is blue. All blue things are round. Answer: true.
Repetition
Step paraphrases information al-
Context: Parker chews 4 pieces of gum day. A pack of gum has 15 pieces of chewing gum pack. How many packs of gum will he need to last him 30 days?
ready mentioned in previous reason- ing steps
Model Expl.: He chews 4 pieces of gum a day and there are 30 days in month so he chews 4*30= = 120 pieces gum a month. Parker chews 120 pieces of gum month. He needs I 120 pieces of gum month and a pack of gum has 15 pieces of gum so he will need 120/15 = 8 packs of gum.
Missing step
content the generated rea- soning is incomplete and lacks re- quired information to produce the correct answer. If this missing steps are added, the model would produce the correct answer
Context: A farm has 10 2-legged animals and 15 4-legged healthy, how many pairs of animmll legs in total are on the farm? Answer: 40
animals. If all animals
Model Expl:: 10 2-legged animals have 1092=20 legs. 15 4- 4-legged animals have 15 15#4=60 legs. In total on the farm,
there are 20+60=80 legs.Answer: 80 missing step: Total number of pairs 80/2=40
Coherency
Steps contradict each other or do
Context: If the reactor is intact, then it is safe. If the reactor is safe, then no graphite is outside it.
not follow a cohesive story
Model Expl: The reactor is intact. All graphite is inside
There some graphite outside of the the reactor.
Commonsense
Model lacks relations that should known from general knowledge about the world (i.c. how com- pute velocity. how many inches one foot, all ducks are birds, etc.)
Question: Jordan has 2 children who wear diapers. Fach child requires diaper changes per day. Jordan's 's wife changes half of the diapers. How many diapers does Jor- dan change per day?
Model Expl.: Jordan has 200= diapers to change per day. Jordan's 's wife changes 10/2=5.0 diapers per day. For both children, Jordan changes 10+5=15 diapers per day.
Arithmetic
Error in math calculations
Reference: a truck was carrying 20 tons of fertiliser packed in bags, the total number of hags in a truck is 20a20= = 400 bags
Model Expl: If a truck was carrying 20 tons fertiliscr packed bags, the total numher of hags in truck 20*20 40 bags
18

---

Published as a conference paper ICLR 2023
D ROSCOE METRICS DETAILS (CONT. FROM $4)
ROSCOE metrics are constructed under four categories: semantic alignment, semantic similarity, logical inference, and logical coherence. The details of each metric is explained in $4. At the core of ROSCOE se- mantic alignment metrics is the reasoning alignment score, which we designed to measure the grounding of hep-by-step reasoning with respect to the source text. Fig. 3 illustrates the reasoning alignment.
raligu(h s) (0.8.0.7)
4
0.7
-0.5
0.2
4.
0.01
0.8
-0.4
0.01
-0.8
-0.4
0.3
r-align (h s) (0.88007,006009900..4)
42
0.2
Faithfulnesss Step 0.8 0.7
4,3
0.01
-0.1
0.04
0.01
0.6
-0.4
0.3
h11
0.6
-0.5
-0.2
0.9
00.5
0.2
-0.5
Paithfulness-ond
0.8 0.7 0.7 0.6
0.7
422
2+5
ralign(s h) 0.8
Figure Reasoning alignment illustrating the measurement of the Faithfulness-Sun Steepeesss-------seeeee and Faithfulness-Toke semantic alignment scores. h {/1, ]22 is a hypothesis chain with tokens 1/1,1,/1.2. h1.3, h2.1, f2 and {s1, $2, 83 } is context with tokens {s1,1, 82,1, $2,2, $2,3, $3,1: 83,2, 83,3). Aligmment scores from hypothesis to context are highlighted, and alignment scores from context to hypothesis are underscored. The reasoning alignment combines token and step level similarities where each alignment value (cell) is the cosine similarity and explicitly measures the grounding of the token and step-wise reasoning with respect to the source text.
The variation of scorers of the ROSCOE shares some similarities, thus we explain them here:
BARTScore (Yuan et al., 2021) claims that more high level text can be generated using sequence to sequence model. It can support different evaluation perspectives such as factuality (by evaluating from source hypothesis) or informativeness (by evaluating from both directions between reference and hypothesis). BARTScore used measure the probability of generated text from a source text x to target set y:
(1) BARTScorcintroduce two variations: (I) finetuning, in which the BART model finetuned on the task specific dataset to make the pre-training domain closer to the evaluation domain. (2) prompting, in which task specific textual prompt is appended to the source x to get the y. In our experiments we compare the the BARTSSoorbaaselne and one with the prompting variant BARTScore++o compare in the experiments.
BARTSScore = Wy log (V
CTC (Compression, Transduction, and Creation) (Deng et al., 2021), a suite metrics that unifies different perspectives of different tasks (e.g. summarization, style transfer, text rewriting) into information alignment, which measures weather the information in one generation component is grounded in another. The information alignment is defined as follows: let (c.g, dialog context) be the source input, (c.g., external world knowledge) be some additional context, and y be the generated output text (c.g., generated response). The ali alignment measured on token level and it is measured as the vector of scores:
align(a + b) (a1,'.. , ,ON)
(2)
where each score o; indicates confidence that the z-th token aligns with the whole sentence b. Using the information alignment they define list of metries evaluate text for different tasks. In experiments use two these metries that are closer to ROSCOE: the Relevance (CTC Relevance), which measures the consistency of the generated text with the source and its balanced between the reference, and the Consistency (CTC Consistency) which deals with the faithfullness of the generated text to the input context by the alignment between the two.
19

---

Published as conference paper ICLR 2023
E EXPERIMENTAL SETUP DETAILS (CONT. FROM 5)
E.I DIAGNOSTIC DATASETS
In the following we present details of each diagnostics dataset used in work. Table 11 illustrates how each dataset is used in our experiments. StrategyQA QA dataset is only used to finetune the SimCSE embeddings model, because contains reference reasoning chains in train and validation partitions, but not in the partition. The rest of the six diagnostic datasets are used for sentence embedding model finetuning, and evaluating our models presented in the experiments results. All datasets with examples are summarised Table 12.
Table 11: Summary of datasets used in our work. Reasoning Chain represent whether it contains human written golden step-wise reasoning explanation. Type indicates whether used for constructing Diagnostic Human judged datasets. Train/Val. indicate whether the dataset is used for training, validation and/or testing. StrategyQA dataset only used for finetuning SimCSE embedding model.
Dataset
Reasoning Chain
Type
Train
Val.
Test
Annotated
Instances
EntailmentBank (Dalvi al., 2021)
Diagnostic, Finetuning
1,840
ProofWriter (Tafjord et al., 2021)
Diagnostic, Finetuning
/
272,430
MATH (Hendrycks et al., 2021)
Diagnostic, Finetuning
/
12,500
ASDIV (Miao et al., 2020)
Diagnostic, Finetuming
2,305
AQUA (Liang et al., 2018)
Diagnostic, Finetuning
97,975
FQASC (Aggarwal et al., 2021)
Diagnostic, Finetuming
9,060
StrategyQA (Geva et al., 2021)
Finetuning
2,290
DROP (Dua al., 2019)
Human judged
X
X
210
X
GSM8K (Cobbe et al., 2021)
Human judged
200
X
ESNLI (Camburu et al., 2018)
Human judged
X
149
CosmosQA (Huang et al., 2019)
X
Human judged
X
X
179
SemEval (Ostermann et al., 2018)
Human judged
X
X
209
EntailmentBank (EntBank) (Dalvi et al., 2021) is a complex question answering dataset which contains ste entailment trees, namely a tree of multi-premise entailment steps from facts that are known, through intermediate conclusions to hypothesis of interest (which in this case the question and answer).
ProofWriter (Tafjord et al., 2021) question answering dataset for logical reasoning. contains 500k questions, answers and proofs over natural- natural-language rulebases. This dataset is mostly used to emulate reason- ing over rules expressed in language, including proof generation. The datasets proofs include intermediate conclusions. In our experiments, used depth-O, depth- 1, 2epth-2, depth-3, 3, and depth-5 OWA sets.
MATH (Hendrycks et al., 2021) is a dataset 12,500 problems from high school math competitions. Given math problem such in Table 12 models generate sequence, such as 3. that encodes the final answer.
ASDIV (Miao et al., 2020) (Academia Sinica Diverse MWP Dataset) is a dataset of 2,305 questions on diverse math word problem solving. It includes diverse operations such as basic arithmetic aggregative operations (e.g., comparisons, set-operations).
AQUA (Liang al., 2018) is dataset of 100,000 algebraic word problems with step-wise solutions shown below. In the original dataset each question decomposed in four parts, two inputs and two outputs: the description of the problem and question, and the possible (multiple choice) answer options, being the
20

---

Published as a conference paper at ICLR 2023
Table 12:
We show instances from seven of the Diagnostics Datasets here. (Continue from $5).
Dataset
Instance
Reference Step-by-Step Solution
EntBank
Earth a kind of celestial object. Stars appear to move relative to the borizon during the night. A star kind of celestiul object celestial body. The earth rotating on its causes stars to appear move across the sky at night. Apparent motion when an object appears move relative to another object's 's position.
Stepl: earth kind of celestial object position appears
Step2: star kind of colestial object to shift relative celestial body to the horizon. Step3: apparent motion is
an object appears move relative to another object 's position
Step4 Therefore apparent motion of stars is when stars uppear to move relative to earth's position Step5: The earth rotating on its axis causes sturs appear to move across the sky night
Question: How does the appearance of constellation change during the night?
Hypothesis: Solve the following entailment problem: "Earth kind of oclestial object. During the night stars appeur to move"
Step6: Therefore the earth rotating on its axis causes apparent
motion of
Step7: Stars appeur to move relative the horizon during the night
Step8: Therefore the earth rotating on its axis causes sturs move relative to the horizon during the night.
ProofWriter
Facts: The cow is not big. The cow is not green. The lion cats the tiger. The lion sees the cow. The lion visits the cow, The lion does not visit the squirel. the lion visits the tiger. The squireel is big. The squirrel is round. The tiger not green. The tiger does not see the cow,
Step1: The squirrel is round.
Step2: something is round then it cats the cow.
Step3: The squirrel cats the
Step4: If something sees the squireel and the squireel cats the
cow then the cow round.
Rules: if something sees the squirrel and the squireel cats the cow then the cow is round. if something is green then it cats tiger. the squirel is round then squirrel visits the il something cats the then sees the squirrel. something sees the tiger und the tiger visits the squirrel then nice. something mund then eats the cow. if something is kind then it cats the cow. if the tiger visits the cow then the cow sees the squirrel. if something sees cow then the cow cats tiger. Question:
Step5: The cow is round.
Step6: If something is round then it cats the cow.
Step7: The cow eats the cow.
Step8: if something eats the cow then it sees the squirrel.
Stepp; the cow sees the squirrel.
Answer: True
The cow does not see the squirrel.
MATH
Context: Tom has red marble, green marble, a blue marble, and three identical yellow marbles.
Step1: There are two cases here:
Step2: either Tom chooses two yellow marbles (1 result), or he
Question: How many different groups of marbles can Tom choose?
chooses two marbles of different colors (() (1)== results.).
Step3: The total number of distinct pairs of marbles Tom choose is + 7.
Answer: 7
ASDIV
Context: A sandwich is priced $0.75. cup of pudding priced at 50.25. Tim bought 2 sandwiches and 4 cups of pudding.
0.75 2 0.25 2.5
Answer: 2.5
Question: How much money should Tim pay?
AQUA
The entrance fee for fair $5 for persons under the age of 18 and 20% more for persons older. Fach ride at the fair costs 50,50. Joe goes with her years old twin brothers, and they each took rides total. Question: How much money does Joe end up spending the fair?
Context:
Step1: Total entrance fee is (2*55)+(1 (1=20*)))= $16
Step2: Total rides fee is (0,50* + 3) + 3 == $4.50
Step3: Total money spent $20.50
Answer: 20.5
EQASC
Question: Where water likely to form beads?
Stepl: Beads of water are formed by water vapor condensing
Step2: Moisture builds up in condenses air and the whenever the surfaces are cold.
Answer: Water beads form on cold surfaces,
StrategyQA QA
Question: Are more people today related Ginghis Khan than Julius Caesar?
Stepl: Julius Caesar three children.
Step2: Genghis Khan had sixteen children.
Step3: Modern geneticists have deternined that out of every 200 today has DNA that be traced to Genghis Khan. Answer: True
correct one. In this work we only used the context and question, the step-wise solution and the correct answer construct our diagnostic dataset.
EQASC (Aggarwal et al., 2021) is a multi-hop question answering dataset with 98K explanation annotations for multi-step factual reasoning. Each instance in the dataset comes with question, multiple answer choices,
21

---

Published as a conference paper ICLR 2023
explanation of each answer choice and free flow explanation of the whole context. In our experiments we used the correct answer's explanation to construct our diagnostic datasets.
Strategy QA (Geva et al., 2021) is another multi-step question answering (QA) dataset, that covers a diverse set of reasoning skills. StrategyQA consists of 2, 2,780 questions, annotated with their decomposition and per-step evidence.
E.2 HUMAN JUDGED DATASET CONSTRUCTION
In the following we present details of each human judged datasets used in our work. Table 11 lists each dataset and illustrates how each dataset used our experiments. Specifically, all six datasets are used for evaluations in the experiments results and model finetuning, and one dataset was used for finetuning only, The dataset details are explained below.
To construct these datasets, we first sample instances from each dataset (see the number of instances sampled Table 11). We use GPT-3 with few-shot in-context examples and a prompt to generate step-by-step reasoning (e.g.. "explain step-by-step") for each sampled instance (see in-context examples and prompts in App. B). Then, using our taxonomy we constructed a list of evaluation perspectives to label the model generated step-by-step reasoning step of each of these datasets. We explain the details of the perspectives used to label human judged datasets in 3 5 and App. F. All datasets with examples are summarised in in Table 13. In the following we present details of each human judged datasets.
DROP (Dua et al., 2019), Discrete Reasoning Over the content of Paragraphs, a dataset of 96K of instances with context and question. To solve the tasks, system must resolve references in the context that match with the question, and perform discrete operations over them (such as addition, counting, or sorting). These operations require comprehensive understanding of the content of the input context.
GSM8K (Cobbe et al., 2021) is a dataset of 8.5K linguistically diverse grade school math word problems. On this dataset, even the largest transformer models fail achieve high test performance, despite the conceptual simplicity of this problem distribution.
CosmosQA QA (Huang et al., 2019) is a dataset of 35K problems that require commonsense- based reading comprehension, formulated as multiple-choice questions. The questions focus on reading between the lines over diverse collection of people's everyday narratives, asking such questions as "what might be the possible ..."""" or "what would have happened if..." The dataset does not introduce step-by by--by-tepp reasoning
reason of !.."" output, and contains multiple choice answers.
ESNLI (Camburu et al., 2018) is the extended version of the Stanford Natural Language Inference cor- pus (Bowman al., 2015) of 570K labeled sentence pairs with entailment or contradiction labels. ESNLI includes human labeled explanations of the entailment decision.
SemEVAL (Ostermann et al., 2018) is a dataset on machine comprehension using commonsense knowledge. It contains questions that require commonsense knowledge for finding the correct answer.
E.3 SYNTHETIC DIAGNOSTICS DATASET GENERATION WITH PERTURBATION RULES
To construct the diagnostics datasets we apply synthetic perturbations on half of the chains from six datasets (for details see App. E. and the summary Table 11). Also, in Table 14 we illustrate these synthetic perturbations applied on reasoning steps } of gold reference chains of all the datasets. In there, g" indicates grammar error, which includes changing verb tense, dropping verb, or random word swap. s* represents change the semantics of one step in the chain by replacing named entities. To simulate extrinsic hallucinations, random steps from other chains within the same dataset.
22

---

Published as a conference paper at ICLR 2023
Table 13: We show instances from five of the Human Judged Datasets used in our work. Only GSM8K and ESNLI include human labeled explanations.
Dataset
Instance
Reference Answer & Reference Step-by-Step Solution
GSMSK
Question: Tina buys 12-packs soda for party. Including Tina, 6 people are at the party. Hulf of the people the party have sodas each, 2 of the people have 4, and person has S. How many sodas are left over when the party is over?
Answer: 11
Stepl: Tine buys 12-packs of soda for 3=12=36 sodas
Step2: 6 people attend the party, so half of them is
6/2= 3 people
Step3: Each of those people drinks sodas, so they
drink 3*3=9 sodas.
Step4: Two people drink 4 sodas, which means they drink 294=8 sodus.
Step5: With person drinking 5, that hrings the total drank to 5+9+8+3=25 sodas
Step6: As Tina started off with 36 sodas, that means are 36-25=11 sodas left.
CosmosQA
woman had topped herself jumping off roof of the hospital she had just recently been admitted to. She was there because the first or perhaps latest suicide attempt was unsuccessful. She put her clothes on, folded the hospital gown and made the bod. She walked through the unit unimpeded and took the elevator to the top floor Question: What would have happened to the woman if the staff at the hospital were doing their job properly?
Context:
Answer: The woman would have been stopped before she left to take the elevator to the top floor and she would have lived.
DROP
Context: Denver would retake the lead with kicker Matt Prater nailing 43-yard field goal. yet Carolina ans answered as kicker John Kasay ties the
Answer: John Kasay
game with a 39-yard field goal.
Curolina closed out the half with Kasay nailing 44-yard field goal. In the fourth quarter. Carolina sealed the win with Kasay's is 42-yard field goul.
Question: Which kicker kicked most field goals?
ESNLI
Premise: A child in yellow plastic safety swing laughing as dark-
Answer: neutral
haired woman in pink and coral pants stands hehind her.
Explanation: Child does not imply daughter and woman does not imply mother.
Hypothesis: A young mother playing with her daughter in swing.
SemEVAL
Context: Now am going to set dining table up dinner. First put away all the stuff that not supposed to be on the table, Next I clean the tuble with a tissue paper. Then arrunge some of the decorations on the table. After that put down the plates and glasses. lastly spaces on the table put down what are going to cut but I dont put down dessert yet. There is one more thing I do hefore am finished setting up the dining table. sweat of my chin and forehead. It was hard hard work but still it was much fun. Oh no my family is done with dinner and now have to bring them dessert which fruit I made myself.
Answer:
remaining
my family down for dinner. swept a drop
Question: When did they clean the dining table? Hypothesis: After was set.
To construct diagnostic data from math datasets, we introduce four additional perturbations to simulate step- wise explanation errors that might arise in arithmetic reasoning task (Arithmetic error), general knowledge about relationships and equation construction (Common sense error), and misinformation about object/subject characteristics (Factuality or Hallucination):
Shuffle numbers: randomly shuffles all numbers in the chain,
Shuffle operations: randomly shuffles all math operations in the chain,
Random number: randomly replaces one number in the chain,
Random operation: randomly replaces one math operation in the chain.
23

---

Published as a conference paper ICLR 2023
Table 14: Synthetic perturbations and corresponding error types of steps {r.} in reference chains used when constructing diagnostics datasets. 9 represents grammar error, s (+) represents semantic change.
Perturbation Type
Error Type
Reference Reasoning Steps
Hypothesis Reasoning Steps
Repeat step
Repetition Missing step
73,
12.77.
Remove a
T2,
r2,
Shuffle steps
Self-coherency
71,
13.
Swap a step
Self-coherency
73,
12,
Negate step
Factuality
r3,
[1, (1.
Hallucination
Hallucination
71, r3.
Grammar error
Grummatical
21, (ra)
Semantic change
Factuality
(128)
24

---

Published as a conference paper at ICLR 2023
F HUMAN ANNOTATIONS (CONT. FROM å 5)
To construct Human Judged Datasets, we perform human annotations on five datasets which we summarize Table 11 (Type="Human judged'). These datasets do not include explanations (except GSM8K and ESNLD, so we construct model generated reasoning steps and label them with reasoning errors. We explain generation process in $5 and App. E.2. We used five expert human annotators to collect reasoning error labels on five datasets. We asked human evaluators to directly rate the generated reasoning errors overall chain level using a Likert scale from 1 to 5. We also asked them to mark whether each error type proposed in our error taxonomy ($3) appeared in each step in step-level evaluations. In Fig. and Fig. 5 we illustrate the UI used to collect the data. Table 15 summarizes questions that experts were asked. Table 16 reports the distribution of errors for each dataset. In general, we found that it was hard to get anonymous crowd workers to annotate our data accurately even when we paid averages of upwards of $30 an hour, hence relying on expert annotators. For the annotation sessions reported in the text of the paper, we find that it takes an average of 754 seconds for expert annotators to complete a session of at most 5 examples, or slightly over a-al-a- minutes per example. This highlights the difficulty of obtaining high-quality annotations on these cognitive challenging tasks.
Hove sttp-by-- stop generation??
conntot prode shruping diiht
tae ihst reddt sio bo
sys oinan redecte anter cider eiee cund oing oik dee afoee iimund for lor lighng
le alonna comToT 'ouddor ane "Dead
saperaa
be thonea SUNEANIEE sepinods selris 1OL shid sornc'
sakee sal iduut STEPS cdl new diveaCriD essa
Cruater wes (hersi and Culen asa yoer wall chose te
wen de th Aepping lout werer the
Gas Phe Phenec Secornene Reupy
Saailllll of callens malicnattle
Figure 4: Screenshot of expert annotation user interface, showing the context for the initial question well as the questions regarding the generated response.
Jub0s seap Thwoenouer Sinalinane, bacaase the quectio adding? they cieseed the kitchee and the premiie only ttit aboat dleating up soy..
uncoeveniinal commoontti granerr cagel other noets, desc linguags
tne uncaumi?
intorartio than on cnuth conesc le ofll lingly yarling shout the cane contaced (bs Chanscweentss of obbeccs as anera, exned ort ditn charged
0yye Dyme
legical deducdee arore iis makes caneciation se cnry proeibrattty atyy strnd daarar recluding deunee
cop makis concluddoo with livinng enough support the conduatorr
information mealind
asted casale being fectaal consitters
corsec??
intoraatinn secaree to anater the quettion antedt seeahe bikig tictaal contrsent uith the contect
Oyy
intornation, possibly mesic trreed, aboudy ie preflct the sand this couldbe areaped with out bepacting conscment?
Cyys
er
worts cuneet lnoletion thee losuss propegated
encinet sppiiiiiyy
astion erent
should
JUDGE Step 2- down mentee anything bre about sttcher, so we can mssore that they die dhein the sdechers
loutt uncorvaniinal controveettl prommar s8p97 noeds
step uneat und'
Owm
Cit
intor retomrte than cant ond the contast with le lanoe/ tariing about the sime cor coraccal IDs Che cherecterbita of nar ent obrects ane avera named or thtn chinged makes concluterr with hovine enough suppor concuators
Ovm
loatcal deductee emar iis makes coneiaaion see prpenituulyy sraned dasses recuciig dhum
Dvmm
information sesared to
foe costte being
compterrt
snteraaton eealree in senver it- reeenee vide momple biinng remeaaee with
mudirrmane die essenr de whe alt only hodk carhee fooe
Figure 5: Screenshot of expert annotation user interface, showing questions asked for each step, using the question in Fig 4. The questions are asked every step generated by the model, with steps separated by sentence-ending periods.
25

---

Published as conference paper ICLR 2023
Table 15: Evaluation perspectives used to Human Judged the datasets. The perspectives, which we used to ask humans to label, align with our laxonomy of reasoning enror.. (Continued from $ 5)
Level
Evaluation
Label
Details
Perspective
Overall
QUAL
Overall quality
[1-5]
Does the generated response answer the question in well-justified manner? (I-incomprehensible and wrong, S=clear and correct)
Overall
COH
Coherency
[1-5] Does the whole generated response make sense? (Ie, does it sound indorstandablehoon-ootaaaa aataa-ootttaaccttosssssiette even if fails to address the context?) (I=sounds nonsense. 5measy to parse).
Step
MISS
Missing Step
Y/N
Is the reasoning the generated response incomplete and lacking required information to produce the correct answer? Specifically, does this response contains steps that, added in, would make well- supported chain?
Step
GRAM
Grammar
Y/N
Does this step contain faulty, unconventional, controversial grammar usage? In other words, does the language in this sounds unnatural?
Step
FACT
Factuality
Y/N
Does this step contain information that contradicts the context while still largely talking about the same concepts? (Ex. Characteristics named objects are wrong, named entities changed.)
Step
LOGIC
Coherency Logic
Does step any logical deduction errors (le. makes conclusion contradictory previously stated clauses, including clauses within this step itself: makes conclusion while having enough support make the conclusion)
Step
HALL
Hallucination
Y/N
Does this step contain information not provided in the problem state- ment that is irrelevant or wrong?
Step
RED
Redundancy
Does this step contain information not required answer question asked despite being factual consistent with context?
Step
REP
Repetition
YNN
Does this step contain any information, possibly paraphrased, already mentioned in previous step (and thus could be dropped without impact- ing correctness)?
Step
COMMON
Commonsense
Y/N
Does this step contain any crrors in relation to general knowledge about the world (i.e. how to compute velocity, how many inches in one foot, etc) not explicitly provided in the context!
Step
MATH
Arithmetic
YN
Does this step contain math equation emrors? that you should consider only current step isolation, rather than issues propagated from prior steps.
Table 16:
Statisties of types of errors in Human Judged datasets. Each column reports the number of examples where the specified error type exists. (Continue from $ 5)
Error Type
DROP GSM8K ESNLI COSMOS SemEVAL
Grammar
8
4
8
Factuality
19
56
15
44
31
Hallucination
4
8
4
9
Redundancy
25
13
14
15
19
Repetition
2
3
Missing Step
109
81
40
99
67
Coherency
20
57
17
48
17
Commonsense
58
18
Arithmetic
2
7
26

---

Published as conference paper ICLR 2023
G SENTENCE EMBEDDING MODEL TRAINING (CONT. FROM $6)
Model training. We use the train portions of the perturbed diagnostics datasets finetune the SimCSE embeddings model (explained
5) and validation portions to select the best embedding model. The test portions used to evaluate metrics against bascline metrics. We randomly select 500,000 samples with replacement from each dataset to create uniform representation and reduce bias.
The hyperparameters used to finetune SimCSE model are described in Table 17. We use NVIDIA Tesla V100 Volta GPU instances with 32GB Graphics Card. We perform hyperparameter search, varying batch size in f32, 64, 256, 512, 1024, 2048), learning rate in f5e-06, 1e-05, 5c-05, le-04), and max sequence length in (64, 128, 512). Not all combinations of batch size and max sequence length were explored due to memory limitations.
Table 17: Hyperparameters used to fine-tune SimCSE model perturbed datasets.
Parameter
Value
Batch size
64
Max sequence length
512
Training epochs
Learning rate
5c-6
Temperature
0.05
Validation. We replace original validation procedure on semantic textual similarity tasks with similarity- based validation perturbed reasoning chains. In particular, during training, select best checkpoint that maximizes cosine similarity between positive and minimizes cosine similarity between hard-negative pairs within the batch of size B as the following:
Σ [cos(s,, (s,, rı) (s, h4)
(3)
2 B
Model is evaluated every 100 steps on the development dataset and the best checkpoint is applied at the inference, Other parameters not described this section are kept as in the original SimCSE model used for initialization.
Inference. We compare ROSCOE scores calculated against three embeddings: finetuned SimCSE model, sup-simese-rrbertt-bbee SimCSE model, and all-mpnet-buse-22 sentence embedding model (Reimers & Gurevych, 2019). During inference, we set the random seed 42, Without this, the embedding- based scores naturally varied by about 0.01.
27

---

Published as a conference paper ICLR 2023
H ADDITIONAL EXPERIMENTAL RESULTS (CONT. FROM {6)
H.1 CONTROLLED EXPERIMENTS WITH DIAGNOSTICS DATASETS
In this section, we presented Somers' D correlation of all metrics all Diagnostics datasets. Table 18 sum- marizes the evaluations when investigated reference-free One of the characteristics of our ROSCOR metrics is that, they can provide judgement of the model generated reasoning steps with and without the human reference reasoning chains. In the experiments section in $6, we discussed the results of our unsupervised scores in comparison to baseline scores when measured reference-free. In Table 19, we summarize the correlation analysis on ROSCOE metrics in comparison to baselines on diagnostic datasets when reference is present for evaluation. Specifically, each score is measured between the human provided reasoning steps (reference) and the model generated reasoning steps (hypothesis). We also display fine-grained meta-evaluations of all metrics on each diagnostics dataset separate tables. Specifically, Tables 20, 26 for EQASC, Tables 21, 27 for EntailmentBank, Tables 22, 28 for MATH, Tables 23, 29 for ProofWriter, Tables 24, 30 for ASDIV, and Tables 25, 31 for AQUA.
To understand if designed reference-free scores capture targeted error types we analyze perturbation- correlations summarized in Fig. 6. Out of the all considered scores, Info-Chain is able to cover 10 out of 12 of errors, except Remove Step and Semantic error perturbations. In general we can note that ROSCOE fails to consistently identify missing step error type represented by Remove Step perturbation across different datasets, while other synthesized error types are covered by at least one score type.
seference-based scores are covering all synthetic errors, with Semantic Coverage Chain showing strong correlations with all types of perturbations (Table 19). We also note that along with ROSCCE scores, the highest correlation among all reference-based scores belong to ROUGE and BERT scores (Tables 26-31). ROUGE scores consistently outperform on Repetition, Hallucination, Remove Step. Shuffle Steps, Swap Steps, Negate Step, and Semantic perturbations, while under performing on Random operation, and Shuffle operations. We attribute this to the fact that ROUGE is an n n-gram based score, so it is better in catching errors were wording has significantly changed, while failing to catch small changes within steps.
It worth noting that some scores, especially those among reference-based evaluations, get the highest possible Somers' D correlation scores of 1.0. What it means is that in some scenarios, there is a perfect correlation between the metric and the error type. In other words, for this metric we can find a threshold such generated chains that have scores greater than the threshold do not have errors of the given type, and in all generated chains with scores less than the threshold have that error. It is especially evident on referenced- metrics that directly compare the reference solution and hypothesis. In this scenario, we build correlation for two groups: 1) non- perturbed hypothesis: the score is calculated by comparing embedding similarities of the reference with itself, and we expect get high scores, 2) perturbed hypothesis: comparing reference with its perturbed version, where the scores should be lower. In some cases, we are able to perfectly separate perturbed and con-perturbed chains based on the corresponding metric values by selecting threshold, in other cases we cannot due to a number of false-negatives (i.c., a chain gets high score, although the error is Chhinn present). As an example, consider the Semantic Coverage- Chain metric calculated on EQASC dataset using all-mpnet-bass baseev2 v2 sentence embeddings, and Hallucination perturbation (Table 26). Here the Somers' D correlation score 1.0. Semantic Coverage-Chard Chain calculated as normalized cosine distance between the chain embedding of the reference solution r, and the chain embedding of the hypothesis h. [1 cos(r. h)] Recall that in our setup, half of the hypothesis chains are perturbed reference chains, and another half is the same as the reference. While Hallucination perturbation is insertion of random step from dataset, it is hard to predict how if will affect the embedding of the chain as a whole, but on the unperturbed chains, where h = r, the Semantic Coverage-Chall should be: [1 + cos(r, r)]/2 1.0. Further review confirmed that in this dataset there are no false-positive instances, i.e., all chains with perturbations had Semantic Coverage-Chun score less than 1.0. That means, we can always identify if the chain contains Hallucination
28

---

Published as conference paper ICLR 2023
eπrоr not, by comparing Semantic Coverage- Chain value with 1.0 (threshold value), which is reflected in perfect Somers' D score.
Highest correlations among reference-free free scores belong the Repetition- * scores, that exhibit perfect correlation on EQASC dataset (Tables 20-25). For other datasets, non non-perfect correlations can be attributed to the small number of false-negan nega-nne,,,, i.e. they give low Repetition- scores for chains with non-duplicated but similar steps, while all chains with duplicates got almost () scores (Fig. 7). In EQASC explanations are created from a set of facts that are not directly related each other, but are intended to give an answer when combined together. Among all datasets considered, these steps most dissimilar, and thus can be separated with similarity- based scores.
1.0
Repetition
Swap Step
Hallucination
Shuffle
Random number
Relative Presence
Random operation
Shuffle numbers
0.6
Shuffle operations
Grammar Error
Negate Step
0.4
Semantic Error
0.2
0.0
Falthfflnness Step
Falthffliness Token
Repetition Token
Info-Step
Chain
Source- Consistency
Self cof-Coonsistecyy
Perplexity Step
Perplexity Chain
Grammar
Repetition-Ste
Figure Relative presence of the strong portupeeturbattonn correlation, measured as the number of datasets where for each score- poree-peruubbtionn pair Somers' D correlation value is in the 90th percentile, normalized by the total number of datascts where this type of perturbation occurs. Statistics collected over ROSCOE reference- free scores with finetuned SimCSE embeddings. (Continued from (7)
Score: Repelition Perturbation: Repetition
Repetition
Repelition
(ProorWr, perturbed)
(ProffWrr parturbed) (ProofW.., original) (Math, perturbed)
(ProofWr, original)
(Math, perturbed)
(Math, original)
(Math, original)
perturbed)
(EniBank, perturbed)
(EntBank, original)
(EntBank, original)
IEQASC, parturbed)
(EQASC, parturbad)
(FQASC, original)
(FOASC, original) (AQUA, se)
(AQUA, perturbed)
original)
original)
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Repettion
perturbed
Repetition Word
aemuried
oeig
orginal
Figure Box-ande nox-and-whiske plots of interquartile ranges scores, for Repetition perturbations and Repetition * scores. While all perturbed subsets have ( or near () scores, all datasets except EQASC have some chains that were also scored as low despite the absence duplicates.
29

---

Published as conference paper ICLR 2023
Table 18: Somers' correlation of all metrics on six Diagnosties datasets. All metrics measured referencc-free (s, h). The highest correlation overall for each dataset in bold. second hest models are underlined. Correlations that are not significant (p- up-value >= 0.05) are omitted when aggregating, and denotes an absence of any significant correlations, Note that ASDIV is 1-step equation dataset, so there are no repetition and self-consistency scores as there steps compare. (Continued from 86, details in App. H.I.).
Metric
EntBank
Math
AQUA
ProofWriter EQASC
ASDIV
ROUGE-I
0.410
0.176
0.257
0.095
0.342
0.305
ROUGE-2
0.391
0.151
0.206
0.090
0.217
ROUGE-L
0.365
0.156
0.264
0.106
0.315
0.269
BLEURT
0.257
0.148
0.252
0.024
0.447
BERTScore
0,380
0.124
0.220
0.117
0.462
0.322
BARTScore
0,358
0.185
0.317
0.415
BARTScorc+
0.315
0.164
0.251
0.054
0.297
BoRT'S
0.186
0.128
0.215
0.011
0.276
PRISM
0.453
0.208
0.191
0.235
0,436
CTC Relevancy
0.258
0. 0.188
0.217
0.394
0,485
0,382
CTC Consistency
0.310
0.282
0.157
0,513
0.270
0.396
ROSCOE Metrics (reference-free metries only)
ROSCOE-SA SA with all-mpnet-buse-t sentence embeddings
Faithfulness- Stepp 0.786
0,362
0.152
0.771
0.785
0,186
Faithfulness-
0,581
0.157
0.157
0.480
0,182
Info-Step
0.638
0,231
0.250
0.538
0.198
Repetition- Token
0.913
0.936
0.972
0.596
1,000
ROSCOE-SS with all-mpnet-base-r2 sentence embeddings
Info-Chain
0.419
0.467
0.214
0.082
0.550
0,280
Repetition-Step 0.909
0.932
0.982
0.631
1,000
ROSCOE-SE SA with sup-stmccee-rbertteeeeee sentence embeddings
Faithfulness-Steg
0.777
0.229
0.224
0.732
0.630
0.266
Faithfulness- Token
0.663
0,200
0.517
0.502
0.515
Info-Step
0.560
0.131
0.183
0.226
0.399
0,275
Repetition-Toon Token
0.919
0.939
0.971
0.606
1,000
ROSCOB-S5 with sup-simncs--mberrtbbbue sentence embeddings
Info-Chain
0.524
0.180
0.195
0.045
0.409
0.289
Repetition-Step 0.901
0,949
0.991
0.621
1.000
n/a
ROSCOR-SA with finetuned sup- susp-simss--rbbeee sentence emheddings
Faithfulness-Stey
0.538
0.614
0.826
0.763
0.907
0.879
Faithfulness-Tokee 0.519
0.204
0.285
0.499
0,492
0.740
Info-Step
0,599
0,511
0.703
0.317
0,804
0.879
Repetition-Tokke Token
0.919
0.939
0.971
0.606
1,000
n/a
ROSCOE SS with finetuned sup-simese-robertaa sentence embeddings
Info-Chain
0.955
0.777
0.933
0.462
0.995
0.857
Repetition-Step 0.908
0.924
0.982
0.624
1,000
n/a
ROSCOE-LA LI
Self-Consistency
0.782
0.190
0.368
0.204
0.793
n/a
Source-onsistency
0.917
0.341
0.424
0.289
0.778
0.771
ROSCOE-LC
Perplexity-Step
0.213
0.160
0.110
0.394
Perplexity-Chain
0.151
0.175
0.229
0.135
0.379
0.485
Grammar
0.604
0.392
0.359
0.788
0.859
0.470
30

---

Published as a conference paper ICLR 2023
Table 19: Somers' D correlation of all reference- metrics on six Diagnostics datasets, Metrics are measured using reference generations on (r, f.). The highest correlation overall for each dataset is in bold. The second best models are underlined. (Continued from $6, details App. H.1.)
Metric
EntBank
Math
AQUA
ProofWriter
EQASC
ASDIV
ROUGE-1
1.000
1.000
1.000
1,000
1.000
1.000
ROUGE-2
1.000
1.000
1.000
1.000
1.000
1,000
ROUGE-L
1.000
1.000
1.000
1.000
1.000
1.000
BLEURT
0.821
0.773
0.946
0.829
1.000
0.93
BERTScore
1.000
1.000
1.000
1.000
1.000
1,000
BARTScore
0.983
0.989
0.984
0.986
1.000
0.954
BARTScore+
1.000
1.000
1,000
BARTScore-P
0.877
0.799
0,905
0.595
0.966
0.83
PRISM
0.939
0.521
1.000
0.997
0.996
1.000
CTC Relevancy
0.457
0.592
0.409
0.725
0.954
0.398
CTC Consistency
0.814
0.804
0,833
0.635
0.974
0.6
ROSCOE Metrics (reference-based metrics only)
ROSCOE-SA SA with all-mpnet-base-t sentence embeddings
Hallucination
1.000
1.000
1.000
1.000
1.000
1.000
Redundancy
1.000
1.000
1.000
1.000
1.000
1.000
Semantic Coverage-Step
Reasoning Alignment
1.000
1.000
0.143
1,000
1.000
1,000
Commonsense
0.438
1.000
1.000
0.379
1.000
1.000
Missing Step
0.993
1.000
1.000
0.876
1.000
1.000
ROSCOR-SS SS with all-mpnet-base-22 sentence embeddings
Semantic Coverage-Chair
1.000
1.000
1.000
1,000
1.000
1.000
ROSCOE- SA with sxp-simcse-roberta sentence embeddings
Hallucination
1.000
1.000
1.000
1.000
1.000
1.000
Redundancy
1.000
1.000
1.000
1.000
1.000
1.000
Semantic Coverage-Step
1.000
1.000
1.000
1,000
1.000
Reasoning Alignment
Commonsense
0.433
1.000
1.000
0.415
1.000
1,000
Missing Step
0.999
1.000
1.000
0.874
1.000
1.000
ROSCOE-SS with sup-simcce--obertte sentence embeddings
Semantic Coverage-Chain
1.000
0.999
1.000
1.000
1.000
1,000
ROSCOE-SA with finetuned sup-simcse-roberta sentence embeddings
Hallucination
1.000
1.000
1.000
1.000
1.000
1.000
Redundancy
1.000
1.000
1.000
1.000
1.000
1.000
Semantic Coverage-SHe
1.000
1.000
1.000
1.000
1.000
1.000
Reasoning Alignment
1.000
1.000
1.000
1,000
1.000
1.000
Commonsense
1.000
0.404
Missing Step
0.999
1.000
1.000
0.873
1.000
1,000
ROSCOE- S5 with finetuned sup-simcse-roberta sentence embeddings
Semantic Coverage- Chain
1.000
0.999
1.000
1.000
1.000
1.000
31

---

Published as a conference paper ICLR 2023
Table 20: Somers' D correlations of all metries per different perturbation applied on EQASC Diagnostics datascts. All metrics are measured reference-free (s, h). The highest correlation overall for each dataset is in bold. The second best models are underlined. Correlation scores with p-value < 0.05 are marked with f. (Continued from $6, details App. H.1.)
Perturbations
Repet.
Halluc.
Grammar Error
Remove Step
Negate Step
Semantic Error
Rouge-1
0.264f
0.342f
0.017
0,227f
0.063
-0.023
Rouge-2
0,071
0.205
0.1061
0.2171
0.099
-0.002
Rouge
0.210
0.315
0.057
0.179
0.083
-0.016
BLEURT
0,366
0.447;
-0.028
0.1951
0.2041
-0.1081
BERTScore
0,288f
0.4627
0.153t
0.160f
0.052
0.051
BARTScore
-0.127f
0.415t
BARTScore+
0,028
0.2127
0.055
0.2971
0,023
-0.028
BARTScore-P
-0.039
0.031
-0.038
0,276
+0.003
-0.023
PRISM
-0.327
0.436
0.267
0.077
0.010
0.123
CTC Relevancy
0.141f
0.001
0.0821
0,485)
0,002
0.220
CTC- CCCCCosistency
0.001
-0,080t
0.0951
-0. 154
0.0781
0,270f
ROSCOE Metries (reference--ree
h))
8OSCO2-SA with walll-mpnt--buseee senience embeedings
Faithfulness Step
0.785
0.017
0.0841
0.2431
Thithfulness-ooke
-0.031
0,480+
-0.001
0.037
0.008
-0.156t
Info-Step
0,006
0,5381
-0.003
0,223t
0.0851
-0.191f
Repetition-Toker
1.000
0.399;
-0,028
-1.000
0.070
0.074
RDSCOE-SS with all mpnet base v2 sentence embeddings
0,189;
0,550;
-0.022
0.220f
-0.132
Repetition-Step
1.000F
0.035
-0.034
-1.000
-0.1207
0.030
RDSCOR-SA with sup-simese simese- rherta have sentence emheddings
Faithfulness-Step
-0.061
0.630t
-0.010
0.007
0.308t
-0.204f
Faithfulness- Token
-0.031
0.502
0.032
0.045
0.107f
-0.1681
Info-Step
-0.044
0.399
40.024
Repetition Taken
1.000
0.148
0.138
-1,000
.0.055
0.080
R08608-85 with sup- cep-ssinse-rrbeeaae base sentence embeddings
Info-Chain
-00025
0,409:
-0.028
0.256t
0.379t
-0.163f
Repetition-Seep
1.000
0.001
-0.003
-1.000
-0.463
0.071
308008 SA with finetuned wog-ssincee-rbberttoee sentence embeddings
Faithfulness-Stpp
-0.044
0.630;
0.2931
-0.046
0.907f
-0.1181
Faithlulness: Token
-0.019
0.485
0.141
0.036
0.492
0.128
Info-Stepp
00041
0.383
0,2661
0.1961
0.804
0.068
Repetition-
1.000F
0.1487
-0.1387
-1.000
-0.055
-00800
RDSCOE-35 linetuned sup
roberta sentence embeddings
Info-Chain
0.995
0,871
0.5881
0.1031
0.967
0.1211
Repetition-Sleg
1.000
0.037
0.386
-1.000]
00.533
00.48
30SCOE-L.
Source Consistency
-0,020
0.576
0.1121
-0.239
0.778
0,268
Self Consistency
0.633;
0.399
.0.133
0.7931
0.4761
XOSCOE-L2
Perplesity Chain
0.6901
0.007
0.379
0,260
0.1181
0.269
Perplexity-
0.937f
0.965;
0.3521
-0.9531
0.0811
0.225t
Grammar
-0.025
0.060
0.8591
-0.145F
0.139f
0.722
32

---

Published as a conference paper ICLR 2023
Table 21: Somers' correlations all metrics per different perturbation applied on Entailment Bank Diagnostics datasets. All metrics are measured reference-free on (s, h). The highest correlation overall for each dataset is in bold. The second best models are underlined. Correlation scores with p-value < 0.05 are marked with f. (Continued from 86, more details in App. H.1.)
Perturbations +
Repet.
Hallue.
Grammar Error
Remove Step
Shuffle Steps
Swap Steps
Negate Step
Semantic Error
Rouge-1
0.391
0,410f
0.074
0.081
0.113
0.065
0.034
-0.017
Rouge-2
0.3561
0.391
0.1607
0.116
0.109
0.088
0.109
0.091
Rouge-L. L
0.209
0.194]
0.112
0.138)
0.365
0.229]
0.003
0.063
BLEURT
0.025
0.164f
0.060
0,231
0.092
0.096
0.2577
-0.036
BERTScore
0.264
0,380]
0.211
0.150
0.364]
0.205
0.050
0,087
BARTScore
0.034
0,063
0.106
0.358
0.248;
0.183
0.1427
0.1641
BARTScore+
0.101
0.047
0.036
0.315f
0,184t
0.155t
0.1307
0,1731
BARTScore-P
0.061
0.012
0.025
0,186f
0.041
-0.008
0.098
0.100
PRISM
0.230
0.453|
0.279:
0.118
0.331
0.1671
0.081
0.1481
CTC. Relevancy
0.258
0.057
0.026
0.080
0.020
0.035
0.029
0.159
CTC-Consistency
0.310
0.159
-0.058
-0,249
-0.1621
-0.023
-0.046
0.194|
ROSCOE Metrics (reference-free on (s. h))
ROSCCE-SA with dlll-npet--base-222 sentence embeddings
Faithfulness Step
0.023
0.786]
0.231;
0.005
0.111
-0.000
0.098
0.121
Faithfulness-Tole 0.098
0.581f
0.250
0.087
0.009
-0.020
0.179;
0.1701
Ineo-Step
0.083
0.638
0.181
0,1611
0.025
-0.001
0.216
0.139
Repetition
0.913f
0.105
0.042
-0.177
0.139F
0.038
-0.085
.0.058
ROSCCE-SA with mpneet buse sentence embedklings
Faithfulness- Step
0.023
0.786t
0.231+
-0.05
0.111
.0.000
0.098
0.121
Faithfullees--oon
0.098
0.581t
0.2507
0.087
0.009
-0.020
0.179t
0,170
Info-Stop
0.083
0.638f
0.1811
0,161
0.025
-0.001
0.216
0,139t
Repetition-Tokee Token
0.913
0.105
0.042
-0.177
0.139
0.038
-0.085
-0.058
ROSCCE with all-mppet--uss--22 sentence embeddings
Info-Chain
0.039
0.419f
0.083
0.071
0.068
0.025
0.040
0.037
Repetition-Step 0.909f
0.148f
-0.061
-0.165t
0.067
-0.068
-0.062
-0.056
ROSCCE- SA with sup-simcce- www.siineee-mbbr---ueee sentence
embeddings
Faithfulness-Sep
-0.002
0.777f
0.158+
-0.015
0.107
0.009
0.358
0.110
Faithfulness-okad
0.063
0.6631
0,280
0.084
0.046
0.024
0.279;
0.159f
Info-Step
0.066
0.560]
0.079
0.148:
-0.035
0.007
0.450
0,117
Repetition Token
0.919
0.1371
0,010
0.216
0.171f
0.005
-0.106
-0.099
ROSCOE-SS 98 with www.--immeee--eeeeeeeeee sentence embedkingg
Info Chain
0.084
0.5151
0.005
0.119
0.0199
0.023
0.524
0.048
Repetition
0.901
0.1391
0.006
-0.188
0.019
-0.066
-0.144|
-0.098
ROSCCE SA with finetuned sup-siincce--oberr--bbae sentence embeddings
Fuitafulaess-Stpp
0.105
0.3921
0.2047
0.063
0.093
0.029
0.5387
-0.115
"uibffulnes-Tokee
0.079
0.519
0,271
0.100
0.055
0.029
0.356
0.041
Info-Step
0.1311
0.364
0.227
0.161
0.053
0.047
0.599
-0.018
Repetition-Token
0,919;
0.137f
-0.010
-0,216t
0.171f
0.005
-0.106
-0.099
ROSCOE-SS with finetuned sup simesse mberte huse sentence embeddings
Info-Chain
0.871
0.851t
0.7527
0.437=
0.937F
0.672t
0.955
0,3001
Repetition-Stp
0.908
0,133t
-0.013
-0.169t
0.135t
-0.043
-0.058
-0.079
ROSCOB-11
Source Consistency
-0.44
0.289|
0.218)
0.052
0.150
-0.075
0.860)
0.257|
Self-Consistency
-0.40
0.403
0.216r
-0.042
0.129t
-0.053
0.7827
0,170
ROSCCE-LC
Perplexity Chain
0.364
0.104
0.116
0,151
-0.006
0.085
0.060
0.133|
Perploxity-Step
0.1991
0.168t
0.075
-0.097
0.143f
-0.021
-0.019
-0.046
Grammar
-0.109
0.076
0.604
0.075
0.044
-0.063
0.033
0.3651
33

---

Published as a conference paper ICLR 2023
Table 22: Somers' D correlations of all metrics per different perturbation applicd on MATH Diagnostics datasets. All metrics measured reference-free (s, h). The highest correlation overall for each dataset is in bold. The second best models are underlined. Correlation scores with p-value < 0.05 are marked with f. (Continued from 86, more details App. H.1.)
Perturbations
Repet.
Hallue,
Grammar Error
Remove
Shufflc
Swap Steps
Negate Step
Random Number
Random Operation
Shuffle Numbers
Shuftle Operations
Step
Steps
Rouge-1
0,176t
0.151f
-0.004
0.0651
-0.020
0.020
0.028
0.017
0.008
0.022
0.011
Rouge-2
0.1071
0.095
0.006
0.121
-0.005
0.029
0.010
0.018
-0.011
0.151
-0.002
Rouge- L.
0.126]
0,156
0.004
0.075
0.008
0.042|
0.023
0.018
0.008
0.082
0.004
BI JURT
0.143t
0,148f
0.000
-0.023
0.001
0.012
0.049;
-0,000
-0.005
-0.036f
0.002
BERTScore
0.124f
0.117f
0.067f
0.089F
0.025
0.029
0.010
0.029
0.016
0.034f
0.017
BARTScore
-0.048F
-0.066
0.015
0.185
-0.026
-0.002
0.029
0.009
0.0751
0.045
BARTScore+
0.015
-0.003
0.0491
0. 062
0.003
0.031
0.002
0.0507
0,047f
0. 0,1644
0.063t
BARTScore-P
-0,005
0,002
0.022
0.1287
0.006
0.011
-0,009
0.024
0.002
0.115f
0,059f
PRISM
-0.115
0.208
0.120]
0.095
0.029
0.017
-0.003
0.102:
0.069
0.117
0.111
CTC- Relevancy
0.104
0.041F
0.13
0.188
0.027
0.029
0.018
0.043:
0.052
0.052
0.001
CTC-Consistency
-0.1067
0,145t
0.2821
0.105t
0.096f
0.035t
0.0801
0.079F
0.033
0.046f
-0.017
ROSCOE Metries (reference-free (s, h))
ROSCOE-SA with all-mpner- bheee--ueeee sentence embbeddnngs
Faithfulness. Step
Faithfulness-Tokee
0.000
0.029
0,1
0,362;
0.016
0.000
0.070:
0.004
.0.044
-0.044
-0.003
0.009
0.018
0.060;
0.028
0.0481
0.005
0005
-0,023
0.0741
0.060
0.003
Info-Step
0.024
0,231f
0.022
0.1187
-0.018
0.009
0.0331
0,0851
0,1091
0,106t
Repetition-Tokee
0.9361
0.069F
-0.018
-0.078f
0.010
0.043f
0.006
41004-
-0.08
0.035t
0.006
ROSCOE-SS with all mpner have v2 sentence embeddings
Info-Chain
0.0591
0.467f
0.016
0.172
0,106
0.091f
0,0631
0,192
0.0891
0
0.135t
Repetition-Step
0.932f
-0.002
-0,036f
-0.114f
-0.001
0.013
-0.026
-0.134f
-0.045t
-0,008
-0,030
ROSCOR-SA with sup-simeee- -we----bbaaaah base sentence embedddings
Faithfulness-Step
-0.004
0.229+
0.049;
0.0467
0.007
0.010
0,1424
0E730
0.033
0,031
-0,010
Faithfulness-Tokee
0.014
0,200F
0.091
-0.004
-0.001
0.009
0.021
00030
0.005
0.021
0.008
Info-Step
+0.020
0.115
0.048
0.092
0.009
0.020
0.131
0,066
0.029
0.042]
Repetition Token
0.939|
0.007
0.093|
0.073t
0.023
0.037|
0.002
0.058
.0.035
0.019
-0.002
ROSCOF-S5 with sop-timese-mbeetateee hase sentence embeddings
Info-Chain
0.028
0.114f
0.038t
0.0717
0.034
0.004
0,180f
0,078F
0.0511
0.0411
-0.007
Repetition-Step
0.949;
0.019
-0.043
-0.094|
-0.012
0.000
-0.0621
-0.119
0.014
0.027
-0.001
RO830E SA with finetuned wwp-ssimcee-vobertt-buue sentence embeddings
Fathhfulnes-Step
0.021
0.223
0.190
0.0387
-0.031
0.005
0.614f
0.376
0,415t
0.277f
0,428f
Faithfulness Token
0.016
0,204
0.106]
-0.005
-0.044
0.008
0.084
0.067:
0.047
0,052
Info-Step
0.012
0.133
0.228
0.0991
.0.021
0.020
0.5111
0.451
0,4511
0.3011
Repetition-Token
0.9397
0.007
-0,93f
-0.033
0.023
0.037f
0.002
-0.0584
-0.035t
0.019
-0,00
KOSCOE-SS with linetuned sup simese
bose sentence embeddings
Info-Chain
0.679
0.588
0.6941
0.216
0.746
0,5301
0.777
0.698
0.757
0.524
0.662
Repetittion Step
0.924
0.024
0.059
0.083
.0.019
0.012
-0.052
-0.121
-0.103
00.11
0.017
ROSCOE-11
Source Consistency
0.011
0.071
0.044
0.041
0.002
0.026
0.215
0.223
0.123
0.341
0.133|
Consistency
0.015
0.069;
.0.003
0.105
00.044
0.011
0.1224
0,147
0.0681
0.190
0.0881
ROSCOE-LC
Perplexity Chain
0.358
0.020
0.175
0.1034
0.100
0.0351
0.003
0.173
0.1091
0.154;
0.170
Perplexity-Step
0.1601
0.1
0.001
-0.093t
-0.016
0.019
0.006
0.003
0.012
0.0481
0.037
Grammar
0.010
0.026
0.392F
-0.020
-0.012
0.005
0.112f
0.034F
0.026
0.057f
0.040
34

---

Published as a conference paper ICLR 2023
Table 23: Somers' D correlations of all metrics per different perturhation applied on ProofWriter Diagnostics datasets. All metrics are measured reference-free on (s, h). The highest correlation overall for each dataset is in bold. The second best models are underlined. Correlation scores with p-value 0.05 are marked with f. (Continued from 86, more details App. H.1.)
Perturbations
Repet.
Halluc.
Grammar Error
Remove Step
Shuffle Steps
Swap Steps
Negate Step
Semantic Error
Rougo-1
-0.124
-0.054t
0.023t
0.0951
0.013
-0.001
0.007
0.006
Rouge-2
40.931
-0.002
0.0681
0.0901
0.018
-0.004
0.037f
0.047:
Rouge
0.120
0.028]
0.029]
0.089
0.106]
0.020
0.020
0.027
BI BLEURT
-0.058t
-0,001
-0.027f
-0.099
0.016
0.007
0.024f
-0,016
BERTScore
-0.0491
0.077f
0.117f
0.0821
-0.064F
-0.0547
0.023f
0.108
BARTScore
-0.059
-0.096f
-0.037-
0.0811
-0.013
-0.010
-0.061f
BARTScore=
-0.055t
-0.067f
0.014
0.054f
-0.011
-0.021
0.006
0.032
BART Score-P
-0.046f
-0,0491
-0.012
0.010
-0,044f
-0.0329
-0.020
0.011
PRISM
-0.159)
0.159
0.222
0.097
0.017
-0.010
0.060]
0.235
CTC Relevancy
0.394
0.392
0.123
0.185
0.131F
-0.052:
0.036f
0.077
CTC-Consisteney 0.496f
0.513f
0.182f
0,2231
-0,063f
-0.022
0.098f
0.131
ROSCOE Metries (reference-free (s. h))
ROSCOR-SA with all mpl--mpee--uue-- -22 sentence embeedinngs
Faithfulnness Step
0.348
0.1651
0.011
0.013
0.233|
Faithfulness-Tokke
0.004
0.436f
0.264
0.055t
0.030f
0.006
0,168f
0.310-
Info-Step
-0.044
0,250
0.121f
-0.062F
0.023
0.015
0,108t
0.174:
Repetition-
Topee Token
0.596t
0.053f
-0.041+
0.1011
-0.006
40.003
+0.055t
-0.0501
ROSCOB-SS with all mpnet buse v2 sentence embhedinngs
Info-Chain
.0.052f
0.083;
0.033
.0.099
0.014
0.001
Repetition-Step
0.631f
0.031f
-0.027+
0.116f
-0.002
0,002
-0.044f
-0.042f
ROSCOR-8A with sup. simese roberta hase sentence emheddings
Faithfulness-Strp
0.004
0.732t
0.171f
0.154t
0.013
0.012
0.481f
0,494=
Faithfulness- Tekknn
0.010
0,517f
0.3347
0.0861
0.031
0.005
0,336f
0.395=
Info-Step
+0.008
0.226]
0.047
0.027
0.172
0.160:
Repetition Token
0.606
0.036
0.065
0.097
0.002
0.009
0.070
.00770
ROSCOR-S5 with sup- wwp-.imncee-mmberta-bee sentence emheddings
Info-Chain
-0.040t
0.004
-0.015
-0,140f
-0.024t
-0.011
0.045t
-0.006
Repetition-Step
0.621f
0.0281
+0.08
0.1151
-0.011
0.001
-0,050
-0.043
RCSCOB SA with finetuned sup-stmccee-bbertt-bbee sentence embeddings
Faithfulness-Stp
0.008
0.7631
0,618t
0.1801
0.008
0.004
0.726
0.536
Faithfulness- Token
0.012
0.499
0.475
0.088
0.036]
0.000
0.436
0.403
Info-Step
-0.018
0.243
0.252
0.046
0.016
0,012
0.3171
0.187
Repetition-Toker
0.606t
0.036F
-00655
0.097f
-0.002
0.009
-0.0701
-0.070
KCSCOR-SS with linetuned xup simese mberia base sentence embeddings
Info-Chain
0.214f
0.248
0,2841
0.0771
0.330
0.2461
0.4621
0.122F
Repetition Step
0.624
0.039
0.034
0.116
0.007
-0.002
0.076
0.034
BOSCOE- LI
Source Connustoonsitccc
0..08
0.027
0.028|
0.010
-0.044|
-0.004
0,289
0.049
Self Consisssitt
0,204
0.084
0.1
-0.022
0..20
0.036
0.065
ROSCOE-L2
Perplexity Chain
0.1651
0.047
0.112
0.064;
0.1351
0.0671
0.012
0.128
Perplexity-Step
0.178t
0.1
0.033f
0.0821
-0.008
0.005
-0.0361
-0,008
Grammar
0.000
0.042f
0.788;
0.1021
0.007
-0.023
0.007
0.515=
35

---

Published as conference paper ICLR 2023
Table 24: Somers' D correlations all metrics per different perturbation applied on ASDIV Diagnostics datasets. All metrics are measured reference-free on (s, h). The highest correlation overall for each datasct is in bold. The second best models are underlined. Correlation scores with p-value < 0.05 are marked with f. (Continued from $6, more details in App. H.I.)
Random
Random Operation
Shuffle Numbers
Shuffle Operations
Perturbations
Number
Rouge-1
0.305+
0.096
0.085
-0.220
Rouge-2
-0.034
-0.014
0.007
-0.038
Rouge-T.
-0,235
BLEURT
0.043
-0.034
0.059
0.015
BERTScore
0.098
0.322f
0.025
0.167
BARTScore
0.107
-0.027
-0.105
0.197
BARTScore+
.0.011
-0.022
.00055
.0.015
BARTScore-P
0.068
0.043
-0.048
0.121
PRISM
-0,009
-0.035
-0.114
0.258
CTC-Relevancy
0.155t
0.000
CTC-Consistency 0.3967
0.189f
0.121
-0.121
ROSCOE Metrics (reference-free on (s. k))
ROSCCE SA with all-mmpner-uss--222 sentence embeddings
FaithfulnessSSep 0,186F
-0.090
0.091
0.091
Faithfulness-Tode Token 0.182
0.080
0.062
-0.091
Info-Step
0.1987
0.085
0.167
ROSCOB 55 with all-mppee--uss--2 sentence embeddings
Info-Chain
0.2801
0.005
0.192
0.091
ROBCGE 5A with wwp..sinccee-meeert--ooee sentence embeddings
Faithfulncss-Stop
0,266
0.082
-0.182f
0.015
Faithfulness- Token
0.273
0.125
0.515
Info Step
0.275
0.125
0.141
0.000
ROSCCE-B5 with sup-sinese-mbberte hose sentence embeddings
Info-Chain
0.2897
0.145t
-0.084
0.030
ROSCOR-SA with finetuned sup-simcse roberta hase sentence emheddings
Faithfulness-Steg
0.6307
0.840f
0.670f
0.879;
Faithfulness- Token
0.576
0.740
0.552
0.545;
Info-Step
0.669
0.844
0.683
0.879
ROSCGE finetuned sup-simcce--eee re-brrrt---eeee
embeddings
Info-Chain
0.773
0.857
0.795
803
ROSCCE LI
Souroe-Comsisencyy
0.760
0.771
0.763
0.500]
Sclf-Consistency
0.203
0.2271
0.206
0.152
ROSCOF-1.C
Peplexity-Chiin
0.300f
0.092
0.214F
0.485+
PesplexitySSep
0.300F
0.092
0.214f
0.4851
Grammar
0.170/
-0.083
-0.007
0.470]
36

---

Published as a conference paper ICLR 2023
Table 25: Somers' D correlations of all metrics per different perturbation applicd on AQUA Diagnostics datasets. All metrics measured reference-free (s, h). The highest correlation overall for each dataset is in bold. The second best models are underlined. Correlation scores with p-oalue < 0.05 are marked with f. (Continued from 86, details App. H.1.)
Perturbations
Repet.
Hallu.
Crammar Error
Remove
Shuftle Steps
Swap Steps
Negate Step
Random Number
Random Operation
Shuftle Numbers
Shuffle Operations
Step
Rouge-1
0.134
0.055
-0.085
0.257f
0.074
-0.032
0.109
0.056
-0.006
0.131
-0.073
Rouge-2
0.180:
0.004
-0.021
0.206
0.100
-0.092
0,044
0.051
0.082
0.1961
-0.061
Rouge L
0.072
0.055
0.059
0.264
0.145
0.008
0.120
0.050
0.004
0.148
-0.037
FURT
0.010
0.047
-0,066
0.2521
-0.003
-0.047
0.071
-0.010
0.020
0.165t
-0.023
BERTScore
0.134
0.111
-0.016
0.166f
-0.011
-0.043
0.028
0.007
0.079
0.220f
-0.036
BARTScore
-0.030
-0,269
0.3171
0.051
0.011
0.169
BARTScore+
0.112
-0.037
0.044
0.251
0.019
-0.053
0,028
0.072
0.066
0.2301
-0,009
BARTScore-P
0.082
-0.097
-0.007
0.2151
0.013
-0.027
0.114
0.059
0,042
0,1821
0.028
PRISM
-0,119
0.026
0.177|
0.188
0.071
0.048
0.057
0.191
0.131
0.182
0.028
CTC Relevancy
0.133
0..01
0.065
0.096
00.068
0.013
0.024
0.119
0..17
0.2171
0.072
CTC-Consistency
0.071
-0.050
0.075
0.041
-0.074
0.012
0.004
0.106
-0.006
0.1571
0.024
ROSCOE Metries (reference-free (s))))
RDSCO2-SA SA with all mpnet (mpper-bbse--2? buse-v2 v2 sentence embedddings
Faithfulneess Step
0.098
0.152
-0.113
0,080
0.057
-0.023
Thithfulness-Tokes
0.125
0.009
0.008
0.157f
0.047
-0.001
0,057
-0.014
0.076
0.070
-0.006
Info-Step
0.088
0.030
-0.020
0.114
0.068
-0.007
0,015
0,063
0.025
0.068
-0.088
Repetition-Token
0.972
0.216f
-0.125
-0,226t
-0.053
-0.061
-0.04
40.004
-0.024
0.037
-0.001
ROSCOE-SS with all mpnet buse v2 sentence embbeddings
Info-Chain
0.066
0.214;
-0.053
0.146t
0.023
-0.035
0.049
0.079
0.1531
Repetition-Step
0.9827
0.143f
-0.093
-0.1697
0.006
0,002
-0.031
-0.068
-0.028
0.010
0.050
RDSCOR-SA with sup www.ssimeeeeeee mberta have sentence embeddings
sup-simese-mberte
Faithfulness-Step
0.089
0.179t
-0.043
0.135
-0.016
-0.009
0.224f
0.122
0.096
0.014
40.156
Faithfulness-Ie Token
0.080
0.062
-0.047
0.117
0.036
-0.011
0.057
0.035
0.024
0.049
-0.059
Info-Step
0.099
0.064
-0.012
0.1621
0.018
-0.023
0.138
Repetition Token
0.971
0.072
0.112
0,190
.0062
-0.054
-0.102
-0.044
0.061
0.026
0.079
RDSCOR-55 with sup.simece--mbrrte sentence
embeddings
Info-Chain
0.130
0.046
-0.025
0.178t
-0.013
-0.024
0.153t
0.108
0.066
0.195t
10.119
Repetition-Step
0.991
0.125
-0.120
-0,2347
0.007
-0.023
-0.063
-0.155F
-0.025
-0.010
0.076
ROSCO5 with finetuned
sentence embeddings
Faithfulness-Seey
0.169:
0.1651
0,280f
0.000
0.018
-0.113
0.826
0.3941
0,329
0.210
0,2361
Faithulness- Token
0.063
0.132
0.014
0.084
0.030
-0.045
0.285
0.079
0.077
0.082
-0.037
Info
0.150
0.038
0.203
0.107
0.041
-0.81
0,703
0,4501
0.327
0.313
0,259
Repetition-Token
0.971
0.072
-0.112
-0.062
-0.054
-0.102
-0.034
0.061
0.026
0.079
ROSCOE-S3 linetuned sup simese
base sentence embedkings
Info-Chain
0.8617
0.465
0,399
0.025
0.463
0.315
0.933
0,5891
0.499
0.3951
0.337
Repetition Step
0.982
0.141
0.171
0.149)
0.094
0.022
00.13
0.339
0.155
0.016
0.028
ROSCOE-LE
Source Consistency
.0044
0.140
-0.061
00.77
-0.877
0.024
0,316
0.2121
0,151|
0.419
0.096
Self CelfiConsistency
0.3681
-0.028
.0,227
0.019
0.2261
0.106
0.1671
ROSCO8-LC LC
Perplexity Chain
0.288
.0.120
0.1791
0.121
0.021
0.101
0.134
0.1901
0.071
0.194
0.229
Perplexity-Step
0.1711
0.182f
-0.051
-0.105
-0.013
-0.097
-0.068
0.060
0.011
-0.004
0.079
Grammar
0.091
0.200f
0.3591
0.020
-0.043
-0.008
0.223f
0.094
0.046
0.086
0.171
37

---

Published as a conference paper ICLR 2023
Table 26: Somers' D correlations of all metries per different perturbation applied on EQASC Diagnostics datasets.
All metrics are measured reference-based on (s, h). The highest correlation overall for each dataset is in bold. The second hest models are underlined, Correlation scores with p-value < 0.05 are marked with f. (Continued from f6, more details in App. H.1.)
Grammar
Remove
Negute Step
Semantic Error
Perturbations
Repet.
Hallu.
Error
Step
Rouge-1
1.001
1.0001
0.622f
1.000f
1.000F
0.946f
Rouge-2
1.000
1.000
1.000
1.000:
0.946
Rauge-L.
1,000
1,000
0.998
1.000
1,000
0.946
BLEURT
0.8291
0.999
0.5121
1.000
0.9831
0.703
BERTScore
1,000
1,000
1,000
1.000
1.000
0.943
BARTScore
0.784
0.925
0.8881
1.000
0.861
0.897
BARTScore+
0,4431
0.725t
0.5301
L,0001
0.5971
0,885t
BARTScore-P
-0,0951
0.2841
0.404f
0.9661
0.476f
0.587
PRISM
0.879
0.995
0.996
0.927
0,932
CTC Relevancy
0.031
0..14
0.2651
0.954
0.321
0.496
TTC-Consistency
0.9741
0.965t
0.393
-0.429
0.711F
0.718F
Metries (reference-hased [s, (s)
ROSD0E-SA with all-mpmet-buss-- senience embeddings
Reasoning Aligument
0.010
0.995
0.999
0.008
1.000)
0.947
Hallucination
-0.000
0.994
0.9981
-0.305t
1.0007
0.949-
Redundancy
-0,004
0.994
0.998
00.778
1,000)
0.949)
Commonsense
0.9981
1,000
1,0000
0,9491
Missing Step
-0.010
0.012
0.9981
1.0001
1.000
0.949
Semantic Coverage-Step
1.000
1,000
0.9971
1.000
1.000;
0.9411
RO8008 with all-mpwer-hase-x22
embeddings
Semantic Coverage-Chain 1.000:
1.000;
0.9991
1.000
1.000;
0.944
RO5008-88 SA with sup-simece--oobeee
siwcsse--obertta-bbbee sentence embeedings
Reasoning Alignment 0.035
0.995t
1.000
0.003
1.000;
0.950F
Hallucination
0.006
0.994
1.000;
-0.302
1.000;
0.949F
Redundancy
0.007
0.994
1.0001
-0,2731
1.000:
0.949-
Commonsense
0.003
0.025
1.000:
1.000
1.000:
0.949
Missing Step
1.000
0.949
Semantic Coverage Step
1,000
L,000
1JMI01
1,000
1,000F
0.948:
ROSC08-SS with sup simcse-ruberee haxe sentence embeeddnngs
Semantic Coverage Chain
1,0001
L,000
1,0001
L,000
1,000
0.946
ROSCOE-SA SA with linetuned sop-simcce-roobette -uuee sentence embeddings
Reasoning Alignment
0.019
0.995
0.949
Hallucination
0.009
0.996
1.000:
-0.273
1.000
0.945
Redundancy
0.008
0.996
1.000
0.248
1.000)
0.946
Commonsense
0.007
-0.017
1.0001
L:M10
1.000F
0.945:
Missing Step
0.006
-0.014
1.000F
1.000
1.000;
0.946F
Semantic Coverage-Supp Step
1.0001
1.000
1.000
0.946
ROSCO8 35 with finetuned wwp-simcseerrbberaabbae sentence embeddings
Semantic Coverage-Chain
1.0001
1.0001
1.0001
1.0001
1.000;
0.956
38

---

Published as a conference paper ICLR 2023
Table 27: Somers' D correlations of all metrics per different perturbation applied on Entailment Bank Diagnostics datasets. All metries are measured reference-based on (s, h). The highest correlation overall for each dataset in bold. The second best models are underlined. Correlation scores with p-value < 0.05 are marked with f. (Continued from 86, more details in App. H.1.)
Grammar Error
Remove
Shuffle Steps
Swap Steps
Negate Step
Semantic Error
Perturbations +
Repet.
Hallu.
Step
Rouge-1
1.000
1.0001
0.5821
1,0007
0.000F
0.000f
1,000
1.0001
Rouge-2
1.000
1.000
1.000:
1,000
0.982;
0.9351
1,000
1.000
Rouge-L. L
1,000
1.000
1,000
1,000
1.000:
1,000]
1,000]
1.000
BLEURT
0.1971
0.821f
0.1631
0.7867
0.253f
0.021
0.640f
0.1741
BERTScore
1,000
1.000
1,000
1,000
1.000)
1,000]
1,000
1,000
BARTScore
0.574;
0.659
0.574;
0.802
0.983
0.864
0.479]
0.656
BARTScore+
0.195t
0,4301
0.2149
0.839;
0.9887
0.8131
0.253t
0.555t
BARTScore-P
0.1591
0.272
0.3371
0.633
0.877f
0.6561
0.355t
0.4151
PRISM
0.707
0.850
0.612
0.781
0.833
0,466]
0.660
CTC Relevancy
0.311
0.250
00,039
0.457
0,266
0.214]
0.036
0.254
TTC-Consistency
0.7681
0.814f
0.133
-0.022
0.479;
0.413f
0,2331
0.474
ROSCOE Metrics (reference-based on (s. h))
ROSCOE-SA SA with all munet--uwe--22 sentence embeddingg
Reasoning Alignment
0.077
1.000]
1.000:
0.056
-0.034
.0.001
1.0000
0.990
Hallucination
0.002
1.000
1,0007
-0.2001
0.041
0.017
0.995
Redundancy
-0.072
1.000
1,000
00.115
0.034
0.055
[0000]
0.993
Commonsense
.0.016
0.4384
0,264;
0,025
-0002
0.327
Missing Step
-0.077
-0.038
0.9867
0.993
0,040
0.060
197760
0.9851
Semantic Coverage-Step
1.000
1.000
1.000;
1,000;
0.003
0.036
1,000
0.999
ROSCOE with all--mpet--buse-22 sentence embeddings
Semantic Coverage-Chain 1.000;
1.000
1.000:
1,000;
1.000
1.000
1,000
1.000
ROSCOE SA with rip.simese--nberrt-bueee sentence embeddings
Reasoning Aligument
-0.024
1.000t
1.000;
0.039
0.035
0.004
1.000t
0.990t
Hallucimation
-0.011
1.000;
1.000;
-0.099
-0.042
-0.008
1.000
0.990
Redundancy
-0.011
1.0001
1,0007
-0,023
0,034
0.034
1,0001
0.994
Commonsense
-0.021
0.090
0.4331
0.347
-0.049
0.002
0.276
0.342
Missing Step
0.999
0.034
0.034
0.966
0.983
Semantic Coverage-Step
L0000
L0000
1,000
1.000
0.106f
0.040
1.0001
0.994
ROSCOE-SS SS with wwp.atpcee- caweeee-rbbeettbbbbee sentence embeddiings
Semantic Coverage Chain 0.9271
0.929
0.922
0.951
1.000;
0.9681
0.940
0.941
BOSCOE-SA with linetuned sut--simese-rrbbrre seaeccsee-rbbettbbuzeeeee sentence embeddings
Reasoning Alignment
0.040
1.000
0.007
.0.000
1,000
0.999
Hallucination
-0.012
1.000
1,000:
-0.059
0.134
0.040
1.000
0.991
Redundancy
0.025
1.000
1.000
-0.074
0.041
-0.017
1,001
0.994
Commonsense
-0,022
-0.039
0.3684
0.324;
0.129;
0.039
0.304
0.445
Missing Step
0.025
0.008
0.9887
0.9997
0.041F
-0.017
0,971
0.9821
Semantic Coverago-Step
1.000
1.000
1.0001
1,000
0.117F
0.038
1.000
0.994
ROSCOE 55 with finetuned wwp-siincse-rbbern--buue sentence embeddings
Semantic Coverago-Chain
0.912f
0.9341
0.9111
0.9501
1.000;
0.9801
0.940f
0.9311
39

---

Published as a conference paper ICLR 2023
Table 28: Somers' D correlations of all metrics per different perturbation applied on MATH Diagnostics datasets. All metries are measured reference-basee on (s, h). The highest correlation overall for each dataset is in bold. The second best models are underlined. Correlation scores with p-value < 0.05 are marked with f. (Continued from 86, more details App. H.1.)
Perturhations +
Repet.
Hallu.
Grammar Error
Remove
Step
Shuffle
Steps
Swap
Steps
Negate Step
Random Number
Random Operation
Shuflle Numbers
Operations
Shuffle
Rouge-1
0.994
0.961
0.451
0.996]
0.000)
0.000]
1.000
1.000
0.0001
0.433
0.000]
Rouge-2
0.994;
0.961f
0.767
0.996;
0,9901
0.953f
1,0001
1.0007
0.0001
0.999-
0.000f
Rouge-L
0.994
0.961f
0.7697
0.996
0,9981
0.9971
1.0001
1.0007
0.0001
1,000=
0.000
BLEUKT
0
0.041
0.773
0.1031
0.0591
0.156
0.121
0.0691
0.028
0.020
BERTScore
0.972
0.942)
0.978
0.978
1.000)
0.993
0.964
1.000
0.9991
0.998
0.998
BARTScore
0.413;
0.531
0.812f
0.8371
0,9661
0.8031
0.429
0.9381
0.921
0.989-
0.932
BARTScoree
0,022
0.2081
0.539
0.829
0.611
0.437
0.153]
0.804
0.466]
0.963
0.606
BARTScore.
-0.084|
0.041
0.2481
0.474;
0.3691
0.170:
0.015
0.529)
0.3671
0.799;
0.536
PRISM
0.337f
0.465t
0.3861
0.506f
0.4121
0.392f
0.266t
0.437F
0.3471
0.521f
0.2851
CTC- ReCeeeevnncy
-0.0451
0.1681
0.2951
0.592f
0.2531
0.110
0,141f
0.1407
0.0481
0.1697
0.041
CTC- Consistency
0.673:
0.804
0
0.091
0.393
0.240]
0.506]
0.372
0.112
0.408
0.1481
ROSCOB Metries (reference-based
h))
RO8GOE -SA with all mll-mepet-aase-12 sentence embeddings
Reasoning Aligament
-0.012
0.998t
0.8351
0.005
-0.017
-0.013
1.0001
1.000
1.0001
1,000=
1,0001
Hallucination
-0.007
0.999;
0.854
-0.0381
-0.019
-0.007
1.000
1.000
1.0001
1,000:
1,000
Redundancy
0.007
0.999]
0.853
0.040]
.0.019
-0.005
1
1.000
1.0001
1,000:
1.000
Commonsense
-0.030f
-0.004
0.852f
0.9911
-0.022
-0.005
1
1.0001
1.0001
1,000-
1,000
Missing Step
-0.330]
-0.003
0.850
0.991
0.001
-0.004
0.999
1.000
1.000
1.000
1,000
Semantic Coverage-Ster
0,908;
LIM00
0.852;
1.000;
-0.005
-0.008
1.000
1.000;
L0000
1.000=
1,M10
ROSCOE-S -S50E-SS-R SS with all mpmer 2 sentence embedkings
Semantic Coverage Chain
0.940
0.9461
0.799
0.944;
1.000
0.9801
0.933
1.000
0.9991
0.998F
0.9921
ROSCOE-S3
sup sap-ssimcce-roberti
sentence embeddinngs
Reasoning Alignment
00.012
0.9981
0.990;
.0.04
0.003
0.018
0.998
0.997
0.9971
1.000:
L,000
Hallucination
-0.006
0.999t
0.993;
-0,045t
0.012f
0.003
0.998t
0.9961
0.9971
1,000=
1,000
Redundancy
-0.006
0.999t
0.993;
-0,044f
0.012f
0.003
0.998
0.996
0.9971
1.000-
1,000
Commonsense
-0.006
0,002
0.983
0.9891
0,011
0.003
0.995
0.9961
0.9971
1,000
1.000
Missing Step
-0.006
0.001
0.9831
0,9891
0.003
0.995
0.9961
0.9971
1,000=
L,000
Semantic Coverage-Step
0.914
1.000
0.993
1.000
0.0381
0.0171
0.998
0.997
0.9971
1.000
1.000
RO830E 35 with sup--timsse-nbertt-lleee sentence embeddings
Semantic Coverage-
0.674t
0.773t
0.762f
0.763
0.9994
0.8681
0.743f
0.9851
0.9581
0.975:
0.8861
ROSCOE SA with finetuned aop-ssimccee-rbbrta-bbre sentence embeddings
Reasoning Alignment
0.011
0.9991
0.992;
0.019
-0.001
0.025
0.998t
0.9981
0.9981
1.000-
1,000
Hallucination
-0.008
0.999]
0.993
-0.0321
0.011
0.015|
0.998
0.997
0.9971
1,000
1.000
Redundancy
0.008
0.999]
0.031|
0.0161
0.998)
0.997
0.997
1,000:
1,000
Commonsense
-0.008
0,000
0.9834
0.991
0.010
0.015t
0.995t
0.997t
0.9971
1.000-
L,M00
Missing Step
0.008
0.001
0.983
0,990
0.010
0.016)
0.995
0.997
0.9971
1,000
1.000
Semantic Stverage-Step
0.913f
1/000
0.993;
1.000
0.036t
0.0161
0.9981
0.997t
0.9971
1,000F
LJKI0
ROSCOE-SS with linetuned sap www.se----eeeeeee robertu buse sentence embeddingg
Semantic Coverage- Cheirae-CChiin Chain 0.674
0.770
0.769)
0.775t
0.999t
0.865t
0.741
0.985
0.9571
0.973
0.8971
40

---

Published as a conference paper ICLR 2023
Table 29: Somers' correlations of all metrics per different perturbation applied on ProofWriter Diagnostics datasets. All metrics are measured reference-based on (s, h). The highest correlation overall for each dataset is in bold. The second hest models are underlined. Correlation scores with p-value < 0.05 are marked with f. (Continued from f6, more details in App. H.1.)
Grammar Error
Remove
Shuffle Steps
Swap Steps
Negate Step
Semantic Error
Perturbations +
Repet.
Hallu.
Step
Rouge-1
1.000
1.0001
0.6631
1,0007
0.000F
0.000f
1,000
1.000
Rouge-2
1.000
1.000
0.9931
1,000
0.812
1,000
1,000
Rouge-L. L
1,000
1.000
0.993
1,000
1.000:
1.000
1,000]
1.000
BLEURT
0.3491
0.829
0.2211
0.695
0.076
0.0421
0.7881
0.5971
BERTScore
1,000
1.000
1,000
1,000
1.000:
1,000]
1,000]
1.000
BARTScore
0.334;
0.430
0.303
0.417;
0.9861
0.748)
0.343]
0.391
BARTScore+
0.245t
0.391f
0.2511
0.5987
0.996f
0.775t
0.278t
0.515t
BARTScore-P
0.020
0.100F
0.1244
0.2607
0.5951
0.3181
0.1141
0.2601
PRISM
0.8291
0.956
0.822
0.924
0.970
0.871
0,947
CTC Relevancy
0.376
0.409)
0.354
0.725
0.621
0.396
0.391
0.419
TTC-Consistency
0.5371
0.635t
1.99
0.009
0.6081
0.412f
0,4041
0.376
ROSCOE Metrics (reference-based on (s. h))
ROSCOE-SA SA with all mupeet--eue--222222222-2222 sentence embeddiings
Reasoning Alignment
.0.017
0.997|
0.994
0.049
-0.007
.0.000
1
0.9901
Hallucination
0.015
0.997t
0.990:
0.012
-0.001
-0.004
1,0001
0.939
Redundancy
0.007
0.997
0.994
0.048
-0.000
0.010
1.000]
0.992
Commonsense
0.000
0.1871
-0,002
-0.010
0.1691
Missing Step
0,003
-0.001
0.872f
0,8437
0,004
0.011
19760
0.870
Semantic Coverage-Step
0.7931
1.000
0.994i
1,000
-0.008
0,0211
1,000
0.9841
ROSCOE with all--ppet--uss--1222 sentence emboddings
Semantic Coverage-Chain 1,000
1.000
0.994F
1,000;
1.000
1.000
1.000
1.000
ROSCOE SA with sap.simese--rberrt-besee sentence embeddings
Reasoning Aligument
0,012
0.998t
1.000;
-0.041f
0,000
0.001
100011
1.000
Hallucimation
0.016
0.9981
0.998;
0.031
-0.005
20000
1
Redundancy
0.010
0.9981
1,0007
0,0377
0.015t
0.003
1000
1,000
Commonsense
0.017
0.006
0.242
0.4155
-0.005
0.002
(1:38)
0.234
Missing Step
0.839
0.016
0.004
0,866
Semantic Coverage-Stey Step
0.8181
L,000
1,000
1.0007
0.059F
0.021
1.0001
LOMII
ROSCOE-SS SS with wwp--ppppseeeeee simese-ruberte sentence embeddings
Semantic Coverage Chain 0.995
0.9981
0.995
0.997
1.000;
0.999
0.9981
0.9971
BOSCOE-SA with linetuned www.ssince--rbbeteeeee sentence embeddings
Reasoning Alignment
0.016
0.9961
1.000
.0.300
0.000
0.005
1,0001
1,000
Hallucination
0.003
0.9971
0.998
0.016
0.005
-0.010
1
1.000
Redundancy
0.020]
0.997]
1.000
0.050)
0.019)
0.003
1/001
1.000
Commonsense
0.001
-0.009
0.2431
0.404;
0010
-0.009
(.253)
0.2351
Missing Step
0.015t
-0.007
0.871f
0.8397
0.020F
0.003
0,831
0.8661
Semantic Coverage-Step
0.8171
1.000
0.999;
1,000
0.060;
0.0221
1,000
1.000
ROSCOE 55 with finetuned swp-stincww--obbrre-ee roberto--beeee sentence embeddings
Semantic Coverago-Chain
0.9941
0.9981
0.9951
0.996t
1.000;
0.9991
0.9981
0.9961
41

---

Published as conference paper ICLR 2023
Table 30: Somers' D correlations of all metrics per different perturbation applied ASDIV Diagnostics datasets. All metries measured reference-based on (s, h). The highest correlation overall for each dataset bold. The second best models are underlined. Correlation scores with p-value < 0.05 are marked with f. (Continued from $6, more details in App. H.1.)
Random
Random Operation
Shufflc Numbers
Shuffle Operations
Perturbations
Number
Rouge-1
1.0001
0.000f
0.027f
0.000f
Rouge-2
1.000
0.0001
1.000
0.0001
Rouge L
1,000
0.000
1,000
0.000)
BLEURT
0.876
0,9301
0.125
-0.212
BERTScure
1,000
1.000
1,000
1.000
BARTScore
0.871
0.823
0.954
0.924
BARTScore+
0.9481
0,8391
1,0001
0.955t
BARTScore-P
0.7381
0.642f
0.8301
0.7121
PRISM
0.998
1,000
1.000
CTC Relevancy
0.398
0,188
0.160)
0.136
Consosssttnncy
0.6001
0.131f
0.2941
-0.106
Metries (reference-based (s, h))
ROSCOE- SAt with all mmper--uxe-r2 sentence embedddings
Reasoning Aligmment
1.000
1.000)
1.000)
1.000
Hallucination
1.0001
1.0001
1.000;
1,000
Redundancy
1.000
1.000
1.000
1.000
Commonsense
L,000;
1.000
1,000;
L0000
Missing Step
1,0001
1.0001
1,0007
1,000
Semantic Coverage-Step
1.000
1.000
1.0007
1.000;
ROSCCE with all--mper--use--
embeddings
Semantic Coverage-Chain
1.000
1.000;
1.000
1.000;
ROSCCE SA sup-simesee-rabrttobboe
sentence
embeddings
Reasoning Alignment
1.0001
1.000t
1,000;
1.000
Hallucination
L,000;
1.000F
1.000;
1.000
Redundancy
1,0001
1.0001
1,000;
1.000
Commonsense
1.000:
1.000:
1.000:
1.000
Missing Step
Semantic Coverage- Step
LI000
1.000
1,0007
LOI00
ROSCCE-SB with wwwp-simns---eeeee ruberta-bbbe Senience
embeddings
Semantic Coverage Chain
1,0001
1,0001
1,0001
10000
BOSCCE-SS SA with linetuned www.siicee--bbbrre-bbaa sentence embeddhing
Reasoning Alignment
1.000
1.000
10000
Hallucination
1,000
1.000
1.000
1.000
Redunndacy
1.000
1.000
1.000)
1.000
Commonsense
1,000
1.0001
1,000;
1,010)
Missing Stop
L.000
1.000;
1.000;
1.000
Semantic Coverage-Step
1.0001
1,000:
1.000
ROSCCE 55 with Tinetuned
wwp-sanncs--roberr--bbae sentence embeddings
Semantic Coverage-Chan
1.0001
1.0001
1,000;
1.0001
42

---

Published as a conference paper ICLR 2023
Table 31: Somers' D correlations of all metrics per different perturbation applied AQUA Diagnostics datasets. All metries measured reference-based on (r, h). The highest correlation overall for each dataset bold. The second best models are underlined. Correlation scores with p-value < 0.05 are marked with f. (Continued from 86, more details in App. H.1.)
Grammar error
Remove
Shuffle steps
Swap steps
Negate step
Random number
Random operation
Shuffle numbers
Shuftlc operations
Perturbations
Repet.
Halluc.
step
Rouge-1
1.000
1,0007
0.3941
L.000
0.000F
0.000f
1,000F
1.0001
0.000f
0.000=
0.0001
Rouge-2
1.000:
1.000;
0.8661
1,000
0.9841
0.9671
1,000
1.0001
0.0001
1,000:
0.0001
Rouge-L L
1,000
1,000
0.866
1.000
1,000
1.000]
1.000)
1.000
0.000
1,000:
0.000
BLEURT
0.193
0.803f
0.129
0.9461
0.076
0.028
0.640f
0.6271
0.3851
-0.041
-0.113
BERTScore
1,000
1,000
1,000
1,000
1.000
1,000]
1,000)
1,000
1.000
1,000:
1,000
BARTScore
0.568
0.596
0.6881
0.870
0.958
0.8831
0.5201
0.930
0.834
0.984:
0.849
BARTScore+
0,283f
0.597f
0.6131
0.953
0.9601
0.9011
0,368f
0.9801
0.521
0.995+
0.8141
BARTScore-P
0.045
0.025
0.400
0.679f
0.6501
0.5471
0.148
0.846f
0.559f
0.905F
0.7261
PRISM
0.815
0.900
0.7241
0,888
0,901
0.689
0.9421
CTC Relevancy
0.054
0.044
0.102
0.409
0.116
0.141
0.062
0.290
0.030
0.409:
0.023
TTC-Coonisteecy
0.833f
0.766t
0.106
-0.150f
0.270
0.2571
0.256f
0.7741
0.232f
0.741f
0.063
ROSCOE Metrics freference-based on (s, h))
RDSCOE-SA with aul-mppet-buse--2 sentence embeddings
Reasoning Alignment
0.030
0.997
0.985)
0.014
0.054
.0.076
1,000)
1.000
1.000
1,000:
1.000
Hallucination
0.029
0.992;
0.9924
-0,084
0.24
-0.030
1.0001
L.M101
1,000=
1.0001
Redundancy
0.032
0.992
0.9921
-0.088
0.023
[0000]
1.000
1.000
1,000:
1.000
1
Commonsense
0.029
0.9921
0.24
.0.038
1.0001
LOMI0
1,000=
1.MI0
Missing Step
0.032
0.023
0.992t
1.0001
0.023
-0.047
1.0001
1.0001
L,000T
1.000-
1.0001
Semantic Coverage-Step
0.947f
1,000;
0.9871
1.000
0.006
-0.009
1,000
1.000
1.0001
1,000
1.000
ROSCOB 58 with all-mpmet--ase-- sentence embeddingg
Semantic Coverage-
1.000;
1,000;
0.9951
1.000
1.000
1.000;
1.000;
1.000
1.000;
1.000:
1.0001
RGSC33-8A SA with sup- www.ecse--rbbert--buee sentence embeddings
Reasoning Alignment
-0.061
1,000;
1.0001
0.018
0.049
0.006
1.000f
1.0001
1,0001
1,000
1.0001
Hallucination
-0.053
1,000;
1JM00;
-0,699
-0.017
-0.041
1,000
L.MI0
L,MIO
1,000=
1.MI0
Redundancy
-0.054
1,000;
1,0001
-0,063
-0.016
-0.041
1,0001
1,0001
1,0001
1,000=
1,0001
Commonseuse
-0.053
0.065
0.992
1.000
-0.017
-0.041
1,000
1.000
1.000
1.000
1.000
Missing Step
.0.54
1,000
-0.016
-0.041
1,000
1.000
L,000
1,000:
Semantic Covergee-Seey Step
0.966
1.000;
1JMI01
1.000
0.0417
0.008
1.000F
LIKI0
LIMIO
1,000:
LIKI0
ROSCO2-S8 with wimpp-siiceeeeeeeeeeeeeee
semmsse-mbertee-eeeee muberte-buse sentence emnbeddings
Semantic Coverage Chain 0.965
0.9651
0.9441
0.9881
1,000:
1.000
0.953
1.0001
L,M00
1,000
0.9731
RDSCO2- SA with linetuned gep-s-imcse-rubette--e base sentence embeddings
Reasoning Alignment
00.899
1.000)
1.000
L,0I0
1.000
Hallucination
-0.039
1.000;
1,000
0.003
-0.024
-0.025
1.000
1,000
1.000
1,000
1,000
Redundancy
.0.43
1,000
1.000
0.006
0.016
-0.016
1,000)
1.000
1.000
1,000
1.000
Commonsense
-0.039
0.083
0.992t
LJK00
-0.04
-0.025
1.000F
1,M001
LIMI0;
1.000:
1.0001
Missing Step
-0.043
0.089
0.992f
1.000
-0.016
-0.016
1.000F
1.000F
1.000
1.000-
1.000
Semantic Sov
0.967f
1,000;
1.0001
L.000
0.041
0.008
1,000
1.000
1.000:
1.000
ROSC3B 55 with finetuned sup--iiicse-robbrtta- sentenoe embeddings
Semantic Coverage-Chain 0.957$
0.957f
0.9591
0.9801
1.000;
1.0001
0,9711
1.0001
1.0001
1.000:
0.9721
43

---

Published as a conference paper ICLR 2023
H.2 EXPERIMENTS WITH HUMAN JUDGEMENT DATASETS
In this section, we present Somers' D correlation of all metrics on all Human Judged datasets in separate tables. Specifically, Table 32 summarizes meta-evaluations for ROSCCE metrics comparison to baselines all human judged datasets. Fine-grained evaluations are presented in Table 33 for DROP, Table 34, 38 for GSM8K, Table 35, 39 for ESNLI, Table 36 for CosmosQA, and Table 37 for SemEVAL. Human evaluation perspectives used in evaluations are described in App. Table 15.
Looking at how emrors are captured by ROSCOE reference- scores (Fig. 8), we observe strongest correla- tions between Redundancy error and Repetition- *, Self-Consistency scores. Repetition error is not present in this analysis as it has at most 3 occurrences per dataset. Out of the all considered scores, Self-Consistency able cover out of 7 evaluation perspectives, except Missing Step.
MISS
QUAL
COH
0.8
FACT
RED
Relative Presence
LOGIC
COMMON
0.6
0.4
0.0
Falthfulness Step
Fuithfulness- Token
Repetition Tokeen
Token
Repetition Step
Info-Step
Info-Chain
Source-Consistency
Self- conssssssstyyyy
Perplexity Step
Perplexity Chain
Grammar
Figure 8: Relative presence of the strong score-error correlation, measured as the number of datasets where for each score and evaluation perspective pair Somers' D) correlation value in the 90th percentile, normalized by the total number of datasets where this type of perturbation occurs. Statistics collected over ROSCOE reference-free free scores with finetuned SimCSE embeddings, and evaluation perspectives where at least 10 eтгоrs are present in a dataset.
We further look at specific human annotated examples where our ROSCOE gives highest and lowest scores to understand strength and weaknesses of the proposed approach. Results are summarized in Table 40. Similar analysis for diagnostic datasets is summarized in Table 41.
44

---

Published as conference paper ICLR 2023
Table 32: Somers' correlations metrics on human judged datasets. metrics are measured reference-fre on (s. h). The highest correlation overall for each dataset in bold and second best is underlined. Correlations that are not significant (p-value 2 0.05) are omitted when aggregating, and "-" denotes an absence of any significant correlations. (Continued from $6, more details App. H.2.)
DROP
GSM8K ESNLI COSMOS
SemEVAL
Rouge-1
0.239
0.180
0.559
-0.264
-0.520
Rouge-2
0.320
0.502
0.180
Rouge-L
0.278
0.252
0.557
-0.441
-0.478
BLEURT
0.541
BERTScore
0.275
0.235
0.590
-0,420
-0.295
BARTScore
-0.835
40,546
0.549
-0.544
BARTScore+
-0.665
0,482
-0.186
BARTScore-P
-0.642
0.255
PRISM
-0.733
-01.55
0.580
-0.376
CTC-Relevance
0.333
-0.371
0.334
-0,349
TTC-Consistency 0,462
-0.174
0.647
0.275
-0,301
ROSCOE Metries (reference-free on (s, h))
ROSCOR-SA with finetuned sup-simgg wwtpssieeeeeeett-eeeeee roberta-buse sentence emheddings
Fathhfulnes-SSep
0..496
0.403
Faithfulness-Toko
0.417
0.521
-0.320
Info-Step
0.500
0.178
0.493
Repetition-Token
0.578
0.392
0.441
0.555
0.337
ROSCOE SA with all coll-mmnettbbae
baseetbase--2222 sentence embeddiings
Faithfulness-Stey
0.297
0.423
-0.424
0,330
Faithfulness- Token
0.290
-0,443
0,524
Info-Step
0.301
0.542
-0.429
Repetition- Token
0.790
0.500
0,799
0.638
0.485
ROSCOE- SA with sup.siinceee-bbbeetbbbuu sentence embeddingss
Faithfulnexs-Ster
0.,477
0.502
Faithfulness Token
0.454
0.540
-0.420
Info-Step
0.510
0.599
-0.409
-0.321
Repetition-Toker 0.578
0.392
0,441
0.555
0.337
ROSCOE-S3 S3 with finetuned
coaennssee--beet--buussee sentence embeddings
Info-Chain
0,446
0.385
0.310
Repetition-Stes
0.824
0.514
0.530
0.593
0.411
ROSCOE-SE 55 with all-mppet--ase-22 sentence embeddings
Info-Chain
0.406
0.507
-0.198
0.367
Repetition-Step 0.791
0.471
0.487
0.642
0.508
ROSCOE-S5 55 with sup-sticcee-mbbrrteeeeeeeo
sentence embeddings
Info-Chain
0.271
0.531
-0.367
Repetition-Step
0.484
ROSCOR-LI
Sourre-Consistern
0.390
0.172
0.425
0.444
Self Consistency
0.584
0.345
0.531
0.417
0.372
ROSCOE-LC
Grammar
-0.184
0.255
0.517
Perplexity-Ste
0.205
-0.307
0.345
Perplexity-Chaee -hannn
-0.61
-0.273
0.447
-0.212
-0.373
45

---

Published as conference paper ICLR 2023
Table 33: Somers' D correlation all metrics on DROP human judged dataset analyzing step-by-step reasoning on overall chain and step-level perspectives. All metrics are measured reference-free on (s, h). The highest correlation overall for each aspect on each dataset is in bold, second best are underlined. Correlation scores with p-ealue < 0.05 are marked with f. (Continued from 6, more details in App. H.2)
QUAL
COMMON
LOGIC
MATH
Rauge-1
0.157
-0.160
0.436
0.2391
-0.335
-0.731 f
-0.736
0.030
0.702
0.196
0.1734
Rouge-2
0.137-
-0.146
0.488
0,3201
-0.284
-0,716f
-0.442
0.027
0.584
-0.035
0.129
Rouge-1.
0.131
-0.201
-0.749
0.027
BLEURT
0.121
-0.101
0.256
0.3281
-0.333
-0,7251
-0.370
0.078
0.514
0.101
0.087
BERTScore
0.133
-0.115
0.494
0,275
-0.177
-0.6471
-0.043
-0.003
0.707
0.098
0.142
BARTScore
-0.088
-0.392f
-0.831
BARTScore+
0.039
-0.159
0.536
0.192
-0.553
-0.6651
-0.841
-0.141
0.303
0.261
0.066
BARTScore-P
.0.007
-0.152
0.546
0.169
-0.473
-0.642
-0.994
.0.039
0.380
0.265
0.012
PRISM
0.129:
-0.081
0.465
-0.379
-0,7331
0.668
0.165
CTC-Relevance
-0.027
-0.100
-0.072
0.333f
-0.041
-0.6221
-0.087
-0.091
0.394
0.134
-0.056
CTCCConsiisency
0.030
-0.133
0.243
0.4621
-0.148
-0.6571
-0.106
-0.041
0.769
0. 0.106
-0.002
ROSCOE Metries (reference-free on (s)
ROSCOB-SA linetuned wuw-siicce-mobertteeeeooeoooeeee
embeddings
Faithfulness-Segg
0.096
-0.095
0.572
0.496t
0,143
-0,5661
-0.322
0.278
0.760
-0.022
0.042
Faithfulness-Tode Token
0.177;
-0.016
0.472
0.417f
-0.189
-0.525
-0.038
0.131
0.678
0.002
0.130
Info-Step
0.142
-0.069
0.643
0.216
-0.581
-0.284
0.287:
0.846
0.024
Repetition-Toker Token
0.055
0.2101
0.018
0,170
0.340
0.578t
0,952
0.026
-0.822
-0.140
0.062
ROSCOE- Sa with all- hill-mpnet-bax---e sentence
embeddings
Faithfulness-See
0.054
-0.156
0.153
0.2971
-0.073
-0.578
-0.111
0.116
-0.221
0.001
-0.042
Faithfulness- Token
0.156
-0.042
0.362
0,2901
-0.888
-0,504
-0,216
-0.058
0.288
0.063
0.085
Info-Step
0.090
-0.116
0.308
0,301
-0.153
-0.619f
-0.043
0.021
0.250
0.078
0.020
Repetition Token
0.130
0.370f
0.027
0.087
0.313
0.790f
0.519
0.114
-0.822
-0.181
0.053
RCSCOE -S with wwp-simncee-mbertt-butt
sentence embeddings
Faithfulness--teg
0.084
-0.128
0.195
0.4771
-0.061
-0.6431
-0.596
0.085
0.236
-0.061
-0.051
Faithfulness- Token
0.186i
-0.031
0.414
0.454
-0.245
-0.574
-0.139
0.098
0.514
0.047
0.093
Info-Step
0.141
-0.095
0.443
0.510
-0.121
-0,6921
-0.471
0.040
0.731
0.066
0.010
Repetition- Token
0.055
0,2101
0.018
0.170
0.340
0.578f
0.952
0.026
-0.822
-0.140
0.062
ROSCOE- 55 with finetuned sup-siincse-rnberaabbas sentence embeddings
Info-Chain
0.108
-0.128
0,462
0,222
-0.393
-0.446ł
-0.375
0.025
0.207
0.4461
0.114
Repetition-Step 0.036
0.400;
-0.543
-0.258+
0.165
0.824F
0,909
0.052
-0.822
0.192
-0.016
ROSCOR-SS mpnectteetttee base-v2 sentence emheddings
Jafo-Chain
0.165
0.020
0.395
0.240
-0.027
-0,480
-0.135
0.122
0.394
0.4061
0.106
Repetition-Step
0.052
0.358
-0.111
-0.115
0.260
0.791
0,856
0.105
-0.822
0.066
0.015
ROSCOR-SE iS sup-simese-
sentence embeddings
Info-Chain
0,138-
-0.090
0.504
0.27
-0.158
-0,664
-0.130
-0.089
0.611
0.381
0.093
Repetition Step
0.073
0.357
0.143
-0.885
0.451
0.799)
0.918
0.128
40,822
0.047
0.021
ROSCOB-LI
Course-Consistency
0.200
0.243
0.462
0.3901
-0.004
0.085
0.697
0.009
0.365
0.420
0.184
Self-Consisteccy
0.032
0.295+
-0.076
0.198
0.201
0.584
0.139
0.187
-0.707
0.344
-0.075
ROSCOB-IC
Grammar
0.220
0.141
0.250
0.260
0.536
0.001
-0.553
0.169
0.260
0.450
0.111
Perplexity Step
0.185
0.034
0.214
-0.022
-0.112
-0.3201
-0.827
-0.043
0.332
0.259
0.205
Perplexity-Chal
0.087
-0.152
0.185
-0.104
-0.515
-0.6111
-0.952
-0.178
0.663
0.103
0.117
46

---

Published as conference paper ICLR 2023
Table 34: Somers' D correlation of all metrics GSM8K human judged dataset analyzing step- step-by-step reasoning overall chain and step-level perspectives. overall for each aspect on each dataset is in bold, second best underlined. Correlation scores with p-value < 0.05 are marked with f. (Continued from & 6, more details App. H.2)
metrics are measured reference-free on (s, h). The highest correlation
QUAL
CON
COMMON
FACT
BALL
RED
REP
LOGIC
MATH
GRAM
MISS
Rouge-1
0.122
0.202
0.071
0.156
0.275
0.037
0.222
0.073
0.233
0.173
0.1801
Rouge-2
0.089
0.139
0.060
0.102
0.175
-0.016
0.561
0.030
0.113
0.051
0.148
Rouge-L
0.176t
0.2681
0.195f
0.169
0.180
0.054
0.558
0.120
0.212
-0.125
0.2521
BLEURT
0.160
0.2481
0.134
0,256
0.099
0.075
0.227
0.124
0.057
0.077
0.210
0.1731
0.220
0.2351
BARTScore
0.009
0.035
-0.047
0.044
-0.246
-0.261
0.424
-0.002
-0.546
-0.321
-0,003
BARTScorre+
0.064
0.132
0.078
0.054
-0.003
-0.039
0.879
0.085
-0,298
-0.281
0.090
BARTScore-P
0.042
-0.116
-0.102
PRISM
-0.112
-0.075
-0.099
-0.037
-0.385
-0.455f
-0.086
-0.171
-0.341
0.130
-0.093
CTC- ReC--eeeeencee
-0.086
-0.148
-0.077
-0.111
0.009
-0.371
0.566
-0.106
-0.093
-0.061
+0,088
CTC- Consistency
-0.1571
-0.2031
-0.006
-0.129
-0.013
-0.318
0.556
-0.174
-0.019
-0.056
-0.2041
ROSCOE Metries (reference-free (s, (s)
ROSCOE-22 SA with finetuned sip-siincs--rbbert -a--robbrta-baee sentence embeddings
Faithfillness Step
0.012
0.083
0.097
0.014
-0.138
-0.019
0.182
-0.024
-0.101
0.286
0.027
Faithfulness-
-0.012
0.036
-0.052
-0,067
-0.099
-0,304
0.788
-0.042
-0,408
0.204
0.016
Info-Step
0.059
0.137
0.178f
0,082
-0.025
0.059
0.364
0.031
-0.303
0.329
0.108
Repetition-Toke Tokeen
0.200
0.193
0.186F
0.075
0.224
0.392
0.788
0.183
0,443
0.173
0.270
RCSCOE-SA. with all call-mpee--bbne iapee--bss---
buse-v2
embeddings
FaithfullnessSee Step
-0,039
-0.016
0,002
-0,027
0.012
-0.376
0.581
-0.173
-0.127
0.102
-1021
Faithfulness- Token
Token
-0.078
-0.002
-0.090
-0.079
-0,141
-0.4431
0.485
-0.137
-0.623f
0.074
-0.068
Info-Step
0.095
0.121
0.099
0.139
0.112
-0.065
0.662
-0.016
-0.056
0.092
0.148
Repetition-Tokerr Token
0,214
0,2081
0.184f
0.081
0.500
0,238
0.747
0,208
0.498
0.339
0,3061
ROSCOE-SA with sup sup-simece- reberee-bberabbae base sentence emboddings
Faithfulness-Stey
-0,061
-0.034
-0.006
0.002
-0.158
-0.142
0.237
-0.1921
-0,221
0.168
-0.045
Saithfulness-Tokee
-0.051
-0.008
-0.087
-0,084
-0.168
-0.352
0.732
-0.098
-0.424
0.145
-0.034
Info-Step
0.059
0.102
0.106
0.124
0.040
0.046
0.545
-0.053
-0.236
0.099
0.131
Repetition-Toker
0.200
0.193t
0.186t
0.075
0.224
0.392
0.788
0,1831
0.443
0.173
0.270
ROSCOR-SS with finetuned sup-singg aappsiimse--bbree robert base sentence emheddings
Info-Chain
0.097
0.064
0.059
0.080
0.035
0.385
0.722
0.103
0.121
-0,110
0.130
Repetition Step
0.1991
0,1661
0.168
0.254
0.514
0.222
ROSCOB-SS with all-mpnet-buseee sentence embeddings
Info-Chain
0.059
0.071
0.020
0.043
0.039
0.128
0.288
0.004
0,260
-0.026
0.098
Repetition Step
0,218
0,16
0.167
0,227
0.471
0.245)
ROSCOE-SS with coreeiinnce--reetttoe baaee sentence embeddings
Info-Chain
0.042
0.051
0.043
0.057
0.240
0.041
0.808
-0.007
-0.211
0.038
0.130
Repetition-Step 0.322
0.299
0.275
0.227f
0.466
0.6381
0.879
0.1921
0.5631
-0.311
0.354
RCSCOE
Yource-Consistenge 0. 108
0.037
-0.019
0.172
0.121
0.030
0.551
0.082
-0.005
0.219
0.097
Self Consistency
0,2831
0.2671
0.177
0.345
0.207
0.354
0.980
0.2191
0.087
-0.230
0.2231
RCSCOE-LC
Grammar
00.134
-0.199
-0.260
-0.081
-0.234
-0.246
0.056
-0.264
-0.207
0.298
-0.1841
Perplexity-Step
-0.2971
-0.278
-0,366
-0,3071
-0.608f
-0.5911
-0.136
-0.334F
-0.514f
0.148
-0.331
Perplexity-Chuin
-0.332f
-0.336F
-0.3221
-0.273t
-0.695t
-0.6821
-0.556
-0.354
-0.697f
-0.084
-0.4081
47

---

Published as conference paper ICLR 2023
Table 35: Somers' D correlation metrics on ESNLI human, judged dataset analyzing step-by-step reasoning on overall chain and step-level perspectives. All metrics are measured reference-free on (s, h). The highest correlation overall for each aspect on each dataset is in bold and second best is underlined. Correlation scores with p-ealue < 0.05 are marked with f. (Continued from 6, more details in App. H.2)
QUAC
COMMON
FACT
LOGIC
GRAM
MISS
Rouge-1
0.213
-0.121
-0,373
0.152
-0.000
40,284
-0.276
0.113
-0.403
0.559
Rouge-2
0.216
-0.075
-0,336
0.169
-0.097
-0.240
-0.255
-0.080
-0.432
0.502
Rouge-1.
40.259
BLEURT
0,154
-0,202
-0.271
0.112
-0.041
-0,230
-0.322
0.267
-0.6411
0.541
BERTScore
0.255
0.019
-0,205
0.222
0.153
-0.179
-0.141
-0.360
-0.252
0.590
BARTScore
0.1891
-0.096
0.079
-0.200
40.688
0.549
BARTScore+
0.2091
-0.059
-0.192
0.166
0.204
-0.147
-0.168
-0.160
-0.5781
0.482
BARTScore-P
0.092
11.096
-0,208
0.213
0.092
-0.209
-0.192
-0.587
-0,370
0.255
PRISM
0.264
0.089
-0.151
0.093
-0.235
-0.042
-0.329
0.580
CTC-Relevance
0,071
-0.051
-0.074
0.063
-0.020
-0.098
-0.093
0.720
-0.205
0.334
CTC-Consistency
0.029
0.218
0.647
40.009
0,483
0.021
0,3501
0.533
-0.071
-0.060
ROSCOE Metrics (reference-free on (s. h))
ROSCOE-SA with
sup-
sotp---iime---bet---eeeeeee--e-- sentence embeddings
Faithfulness-Step
0.236f
-0.163
-0.027
0.096
-0.146
-0.139
-0.293
0.560
-0.386
0.403
Faithfullness- Töken
0.279)
0.010
-0.145
0.232
0.2011
-0.279
-0.119
0.067
40.293
0.521
Info-Step
0.222
-0.127
-0.140
0.143
-0.068
-0.220
-0.333
-0,310
0.493
Repetition-Toker
-0.037
-0.020
0.403
0.244
0,330
0,441
0.153
0.240
-0.059
-0.436
ROSCOE SA with all mmnet--aae-r2 sentence embeddings
Faithfulness-Stey 0,261
-0.117
-0.058
0.242
-0.425
-0.2961
-0.248f
0.480
-0,244
0.423
"aithfulness-Tooee 0.270
-0.083
-0,216
0.136
0,313
-0,248
-0.197
0.347
-0.433
0,524
Iteo-Step
0.2901
-0.006
-0.09
0.286
-0.102
-0,245
-0.246
0.373
-0.342
0.542
Repetition-Tokke
-0.040
0.129
-0.189
0.169
0.7991
0.6171
0.151
-0.267
0.084
-0.445
ROSCOE- SA with wwp-srincse-rbbbrttato baate sentence embeddings
Faithfulness-Ster
0.262
-0,083
-0,244
0.257
-0.163
-0.3721
-0.283f
0.413
-0.375
0.5021
Faithfulness- Token
0.275
0.004
-0,330
0.206
0.214
-0.3031
40.140
0.347
-0.321
0.540
Info-Step
0.295
0.046
-0.126
0,299
-0.071
-0.3061
-0.208
0,333
-0,3971
0.599
Repetition Token
-0.037
-0,020
0.403
0.244
0.330
0.441f
0.153
0.240
-0.059
-0.4361
ROScos-Bs 35 with finetuned sup-simeeee-rbbettbbboor sentence embeddings
Info-Chain
0,083
-0.197
-0.466
-0,225
-0.510
-0,259
-0.067
0.267
0.074
0.310
Repetition-Step -0.021
0.103
-0.063
0.249
0,310
0.530f
0.178
-0.107
0.053
-0.433
ROSCOR-SS SS with mellmmmnnt-baee sentence emheddings
Info-Chain
0.191
-0,233
-0.438
0.052
-0.153
-0.253
-0.3681
0.533
-0.452
0.507
Repetition-Step 0.007
0.280
0.173
0.365t
0.694
0.4871
0.225
0.213
0.193
-0.472
ROSCOR-S2 S5 with sup-simece- roweeee--rbbeet--beeeoee sentence embeddings
Info-Chain
0.211f
-0.083
-0.192
0.212
0.024
-0.319
-0.257f
0.400
-0.455
0,531
Repetition-Sie
0.015
0.180
0.397
0.332
0.367
0.484
0.115
0.013
0.125
-0.501
ROSCOE-LI
Source-Cnsissenng
0.012
-0.028
0.334
0.4251
-0.055
0.112
0.600
-0.258
-0.2221
Self-Consistency
-0.028
0.354
0.156
0.087
0,561
0,370
0,531
-0.333
0.222
-0.351
ROSCOE-IC
Grammar
0.063
0.069
-0.411
-0.003
0.122
-0.286
0.2551
-0.027
0.411
-0.072
Perplexity Step
0,084
0.087
-0.655t
-0.153
0.255
-0.254
-0.148
-0.147
0.090
0.345
Perplexity-
0.027
0.081
-0.616
-0,289
0.075
-0.447
-0.155
-0.533
0.249
0.4471
48

---

Published as conference paper ICLR 2023
Table 36: Somers' correlation metrics on COSMOS human judged dataset analyzing step-by-step reasoning on overall chain and step-level perspectives. All metrics are measured reference-free on (s, h). The highest correlation overall for each aspect on each dataset is in bold and second best is underlined. Correlation scores with p-value 0.05 marked with f. (Continued from 6, more details in App. H.2)
QUAL
CCH
COYMON
FACT
HALL
LOGIC
MISS
Rouge-1
-0.011
-0.007
-0.182
-0.077
0.292
-0.576
-0,807
-0.2641
-0.644
0.113
Rouge-2
0.021
0.028
(1.075
-0.131
0.239
-0.561
-0.174
00.108
-0.314
0.180
Rouge-L
-0.133
-0.094
-0.044
0.252
-0.6371
-0.114
-0.4411
BLEURT
0.098
0.088
0.019
-0.054
0.097
-0.666
-0.617
-0.1811
-0.5221
0.218
BERTScore
0.095
0.113
0.059
-0.055
0.234
-0.478
-0.492
-0.058
-0.4201
0.114
BARTScore
0.024
0.159
-0.026
0.001
-0,544f
-0.208
-0.122
-0.420
BARTScore+
0.048
0.061
-0.102
+0,004
0.159
40.507
-0.602
-0.1861
-0.4991
0.159
BARTScore-P
0.009
0.021
-0.149
0.010
0.267
-0.385
-0.508
-0,2071
-0.453
0.142
PRISM
00.16
-0.446
003376
TTC-Relevance
0.070
0.035
0.246
0.155
0.233
-0.294
-0.780
-0.001
-0.349
0.016
CTC- Consistency
0.093
0.097
0.2751
0.084
0.140
-0.032
-0.201
0.051
0.064
-0.006
ROSCOE-SA SA with finetuned
sentence embeddings
Faithfulness Step
0.089
0.109
0.149
0.256
-0,322
-0.216
-0.019
0.034
Faithfulness-Tee Tokeen
0.038
0.039
-1.032
.0.093
0.285
40.422
-0,220
-0.026
-0,3201
0.012
Info-Step
0.109
0.082
-0.011
0.119
0.302
-0.340
-0.811
-0.039
-0.135
0.057
Repetition-Toke
0.050
0.120
0.187
0.103
0.103
0.5551
0,231
0.192f
0.4581
-0.2331
ROSCOE-S5 SA with all-mpnet--asse-22 sentence embeddings
Faithfulness Step
-0.011
0.044
-0.008
-0.012
0.127
-0.424
-0.235
-0.061
-0.268
-0.09
Faithfulness-Tokee
0.035
0.078
-0.124
-0.128
0,323
-0.515t
-0.667
-0.036
-0.230
-0.008
Info Sito-Step
0.011
0.043
-0.047
-0.023
0.186
-0.4291
-0.481
-0.081
-0,268
-0,037
Repetition- Token
0.045
0.092
0.275;
0.122
0.298
0.638f
0,398
0.252f
0.386
-0.1841
ROSCOB with sup-riincse-mmeetaa-uu
embeddings
Faithfulness Step
0.036
0.022
0.072
0.111
0.184
-0.811
-0.318
-0.076
-0.337
-0.053
Faichfulness-
0.036
0.030
-0.003
-0.058
0.205
-0.486f
-0.333
-0.056
-0.4201
-0.020
Info-Step
0.026
-0.025
0.069
0.079
0.248
-0.409+
-0.720
-0.088
-0,292
0.000
Repetition-Tokke Token
0.050
0.120
0.187
0.103
0.103
0.5551
0,231
0.192
0.4581
-0.2331
ROSCOE-88 55 with finetuned sup-sincce-robbrtte sentence embeddings
0.089
0.048
0.139
Repetition-Ste Step
0.011
0.050
0.178
0.132
-0.046
0.5931
0.670
0.2951
0.330
-0.244
ROSCOE- 55 with all-mpnet--ase--2 sentence embeddings
Info-Chain
0.073
0.091
-0.174
0.033
0.145
-0.215
-0.409
-0.1981
-0.411f
0.103
Repetition-Step
0.047
0.127
0.124
0.060
0.153
0.6421
0.617
0.346
0.563
-0.184j
RDSCO8-S3 58 with sup-timccee--obertta sentence embeddings
Info- Info-Chain
0.114
0.055
0.034
-0.005
0.218
-0.367
-0.879
-0.046
-0.222
0.153
Repetition-Step 0.061
0.127
0.095
0.076
0.246
0.6581
0.500
0.2561
0.4961
-0.145
ROSCOE-II
Source- Consistency
0.184
0.1831
0.150
0.2851
0.241
0.444|
0.091
0.111
0.303
0.011
Self-Consistency
0.048
0.080
0.190
0.173
-0.021
0.417
0.610
0.192
0.4011
-0.2521
ROSCOE-1L LC
Grammar
0.189;
-0.013
Perplexity-Step
0.122
0.157
-0.208
-0.021
0.034
0.028
-0.140
-0.113
-0.295
0.064
Perplexity-Chiie
0
0.047
40.193
0.001
.1,073
40.311
-0.561
-0.2121
40,5421
0.130
49

---

Published as conference paper ICLR 2023
Table 37: Somers' correlation metrics on SemEVAL human judged dataset analyzing step-by-step reasoning on overall chain and step-level perspectives. All metrics are measured reference-free on (s, h). The highest correlation overall for each aspect on each datasel is in bold and second best is underlined. Correlation scores with p-value 0.05 are marked with f. (Continued from 6, more details in App. H.2)
QUAL
COMMON
FACT
HALL
RE2
LOGIC
Rouge-1
-0.1991
40.208
-0.246
40.118
-0.053
.0.520
-0.398
-0.206
40.230
0.108
Rouge-2
-0.085
0.060
-0.478
-0.063
0.561
-0.232
-0.083
-0.073
0.141
0.090
Rauge-1.
-0.198
40.396
-0.115
BLEURT
-0.3131
-0.3831
-0.372
-0.208
-0.034
-0.4821
-0.383
-0.356
-0.104
-0.074
BERT
-0.051
0.064
-0.517
-0.035
0.524
-0.002
-0.218
-0.2951
0.266
0.063
BARTScore
40.137
0.209
-0.284
BARTScore+
-0.046
-0.098
0.652
-0.033
-0.073
0.056
0.204
-0.204
0.048
-0.016
BARTScore-P
400755
-0.168
0.633
40080
0.107
0.196
0.277
-0.230
(.054
-0,022
PRISM
-0.082
-0.469
-0.115
0.073
-0.174
-0,354
-0.134
0.075
CTC-Relevance
-0.1467
-0.219F
0.256
-0.047
0,442
-0.071
-0.121
-0.349
0,101
-0.145
CTC-Connistency
-0.178F
-0.2411
0.101
0.009
0.583
+0,301
-0.296
-0.3351
0,208
-0.142
ROSCOE Metries (reference-free on (s.h))
RCSCOE-SA
sup-simccxe-olle
embeddings
PuthffulnessSSep
0,157f
0.198
0.275
0.189
0.694
0.233
-0.262
-0.115
0.092
0.115
Faithfulnnes: Token
0.030
0.182
.0.150
0.009
0.714
0.116
40.422
-0.101
0.262
0.085
Info-Step
0.068
0.113
0.111
0.055
0,835
0.003
-0,252
-0.187
Repetition-Tooe Token
0.062
0,150
0.401
0.021
-0.078
0.337f
0.670
-0.074
-0.007
-0.131
ROSCOE-SA S& with all womet-bunseeeeeeee sentence
embeddings
Fuithfulness-Sue
0.197f
0,221f
0.005
0.201
0,461
0.330F
-0.160
-0.019
0.182
0.108
Farthfulness- Tokee
0.030
0,161
-0,208
0.013
0,597
-0.044
-0.495
-0.111
0.063
0.1861
Itep
0.111
0.127
-0.111
0.125
0.544
0.161
-0.073
-0.176
0,210
0.076
Repetition- Token
0.134f
0.178t
0.662
0.066
0.364
0.485f
0.772
-0.004
0.157
-0.115
ROSCO8-55 SA with sup-sime neee--obert--bee bose sentence embeddings
Fuithfulness-Ster
0.028
0.123
-0.459
0.112
0.908
0.174
-0.199
-0.162
0.021
0.059
Faithfulness: Token
-0.021
0.133
-0.227
0.021
0.752
0.084
-0.398
-0.134
0.119
0.052
Info-Step
0.040
-0.003
-0,362
0.024
0.777
0.015
-0.296
-0.321
0.038
0.039
Repetition- Token
0.062
0.150
0.401
0.021
-0.078
0.337f
0.670
-0.074
-0.007
-0.131
ROSCOE- 55 with finetuned sup-simcce--mbertte sentence embeddings
Info-Chain
-0.040
0,008
-0.082
-0.005
0.539
-0.164
-0.015
-0.070
0,086
-0,082
Repetition-Stey
0.051
0.143
0.546
0.043
-0.024
0.411f
0.723
0.038
-0.018
-0.1954
ROSCOR-SS with all mapl-mmmett-eee buse-v2
emheddings
Info-
0.076
0.128
-0.700
0.122
-0.005
0.367F
0.117
-0.140
0.068
-0.018
Repetition-Step 0.077
0.117
0.633
0.036
0,141
0.508F
0.684
-0.026
0.025
-0.155
ROSCOR-SS S5 with sup-simcce-robernte
sentence embeddinng
Info-Chain
-0.115
-0.038
-0.169
0.059
0,539
-0,260
-0.476
-0.169
-0.081
-0,009
Repetition Step
0.104
0.132
0.787
0.042
0.136
0.535
0.811
0.006
(1.66
-0.169)
RCSCOE-LI
Counce-Consiseecyy
0.059
0.016
0.546
0.206
-0.029
0.006
0.010
-0.241
-0.139
-0.063
Self-Consistency
0,162f
0.250
0.536
0. 104
0,383
0.372F
0.223
-0.091
0.061
-0.075
BOSCOE-LC
Grummar
-0.076
0.014
0.101
-0.223+
-0.335
0.104
0.393
-0.215
0.5171
Perplexity Step
-0.026
-0.053
0.797
0.037
-0.607
0.020
-0.019
-0.071
40.30
-0.039
Perplexity- Chain
-0.1417
-0.237F
0.324
-0,126
-0.650
-0.373
-0.481
-0.151
-0,284
0.039
50

---

Published as a conference paper ICLR 2023
Table 38: Somers' D correlation of all metrics on GSM8K human judged dataset analyzing step-by-step reasoning on overall chain and step-level perspectives. All metrics measured reference-based on (s. ft). The highest correlation overall for each aspect on each dataset is in bold, second best underlined. Correlation scores with p-value < 0.05 are marked with f. (Continued from 3 6, more details App. H.2)
QUAL
COR
COMMON
FACT
HALL
RED
REP
LOGIC
KATH
GRAM
MISS
Rouge-1
0.572
0.506
0.590
0.5321
0.498
0.533
0.582
0.430
0.686]
Rouge 2
0,551
0.5201
0.566
0,511
0.467]
0.439:
0.758
0.5551
0.617
0.480
0.713
Ronge-I.
0.591+
0.5421
0.605t
0.559
0.4521
0.613:
0.575t
0.715F
0.457
0.730
BLEURT
0.487
0.391T
0,502
0.3891
0.392
0.222
0.682
0.4041
0.233
0,283
0.597
Seore
0.505
0.4251
0.4881
0.475
0.4771
0.585
0.763
04431
0.6771
0.347
0.627
BARTScore
0.429
0.352)
0.522
0.317
0.315
0.128
0.692
0.436
0.279)
0.237
0.555
BARTScore
0,531
0.455+
0.579F
0.4601
0.273
0.303
0.722
0.5011
0.577
0.334
0.679
BARTScore-P
0,343
0.2801
0.376
0.2681
0.210
0.062
0.2981
0.207
0.227
0.441f
PRISM
0.579-
0.5111
0,5931
0,531
0.3631
0.392
0.707
0.5571
0,540f
0.423
0.7281
CTC-Relevance
-0.047
-0.093
-0.065
-0.065
-0, 134
-0,286
0,460
-0.055
0.056
0.242
40.102
CIC-Consistenay
-0.2721
-0.2791
-0.2591
-0.2641
-0.182
-0.3851
-0.399
-0.1921
-0.233
0.028
-0.320
BOSCOE Metries (reference-based on (s. Ix)
ROSCOE-SA. with linetuned sup-sincse-mbbrra
embeddinggs
Hallucination
0.466-
0.460:
0,398
0.445t
0.371
0.4627
0.091
0.4391
0.624f
16699
0,319
0,519f
Redundancy
0.464;
0.463:
0.396
0.379
0.552
0.242
0,459;
0.472
Semantic Coverage Step
-0.030
0.002
0.017
00.154
-0.237
00.84
-0.298
0.023
0.023:1
0.335
0.064
0.002
Missing Step
0,484;
0.419:
0.509;
0.364
0.3571
0.325
0,333
0,4381
0.6851
0.207
0.590
Reasoning Alignment
0.613t
0.5907
0,570
0.573t
0.4641
0.5297
0.460
0,5871
0.725
0.446
0.71
Commensense
0.4117
0.348
0.424f
0,2891
0,264
0.246
0.404
0.3581
0.6771
0.367
0.494
ROSCOE-SA with all mall--ppnettuue
-uuse--- sentence embeddiigs
Hallucination
0.702
0.6211
0.7044
0.630H
0.629
0.422
0.818
0.644
0.340
0.064
0.846
Redundancy
0,453t
0+411
0,451f
0.417f
0.4061
0,353+
0.505
0.4681
0.195
0.204
0,570
Somantic Coverago-Step
-0.066
-0.062
0.005
-0.042
-0.148
0.3297
-0.576
0.006
0.232
0.291
+0.085
Missing
0.501
0.449:
0.520
Reasoning Alignment
0.583+
0.523+
0,5597
0,558
0.3801
0,3817
0.571
0.5697
0.418
0.296
0,716
Commonsense
0.626
0.529
0.657
0.594
0.578
0.828
0.574
0.421
-0.125
ROSCOE-SA with spp rimmeee roherta base sentence embeddings
Hallucination
0.8067
0.707:
0,731f
0.729;
0.625t
0.357t
0.7121
0.575t
0.8%6
Redundancy
0.734+
0.6611
0.665t
0.699f
0.5981
0.4637
0.717
0.6641
0.517f
-0.061
0.840
Semantic Coverage-Step
0.063
0.063
0.127
0.057
0.216
0.300
-0.111
0.161
-0.205
0.492
0.023
Missing Step
0.722
0.631
0.685
0.691
0.517
0.353:
0.808
0.655)
0.482
0.003
0,821
Rensoning Alignment
0.712F
0.628+
0.654F
0.711f
0.4871
0.435=
0.717
0.6551
0.603F
0.161
0.8481
Commensense
0.7807
0.659:
0.721
0.727f
0.4861
0.324
0.813
0.6911
0,510F
-0,217
0.8871
ROSCOE- 35 with finetuned sup-simese--bberaa-bas sentence embeddings
Somantic Coverage-Chain 0.404
0.364:
0.452f
0,324f
0,214
0.381
0.601
0.3501
0.172
-0.196
0.444f
ROSCOE-SS with mann--pp-ss--222222 sentence embedkings
Semantic Coverage Chain 0.170;
0.171:
0.189;
0,203)
.0.55
0.205
0.237
0.1861
.0.014
0.041
0.243
ROSCOE-S5 with sup-siincse-nbeetoobbn sentence embeddings
Somantic Coverago-Chain 0.411f
0.375:
0.422F
0.460f
0,301
0.4397
0.742
0,381f
0.275
0.148
0,506t
51

---

Published as conference paper ICLR 2023
Table 39; Somers' D correlation of all metrics ESNLI human, judged dataset analyzing step-by-step reasoning overall chain and step-level perspectives. All metrics measured reference-based ht). The highest correlation overall for each aspect on each dataset is bold, second best underlined. Correlation scores with p-value < 0.05 are marked with f. (Continued from 3 6, more details App. H.2)
QUAL
COR
CONMON
FACT
HALL
RED
REP
LOGIC
MADE
GRAM
M1S:
Rouge-1
0.255
-0.234
-0.506
0.072
0.359
-0.078
-0.310:
-0.054
0.6621
Rouge 2
0.223
00.189
-0.629
0.103
0.429
0.081
0.344
0.149
-0.493
0.568
Rouge-L.
0.2271
-0.177
-0.089
40.351
-0.500
0.6281
BLEURT
0.221f
-0.170
-0.197
0.078
-0.090
-0.108
-0.202
0.216
-0.447
0.611f
BERTScore
0.362
-0.036
-0.281
0.182
0.269
0.143
-0.153
0,068
-0,369
0.6611
BARTScure
0.121
-0.344
-0.200
0.003
0.007
0..39
-0.322
0.338
-0478
0.513
BARTScwre+
0.129
-0.113
-0.306
0.164
0.048
-0.036
-0.367+
-0.095
-0.378
0.341
BARTScore-P
0.096
-0.037
-0.244
0.188
-0.017
0.028
-0.222
-0.946
-0.244
0.173
PRISM
0.314
0.032
-0.386
0.086
0.438
0.063
-0.131
-0.378
-0.328
0.6841
CTC-Relevanee
0.072
0,331
0.383
0.034
0.352
-0.001
0.4001
0,405
0.231
0.033
CTC-Consistency
-0.051
0.150
-0.078
-0.130
0.083
0.105
0.085
0.676
0.056
0.006
ROSCOE Metrics (reference-based on (s.4)
ROSCGR SA with Finetuned sup-sinccse-mmerri
embeddings
Hallucination
0. 0.156t
0.152
0.461
-0.011
-0.021
0.160
0.132
0.743
-0.275
0.170
Redundancy
0.234
0.553
0.283
0.119
0.500
-0.253
Semantic Goverage-Siey
0.153
-0.094
0.172
-0.192
-0.241
0.065
-0.020
00730
0.086
0.327
Missing Step
0.234;
D.5581
Reasoning Alignment
0.2781
-0.062
-0.003
0.072
0,100
-0.049
-0.047
-0.108
-0.433
0.495t
Commonsense
0.142
-0.094
-0.353
0.213
0.207
-0.184
-0.188
0.676
-0.447
0.368
ROSCCE-SS with all all--mpee--hunte
sentence embedklings
Hallucination
0.174;
0.094
0.350
0.066
-0.210
.0.021
0.181
0.554
-0.089
0.141
Redunduncy
0.219t
0.227
0.531
0.153
0.031
0.116
0.293
0,405
-0.133
0.133
Semantic Coverage-Stcp
0.1851
-0.108
-0.039
0.099
-0.490
-0.229
-0.185
-0.122
0.083
0.3501
Missing
0.303
-0.173
-0.031
-0.800
0.689
-0.6501
Reasoning Alignment
0.428
-0.028
-0.128
0.294f
0.121
0.095
+0.066
-0.351
-0.5471
0.6571
Commonsense
-0.016
00.71
0.946
ROSCCE-SA SA with wip-simese
moberto base sentence embeddinggs
Hallucination
0.1901
-0.311
0.171
-0.149
0.2721
Redundancy
0.166f
0.177
-0.208
0.249
-0.038
-0.007
0,023
-0.405
-0.351
0.235t
Semantic Coverage-Step
0.196
-0,235
-0.058
0.167
-0.341
-0.187
-0.153
-0.338
-0.078
0.425
Missing Step
0.307
0.165
-0.508
0.143
0.124
0.105
00.199
0.405
-0.558
0.623
Reasoning Alignment
0.374;
-0.049
-0.406
0.317
0.114
0.024
-0.148
-0.608
40.603+
0.642F
Commonsense
0.197t
0.004
-0.467
0.087
0.352
0.021
-0.047
0.919
-0.464
0.3891
ROSCOE S5 with finetuned sup-sinesse-robert-bbee sentence embeddings
Semantic Coverage-Chain 0.152;
-0.185
-0.356
0.266
-0.021
-0.311
-0.2557
0,500
0.022
0.2871
ROSCCE-SS SS with all mapee--uue-ett sentence embedklngg
Semantic Coverage Chain 0.320) -0.213
0.203
0.045
-0.066
0.013
-0309
0.446
.0.117
0.679
ROSCOE with sup-simecse-mbeertooee
embeddings
Semantic Coverage-Chain 0.339t -0.196
-0.378
0.269
0.072
-0.088
-0.148
0,324
-0.5391
0.6431
52

---

Claim supported Published as a conference paper ICLR 2023
Table 40: ROSCOE performance analysis on examples from Human Judged datasets. Errors are highlighted in red. (Cont. from $ 6)
Dataset
Context
Reasoning chain
Score type:
Comment
Score value:
Errors
DROP
Over the next year, however, the Polish forces were subject to attrition, the Sejm again refused to raise taxes and pay the army. result- ing in mass desertions of unpaid soldiery. The Polish problems were further aggravuted by the incompetent leadership hetman Michal Kuzimierz Pac, who obstructed Sobieski's 's leadership, while the Ottomans continued to receive reinforcements. Nonetheless in 1674 the Commonwealth resumed the offensive, taking advantage of new Muscovy- Ottoomn conflict that year, and the Polish-Ottoman war remained undecided. Sobieski's 's force of 6,000 defeated 20,000 Turks and Tatars der Ibrahim Shyshman in the battlc of Lwow August 1675. Even after the Battle Trembowla, the Sejm still refused his plens for more funds and larger arny. In 1676, after Sohieski's 's 16,000 withstood the two- week siege of Zurawno, by 100,000 men un- der Ihrahim Pasha, new peace treaty was signed, the Treaty of Zurawno. The peace treary partially reversing those from Buczacz: the Ottomans kept upproximately of the territories they gained in 1672, and the Commonweallhh no longer was obliged to pay any kind tribute to the Empire; a large number of Polish prisoners the Ottomans. How muny the difference beween Sobieski's force and the Turks and Tatars? Claim: 14000. by the Situation?
6000-20000=14000 The answer is yes.
Info-Step:
Common Sense Er- Model should subtract smaller value (6,000) from bigger (20,000) to the ques- tion. Arithmetic error: 6,000 minus 20,000 -14,000.
0.514;
ror:
COMMON.
MATH
thirds
released
DROP
The first Azov campaign hegan in the spring of 1695. Peter the Great ordered his anmy to advance towards Azov. The army comprised crack regiments and the Don Cossacks and was divided into three units under the com- mand of Franz Lefort, Patrick Gordon and Avtonom Golovin. Supplics down the Don from Voronezh.In In 1693 the Ot- toman garrison of the fortress was 3,656, of whom 2,272 were Janissaries. Between 27-July the Russians blocked Azov from land hut could not control the river and pre- vent resupply. After two unsuccessful uttacks August and September lifted on October 1. Another Russian army under the command of Boris Sheremetev set out for the lower reaches of the Dnieper to take the Ottoman forts there. The main fort at Gazi- Gazi-Kerman Kermun was taken when its powder maguzine blew up. as well as Kaam-Kemaan. Tagan and Tavan, but the Russians were not of
The first Azov campaign hegan in the spring of 1695, Peter the Great ordered his army advance towards Azov. The army comprised crack regiments and the Don Cossacks and was divided into three units under the command Franz Lefort, Patrick Gordon and Avtonom Golovin. Supplies were shipped down the Don from Voronezh In 1693 the Ottoman garrison of fortress was 3,656, of whom 2.272 saries, Between June 27-July 5 the Rus- sians blocked Azov from land but could not control the and prevent resup- ply. After two unsuccessful attacks on August 5 and September 25, the siege was lifted October 1. Another Rus- army under the command Boris Sheremetev set out for the lower reaches of the Dnieper to take the Ottoman forts there. The main taken when its powder maguzine hiew as well as Islam- Kerman.
Info-Step: 0.966, RED
Redundancy:
Model repeats sentences from context that directly relevant to the question and can be dropped. Info-Step score is however high, be- cause measures two-way similurity hetween context and reasoning chain, thus being not informative.
shipped
Janis-
the siege was
at Gazi- Kerman
able hold the and withdrew their forces. By the Treaty of Constantinople the remaining Russians were withdrawn and the lower Dnieper was declared demilita- rized zone. What happened first: Russians blocked Azov or Treaty of Constantinople? Claim: Russians blocked Azov. Is the Claim supported by the Situation?
53

---

Published as conference paper ICLR 2023
Dataset
Context
Reasoning chain
Score type:
Comment
value;
Errors
GSM8K
John has 3 boxes, Each box is 5 inches by 6 inches by inches. The walls are inch thick. What is the total inner volume of boxes?
Each box 4==11= 20. 120 cubic inches. So they have to- of inches. So the total vol- ume is 360*1=>360" 1=0=06001==000660 bic inches. Answer: 360
Self- Consistency:
There only 2 Repetition croors in the dataset, have Self-Consistency below 0.362. Most the other chains higher scores, thus Somers' D relation score for Colf-Consistencyy and Repetition is GSM8K dataset.
cu-
0,361, REP
GSM8K
Mary is an avid gandener. Yesterday, she received 18 new potted plants from her fa- pot-
Mary has 2 potted plants on each of 40 window ledges, for a total <2*40=80=80 potted plants. She has 18 new potted plants, for total of 80+18 480+ 180+18898-999 pot- plants, She giving away pot- plant from each of the 40 win- dow ledges, for total of 1+40 #400=400400 plants. She will he re- maining with 498-40=58.58 potted plants. Answer: 58
Repetition-
This is an example of chain that was scored low ally have repetitions, be- cause its steps do sound similar. Two chains with actual repetition rors were scored below 0.001. total, out chains without repetition error where scored as low chains with repetition cror, re- sulting high Somers' D score.
Step: 0.001;
plant nursery. She already ted plants on each of the 40 window ledges of her large country home. Foeling gener- ous, has decided that she will give potted plant from each ledge friends and family tomorrow, How many potted plants will Mary remain with?
No
actu-
GSMSK
Rory orders subs for $7.30 each, 2 bags of chips for $1.50 each and cookies for ..00 each delivery. There's 20% delivery fee added check our and she wants to add a $5.00 tip. What will her delivery order cost?
The order is $7. 30 for the sub, $1.50 chips and $1.00 for cookies so total order is 7.50+1.50+100 $x7.50+ 100==000000000 There's 's a 20% delivery fee added at check out so that's 10a 20 $=10* 20=2.002.00. The order $10.00 and there's $2.00 delivery fee so 1042 Sw 12200=12000012 She also wants add $5.00 tip which will make the order 12+5 $- s=12+5=17.00 17.00007.0000 Answer: 17
Faithfulness-
There was no corre lation (Somers' score 0.014) found between FACT error and Faithfidnness-tep score. The range where the was and was not present is the same (0.60, 0.81). We provide example of the chain with FACT error that was scored among the highest.
Token:
0,801: FACT
ESNLI
People standing beer curry line. Claim: People standing in line. the Claim supported by the Situatio??
The people are standing in lime Source- beer and curry, not standing Consistency: line. The answer is No.
a clear contradiction with a context, this chains gets high Source- Courre-Cossistecyy score. Overall, there is a large number of fulse-positives (with high 0.98), and felse-negatives (with scores as low 0.0) resulting moderate correlation score of 0.425.
Although
0.976; FACT
54

---

Published as conference paper ICLR 2023
Dataset
Context
Reasoning chain
Score type:
Comment
Score value:
Errors
ESNLI
Two women are embracing while holding go packages. Claim: The men fighting outside deli. Is the Claim supported by the Situation?
Two women are embracing while holding to go packages means they are friends family. They not fighting. The answer is No.
Self-
Self-Consistency mea- sures steps dict each other. LOGIC identifies if there is logical deduction er- within of the steps. Nevertheless, Self CeffCoosiiseecyy able catch some logical deduction eroos result- ing in moderate corre lation, since contradic- tions within step often lead to logical errors be- tween steps.
Consisteney:
0.079:
LOGIC
SemEval
spray some Pam into trying pan. chop up small pieces of ham and put them the frying I turn the stove on low heat. crack two eggs into bowl. I add little water whisk eggs until they completely mixed up. pour the egg mixture into the pan. turn the heat medium. put a lid on top of the pan and continue cook
The answer is incorrect, because there mention family. The only people mentioned are the person cook- ing the omelet and the nurse.
Fathhudness- Step: 0.776:
This chain was given lower that 25% of Faithfulhess-- scores within this dataset. See- step misaligned with the context, con- taining irrelevant infor- mation, that resulted in punishment.
HALL
mixture. After few minutes. I check see if all the liquid top of the egg has cooked so it isn't runny anymore.
completely cooked. little shredded cheese on top of the egg mixture. I cook for few minutes until the cheese melts. I get spatula and flip half egg on top of the other half. Then my omelet cooked! Who all the break- fast? Claim: The family.
SemLival
had bought my dad tie for lather's day. I'm sure I could have given it him that, but thought
The answer is incorrect, because it was Father's Day.
Fatthfuness-
Fatthfulnes--ie score low, although reasoning is correct. Model failed to produce higher score because cosine similarity- alignment scores low.
Step: 0.635;
touch would have been wrup it. couldn't wrap the tie like that, so bought small cardhoard box. narrow and the perfect size for tic. put the tie in the box and taped it shut. had bought some blue and white wrapping paper and some tape, the box. I the box top the paper and folded over. cut the excess of the paper off and tuped the paper to the box. I folded all sides the paper to tuck package, When I was done, put a nametag on it. topped off the gift with a small blue ribbon how. What is the reason the present? Claim: was Christmas.
No error
set besides
55

---

Published as conference paper ICLR 2023
Table 41: ROSCOE performance analysis examples from Diagnostic datasets, Perturbations are highlighted in red. (Cont. from 6)
Dataset
Context
Reasoning chain
Score type:
Comment
Score value;
Perturbation
ProofWriter
Charlic is furry. Erin is furry. Erin
The bear visits the cow.
Faithfulness-
Faithfulness-See scores vary from 0.5 to 1.0. Metric able to catch hallucination and gives the chain low score.
green. Erin is round. Fiona bluc. Fiona furry. Gary round. If something furry and blue then is white. If something is blue and round it red, All furry things are blue. Charlie red then Churlie green. Is the following True, False, or Unknown? Erin is round.
is round.
Step: 0.602;
White things
Halucination
Proo/Writer
not young. Bob green. Fiona is cold. Fiona is rough. Gary is red. Gary is young. Green, rough things are red. Gary cold and Gary then Gary is green. If something nice then it is sough. Cold, rough things kind. something is cold and nice then is young. If something kind cold then it is green. All red things the following True. False, Unknown? Gary is not nice.
Anne
Gary red. All things nice. Gary
Faithfulness-
Fairhfulness-Step scores chains without perturbations vary from 0.740 to 0.990. Even though first two sentences are aligned with the context, last sentence in the chain lowers overall average.
Step: 0.740;
is nice.
No perturha-
tion
EQASC
What insect goes through brood process which the larval and pupal steps com- bined. Answer: An ant has a brood which is the larval and pupal process of the ant
Most popal steps go through larval stages after hatching. Brood are the larval and pupal stages of the ant.
Grammar:
In this subset, Granoaa scores vary between 0.258 and 0.996. To score grammatical correctness used pre-trained model, that also punishes chains that sound unnat- ural, like this example. As sult, we moderate correlations hetween Grammuar scores and Se- mantic Error perturbation.
0,258 ;
mantic Error
after hatching
EQASC
What uses beak for cutching prey? An- swer: beak used for catching prey hy eagles.
A beak is used Grommar: 0.529 ; No
correctly identifies grammatical issue. Al- though no perturbation was not added this chain, reference itself contained an error.
The Grammar
catching prey by some birds, Eagles are big bird.
perturbation
EntailmentBank
A different moon phuse occurs upproxi- mately once per week. The first quarter phuse of moon occurs after the new For science activity, Mia had to draw picture of the Moon euch night. If Mia drew new moon, what type of moon did Mia druw about week later? An- swer: first quarter
A different moon phase doesn't 't occur approxi- mately once per week. The first quarter phase of the moon occurs after the new moon. Therefore the first quar- ter phase of the moon will occar one week af- the new moon
Info-Chain: 0.542; Negate
In this subset, Info-Chait Chain scores calculated using finetuned sup- finese-robertooeee belong interval (0.530: 0.999), with 92% of perturbed instances having scores then 0.92, and 95% of non-perturbed instance that scored higher. Even though Info- Choin embeds chuin as whole, turned out to very sensitive to negation perturbations, especially when the model is finetuned, re- sulting in high Somers' D correla- tion of 0.955.
step
56

---

Published as conference paper ICLR 2023
Dataset
Context
Reasoning chain
type:
Comment
Score value;
Perturbation
EntallmenBBank
Humting decreases animal pupulations. If a population decreases zero then organism
Hunting decreases imal populations.
Info-Chain:
In this cuse, negation occurs in the last step of chain, which represents conclusion and does not overlap in wording with the context, and goes undetected by metric.
0.983; Negate
kind organism. Many animals still being hunted for their fur. Because of this, many these animals are danger of. Answer: extinction.
population decreases zero then that organism is extinct. An animal is kind organism. Therefore if hunting creases the animal pop- ulation to zero, then the animal will not be ex- tinct.
step
MATH
What 1 1
One raise to any power
Grammar:
Pre-trained model, used to grammatical errors, gives scores for simple chains. In this subset, Grammar scores belong to interval (0.053: 0,990).
so
0.053 Gram-
matical Error
MATH
Two fair eight-sided dice have their faces numbered from I to 8. What is the ex- pected value of the sum of the rolls of both dice?
expected value of double roll can simply add the ex- pected values dividual rolls, , giving
Grammar: 0.982: Gram- matical Error
example of chain where the classifier model failed to catch the grammar emror (dropped verb "find"). Even though the model able to identify some grammatical errors, Granour score distributions over perturbed unperturbed data overlap significantly. with mean 0.909 and variance 0.005 for un- perturbed subset, and 0.854 mean and 0.014 for perturbed. sult, the correlation score 0.392, which is quite low.
This
+ 4.5
ASDIV
The puppet show begins at 2:00 p.m. lasts hour and 30 minutes. What time does it end?
(hour)
Info-Step:
In subset. Info-Step varies be- 0.005 and 0.796. flagging most of the chains with shuffled steps. Scores are distributed with mean 0.356 and variance 0.056 for perturbed, and 0.705 and vari- ance 0.004 for unperturbed chains. Resulting Somers' correlation 0.879.
(min+hour) (hour)
0.005: Shuffle
60 (min hhour) 30
Operations
(min)) 60 (min/hour)
r30; hhrmm p.m.
ASDIV
I have 28 cents in my pocket. How coins could have on minimization?
28/25=1 1/3=4
r3; 3+1=3;
Info-Step:
example of chain where perturbation in operations got higher scores by the model.
This
0.705: Shuffle
Operations
ASDIV
has $10, She spends on loy truck und pencil How much money does she have left?
10-(3+2)=5
Info-Step:
This an example of chain scored lower spectrum by the Info-Step spectrum, although had no perturbation or other type enor included.
0,561; perturbation
No
57


Reviews:
[{"review_id": "v2wXOZ8x_L", "reviewer": "Reviewer_9E3Y", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduce a new metric (ROSCOE) for the generic step-by-step reasoning task. In particular, ROSCOE includes four fine-grained metrics under four perspectives: *semantic alignment*, *semantic similarity*, *logical inference* and *language coherence*, where each fine-grained metric further contains a bunch of specific metrics. The authors show the effects of the prosed the metric on both synthetic and human-labeled datasets.", "summary_of_the_review": "This paper present a new ROSCOE,  a suite of automatic scores for evaluate the step-by-step reasoning performance. However, the proposed new taxonomy has not be justified, and whether the good ROSCOE scores could generate a good performance on end-task performance has not been validated.", "strength_and_weaknesses": "Strengths:\n\n- The paper is generally well-written and easy to follow.\n- The idea of evaluate the quality of step-by-step reasoning is novel and could benefit a lot of end applications.\n\nWeaknesses:\n\n- Could you justify that the new taxonomy of generic reasoning errors can cover all the possible reasoning errors? For Reasoning Errors, at least, negation should be considered. E.g. \"Anakin didn't catch 10 fish.\"\n- The authors argue that \"The existing works primarily focus on end-task performance.\" However, it is unclear if the good performance of reasoning rationales (by the proposed ROSCOE) means a good performance on end-task performance.\n- Many presentation logics should be improved. For example section 3, \"To evaluate ROSCOE,\", however, it is unclear what exactly “ROSCOE” is. And \"ROSCOE\" is itself a metric, how could we evaluate a metric?\n- The authors propose four fine-grained metrics under four perspectives: *semantic alignment*, *semantic similarity*, *logical inference* and *language coherence*, where each  fine-grained metric further contains a bunch of specific metrics. However, it is unclear how these specific metrics are merged into one scalar value.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IPFguoKr5X", "reviewer": "Reviewer_UUD9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new reference-free metric named ROSCOE for the step-by-step rationales that some LLMs use for reasoning tasks. The ROSCOE is a suite of metrics that cover four dimensions of step-by-step rationales (or logic chains): 1) semantic alignment, 2) semantic similarity, 3) logical inference, and 4) language coherence. The authors use five natural datasets and six diagnostic datasets for justifying the effectiveness of the proposed ROSCOE metrics. The experiments and analysis show that ROSCOE outperforms other baseline metrics such as CTC, BARTScore, and BLEURT.\n\n", "summary_of_the_review": "Generally, a solid paper on a novel metric, and it has a comprehensive analysis. The studied problem is pretty important and the empirical results are promising. The implication part is relatively weak as shown in the weakness and suggestion sections. ", "strength_and_weaknesses": "## Strength\n\n- a comprehensive and novel metric for an important problem --- to understand the quality of the step-by-step rationale (the so-called chain of thoughts). \n\n- very extensive experiments that compare the ROSCOE and other metrics; the reasonable choices of data and evaluation methods, and informative analysis of the behavior of ROSCOE\n\n- a nice taxonomy of reasoning errors for analysis with perturbation in building diagnostic datasets.\n\n## Weakness\n\n- lack of justification for the design of many details in ROSCOE.  \n- Sec 4 (especially 4.1) is hard to read. It's not a great idea to simply list all these sub-metrics without clear motivation and illustrative examples.  \n- the implications of the proposed metric are relatively weak -- how to improve the rationales? what are the disadvantages of existing step-by-step rationales? these questions are not touched on. \n\n## Suggestions: \n- think about how future people should use this ROSCOE metric easily and for more general cases.\n- can we use the ROSCOE analysis to guide the future design of the step-by-step rationale \n- what are the limitations of the proposed metrics so that people can keep that in mind when using the metrics\n- what will be the methods to avoid the above limitation and fix them?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BXSwu06OzXT", "reviewer": "Reviewer_mtZF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a large number of metrics to evaluate step-by-step reasoning traces. They learn an embedding to evaluate similarity between steps. Authors show results on a very large and varied set of datasets.", "summary_of_the_review": "This is a very promising first step towards evaluating reasoning traces. My main concerns are as follows:\n(1) There is insufficient systematic analysis of the proposed metrics. It would be better to propose all metrics, apply relevant perturbations and show which metrics most accurately capture the types of error they are designed for. Following this, the authors could rank the metrics in-terms of sensitivity or  suggest the top N most sensitive metrics going forward.\n(2) It is not clear how the alignment process works, especially when aligning hypothesis statements with those in the source. This will also depend significantly on the model used to embed the sentences. The authors have also only shown evaluation on the same datasets that the metric was trained on.\n\nI would be inclined to accept this paper, since it takes a very important (early) step in the right direction. It is also great that the weights for their embedding model will be available as well as their code. However, my recommendation to accept would be contingent on the authors being clear about the limitations of their approach and adding a clearly labelled section to their paper discussing these. They must also run a more systematic analysis of the perturbations and show how these affect the metrics.", "strength_and_weaknesses": "Strengths:\n1. The authors evaluate their metrics on a large set of varied datasets with all kinds of reasoning: logical, natural and synthetic language and math.\n2. This is an incredibly challenging problem and any steps towards evaluating step-by-step traces are great.\n\n\nWeaknesses:\n1. The authors propose a lot of metrics but have not sufficiently showed the sensitivity of each metric.\n2. The method depends on having a very good similarity metrics.\n3. It is not clear that the alignment metrics make sense (for example between source and hypothesis statements).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9MNVT9zBzB", "reviewer": "Reviewer_MQPy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors collect data about rationale used in reasoning tasks. It includes two major datasets: \"diagnostics\" which includes human generated gold-reference rationale, and \"judgements\" which include machine generated rationale with human judged errors (see table 7 in the appendix for dataset statistics). Their suite of metrics is based off this data, and designed to measure the effectiveness of newly generated rationale. The metrics mostly are intrinsic, and don't account for downstream task performance that I can tell.\n\nRegarding the rationale and metrics, I could be missing a crucial detail... It is mentioned in sec3 that the final step in the chain is the answer, but for the most part, I have interpreted this work as if the chain does not include the answer. Since some of the metrics would not make sense if the last step is the final answer.", "summary_of_the_review": "I see the contribution of the new data as a strong one. Although I don't find the suite of metrics convincing, the value of the dataset still warrants acceptance. Plus, I consider the suite of metrics as an example of what can be accomplished once this data is available. I would probably give a stronger review if the data is further explored in the main text, but I am assuming the data has intrinsic value because it is human generated and am still unsure if it can be used for good downstream performance.", "strength_and_weaknesses": "# Strengths\n\n* Substantial datasets including human-generated rationale. Also includes machine-generated rationale annotated with errors by humans.\n\n* The dataset will enable new work to leverage the data directly to improve performance or create more rationale. But this is not all. We can use this data to measure the qualities of what is considered human-like rationale, and that is what this paper does by presenting a suite of metrics along with their analysis.\n\n# Weaknesses\n\n* Not nearly enough analysis and exploration of the new data. For example, do these rationale lead to improved (or worse) performance? Or categorizing different types of rationale and their properties.\n\n* It should be made more clear what are the assumptions about rationale, e.g. do we expect there to be only one valid reasoning chain?\n\n* There are clear downsides to the introduced metrics. The major one being I am not convinced they would recognize an equally good but different looking rationale. I don't think this is something the baseline metrics would be capable of doing, but also not convinced ROSCOE could do either. Perhaps challenging qualitative results would help. Although evaluating different types of rationale on a downstream task would be most informative.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6OywP5-561S", "reviewer": "Reviewer_8arD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors did a very good job analyzing chain-of-though inferences for several crucial reasoning datasets and constructing their own Human Judged Datasets. ", "summary_of_the_review": "The authors did a fantastic job analyzing chain-of-though inferences for several crucial reasoning datasets and constructing their own Human Judged Datasets. The impact of the metrics could be made more evident.", "strength_and_weaknesses": "The authors introduced 18 custom metrics intended to measure the quality of reasoning. The main strength is the breadth of the analysis. I am less sure about the uses of these new metrics. A few outstanding questions: \n\n1. How are human solutions performing against correct and incorrect machine-generated answers?\n2. Can you extend information about the Human Judged Datasets, including the scale and cost of work and the intent to open-source it?\n3. Can you show more examples of errors in the appendix or provide access to a browser of samples (e.g., similar to https://minerva-demo.github.io/)? \n4. It would be interesting to see a scale of errors, that is, samples for the same problem that are correct according to the metrics, somewhat correct, somewhat incorrect, and completely incorrect.\n5. The elephant in the room is the usefulness of these new evaluation metrics for downstream tasks. Does it help with re-reranking? Can we get better scores than via majority voting?\n6. Nit: Can you explain how the frequent 1.0 score in the tables should be interpreted?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%