Venue: NAACL 2024

_What if you said that differently?_: How Explanation Formats Affect Human Feedback Efficacy and User Perception

 Chaitanya Malaviya, Subin Lee, Dan Roth, Mark Yatskar

University of Pennsylvania

{cmalaviy,subinlee,danroth,myatskar}@upenn.edu

###### Abstract

Eliciting feedback from end users of NLP models can be beneficial for improving models. However, _how should we present model responses to users so they are most amenable to be corrected from user feedback?_ Further, what properties do users value to understand and trust responses? We answer these questions by analyzing the effect of rationales (or explanations) generated by QA models to support their answers.

We specifically consider decomposed QA models that first extract an intermediate rationale based on a context and a question and then use solely this rationale to answer the question. A rationale outlines the approach followed by the model to answer the question. Our work considers various formats of these rationales that vary according to well-defined properties of interest. We sample rationales from language models using few-shot prompting for two datasets, and then perform two user studies. First, we present users with incorrect answers and corresponding rationales in various formats and ask them to provide natural language feedback to revise the rationale. We then measure the effectiveness of this feedback in patching these rationales through in-context learning. The second study evaluates how well different rationale formats enable users to understand and trust model answers, when they are correct. We find that rationale formats significantly affect how easy it is (1) for users to give feedback for rationales, and (2) for models to subsequently execute this feedback. In addition, formats with attributions to the context and in-depth reasoning significantly enhance user-reported understanding and trust of model outputs.1

Footnote 1: Code and data is available at [https://github.com/chaitanyamalaviya/rationale_formats](https://github.com/chaitanyamalaviya/rationale_formats).

0
Footnote 1: Code and data is available at [https://github.com/chaitanyamakavisja/rationale_forms](https://github.com/chaitanyamakavisja/rationale_forms).

0
Footnote 2: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics:

Human Language Technologies (Volume 1: Long Papers), pages 3046–3065

June 16-21, 2024 0204 Association for Computational Linguistics

## 1 Introduction

Question answering models can often be black boxes, as their reasoning process is mostly opaque to model builders as well as end users. This can inhibit the ability of users to provide helpful critiques to models to repair them. Generating _rationales_ (or explanations) along with answers is a viable approach that can alleviate these concerns, but these rationales are inherently not faithful and can sometimes be inconsistent with the answers themselves Ye and Durrett (2022); Turpin et al. (2023); Lanham et al. (2023); Radhakrishnan et al. (2023).

This motivates approaches that decompose the question answering task into two stages (depicted with dashed lines in Figure 1), where we first generate a rationale for the question using the given context (\(X2R\)), then use only this rationale to answer the question (\(R2Y\)) Lei et al. (2016); Eisenstein et al. (2022). A rationale may provide a justification for the answer by presenting an outline for how the question can be answered. By only relying

Figure 1: The framework for incorporating human feedback into decomposed QA models. A model \(X2R\) generates a rationale \(R\) to answer a question based on a passage. A human teacher then provides natural language feedback for \(R\), which is used to generate a revised rationale \(R^{\prime}\) from \(F2R^{\prime}\). Finally, this revised rationale is used to generate the final answer \(Y\). We study various formats of the intermediate rationale \(R\).

 

on the rationale as context, the answer generation model (\(R2Y\)) has a stronger inductive bias to generate an answer that is faithful to the rationale.

Crucially, faithful rationales can allow users to follow the model's line of reasoning, and subsequently provide actionable feedback to the model. This feedback can be used to repair individual outputs or enable generalization to novel instances. However, it is unclear precisely how a rationale should be formatted, i) to best aid the user's understanding of the model's reasoning, and ii) their ability to provide feedback for the response.

Our work specifically addresses the question of how we can format intermediate rationales (\(R\)) for decomposed QA systems, such that they are _easy to repair through human feedback_. Further, we analyze what properties make rationales _interpretable_, and _trustworthy_ to users. Previous work on decomposed question answering mostly consider rationales as text snippets extracted from the context, optionally marked up with coreferences that make the snippets standalone DeYoung et al. (2020); Eisenstein et al. (2022). Although extractive snippets can serve useful for providing minimal context that rationalizes an answer, they do not provide much insight into the model's reasoning process. This may limit a user's understanding and their ability to critique the model. We consider alternative formats of rationales, which vary according to well-defined characteristics (SS3). Examples of these formats and how they vary are presented in Table 1.

Based on the considered rationale formats, we generate responses (rationales & answers) from a decomposed QA system. We then perform two user studies where we measure i) the effectiveness of user feedback in patching rationales in different formats and ii) the ability of different formats to enable users to understand and trust responses. In our first study, we sample _incorrect_ answers corresponding to all rationale formats, and ask annotators to provide natural language feedback for the rationales (SS5). We use this feedback to then revise the rationales (\(F2R^{\prime}\)) and regenerate the final answer (\(R^{\prime}2Y\)). Comparing the regeneration accuracy with different rationale formats provides insight into properties of rationales that make them easy to repair. Further, they provide an upper bound for how much improvement can be expected through automated feedback by repairing rationales Chen et al. (2023); Madaan et al. (2023). In our second study, we elicit judgements of interpretability and trustworthiness for _correct_ answers and their accompanying rationales from users (SS6).

We consider two tasks: general reading comprehension Quoref; Dasigi et al. (2019)) and medical reading comprehension PubMedQA; Jin et al. (2019)). Each block in our overall workflow (Figure 1) is implemented through few-shot prompting of a large language model. Our experiments suggest that rationale formats significantly affect i) users' ease of providing feedback and ii) the model's ability to execute that feedback. In addition to being critiquable, certain rationale formats are more helpful in aiding users to understand and trust the model's answers. One such format is the _annotated report_, which includes a list of extractive phrases and free-text inferences based on each phrase. Finally, among a few characteristics of rationales presented to users, we find that users rate attribution and depth of reasoning as characteristics that are most important to them.

## 2 Problem Formulation

Consider a standard reading comprehension task, where we are given a passage, a question based on this passage, and a reference answer \(Y\) as a labeled example in our dataset. We represent the input information in the task (the passage and the question) with \(X\). Typically, we would train a model \(X2Y\) that predicts \(Y\) given \(X\) by learning \(P(Y|X)\). Assuming \(X2Y\) is a black-box model, without loss of generality, the model may internally compute a latent representation, which is usually not extractable in an interpretable format, from \(X\) to predict \(Y\), internally decomposing the problem. This can restrict the transparency of the model because we cannot provide a faithful reasoning to an end user that supports the answer.

### Decomposed QA Pipeline

In the decomposed QA pipeline, we factor the QA problem in the following manner (see Figure 1 for an illustration):

\[P(Y,R|X)=P(Y|R,X)P(R|X) \tag{1}\]

Since Y is independent of X given R (assume Q is part of R for simplicity), we have,

\[P(Y,R|X)=P(Y|R)P(R|X) \tag{2}\]

Let's first consider a rationale-generating model \(X2R\), that generates a textual rationale \(R\) given \(X\) 

This rationale provides an outline of the approach proposed by the model to answer the question. Consider also an answer-generating model \(R2Y\) that generates an answer \(\hat{Y}\) given the predicted rationale \(R\). In this decomposed model, \(R2Y\) has a strong inductive bias to use the information presented in \(R\) for its reasoning. Further, \(R\) can be explicitly shown to an end user, which increases the transparency of the entire system.

### Debugging the Decomposed QA Pipeline

Next, let's assume a set of spans \(X_{s}\) from \(X\) that are sufficient to predict the answer \(Y\). First, we note that for the answer to be correct, i.e. \(\hat{Y}=Y\), the predicted rationale \(R\) must contain all the information contained in \(X_{s}\), i.e., \(I(X_{s})\). Errors in answers generated by \(R2Y\) can be a result of (1) insufficient or incorrect context, when \(I(X_{s})\not\subseteq I(R)\), and / or (2) limited model capacity of \(R2Y\), when \(I(X_{s})\subseteq I(R)\) . Repairing the modeling pipeline (i.e., \(X2R+R2Y\)) can hence either involve improving the quality of generated rationales \(R\) produced by \(X2R\) or improving the modeling capacity of \(R2Y\). Note that the example may require information beyond what is in the passage (for example, domain knowledge or commonsense knowledge).

We consider the scenario where we repair the rationales generated by \(X2R\) through natural language feedback. We assume a teacher \(T\) who writes feedback for generated rationales \(R\), where they describe flaws in \(R\). Generated rationales can be lacking in various ways: 1) insufficient information: \(R\) may not contain crucial information required to perform the inference (i.e., \(I(X_{s})\not\subseteq I(R)\)), or 2)incorrect information: \(R\) may contain hallucinated content or incorrect reasoning chains that could mislead the answer-generating model.

The teacher \(T\) in our case is an end user, who could optionally be a domain expert depending on the task. We evaluate whether the format of predicted rationales \(R\) is interpretable and easy to repair for \(T\). Based on \(T\)'s feedback \(F_{k}\) for a subset of examples \(X_{k}\subset X\), we revise the initial rationale \(R\) to generate a revised rationale \(R^{\prime}\) using a model \(F2R^{\prime}\). This revised rationale is then used by \(R^{\prime}2Y\) to generate the final answer.

## 3 Intermediate Rationale Formats

Rationales in NLP tasks are usually presented as compressed text snippets extracted from the given input DeYoung et al. (2020); Eisenstein et al. (2022). However, text snippets from the context alone may not make the model's reasoning explicit and transparent to users. We consider alternative rationale formats that describe the model's reasoning. We describe these formats below. A summary of these formats and how they vary according to rationale properties, is given in Table 1.

### Rationale Formats

Markup-and-Mask (markup_mask).This format, proposed by Eisenstein et al. (2022), extracts sentences from the context that are relevant to answering the question. Sentences are decontextualized by markups that resolve coreferences and other ambiguous phrases Choi et al. (2021).

Figure 2: Examples of the different rationale formats considered for representing intermediate rationales.

 

Annotated Report (annotated_report).The annotated report extracts phrases from the context and generates a free-text inference based on each phrase that is relevant to answering the question. This is broadly inspired by the way readers highlight and annotate key spans in documents while reading (also found as marginalia in books).

Procedural (procedural).A procedural rationale is a step-by-step plan that uses predefined operations to answer the question. Similar works that broadly propose a plan-based rationale have been explored in prior work, in different contexts Sun et al. (2023); Wang et al. (2023). The primitive operations we consider include an operation to extract relevant sentences, disambiguate an entity from the question or the plan so far, and locate an entity by answering a subquestion. These are further described in Appendix A.

Subquestions (subquestions).Subquestions simply decompose the original question into multiple questions that provide relevant information to answer the question. These have been explored as a form of rationale in various works Geva et al. (2021); Khot et al. (2021); Press et al. (2022); Dua et al. (2022); Zhou et al. (2023).

Decision Tree (decision_tree).We also consider a tree-structured rationale, inspired by fast-and-frugal trees Martignon et al. (2003) as well as prompting work that explores tree-like structures Yao et al. (2023). This format decomposes the original question into Yes/No subquestions in a tree-like structure and also shows the incorrect tree traversals for completeness.

## 4 Experimental Setup

### Datasets

We consider two datasets for our studies: Quoref Dasigi et al. (2019) and PubMedQA Jin et al. (2019). The first is a general reading comprehension dataset while the second involves medical reading comprehension. In contrast to Quoref, PubMedQA often requires domain-specific knowledge for answering the question. We use all validation set examples of Quorer (2418 examples) and all labeled examples in PubMedQA (1000 examples)

### Sampling Rationales and Answers

We sample rationales and answers for all 5 formats in a decomposed QA pipeline, where both \(X2R\) and \(R2Y\) are implemented using few-shot prompting. We first prompt gpt-3.5-turbo for rationales by providing the passage and question. The question and only the generated rationales are then used to prompt the same model to generate the final answer. In both cases, we sample few-shot exemplars using BM25 Robertson et al. (2009) from a set of 100 manually labeled examples with rationales. We sample as many exemplars as can fit within the maximum sequence length (4096) of the model. This usually amounts to 3-5 exemplars. The prompts used and other hyperparameters are provided in Appendix A.

## 5 Study 1: Repairing Rationales through Human Feedback

### Setup

In this study, we measure the critiqueability or ease of repair of different rationale formats. This is done by collecting natural language feedback from human annotators for rationales corresponding to incorrect answers. We sample examples for which

\begin{table}
\begin{tabular}{c|l|p{71.1pt} p{71.1pt}|p{71.1pt} p{71.1pt}}
**Format** & **Description** & **Attribution Provided** & **Reasoning Exposed** & **Sequential Reasoning Reasoning** & **Free-test annotations** \\ \hline
**Markup-and-Mask** & Quoted sentences from the context, marked up with conferences for pronouns and ambiguous phrases Quoted phrases from the context and an inference from each phrase Step-by-step plan for solving the question with pre-defined operations Breakdown of the original question into subquestions & ✓ & ✗ & ✗ & ✗ \\
**Conditional** & **Coherence** & **Coherence** & **Coherence** & **Coherence** & **Coherence** \\
**Suspension** & **Coherence** & **Coherence** & **Coherence** & **Coherence** & **Coherence** \\
**DecisionTree** & **Coherence** & **Coherence** & **Coherence** & **Coherence** & **Coherence** \\  & **Coherence** & **Coherence** & **Coherence** & **Coherence** & **Coherence** \\ \end{tabular}
\end{table}
Table 1: Descriptions of the rationale formats considered in our work and the characteristics along which they differ.

 

the decomposed QA pipeline predicts _incorrect answers_ for all 5 rationale formats. In all, we collect 490 feedback statements for Quoref and 555 feedback statements for PubMedQA.2

Footnote 2: This corresponds to 5 rationale formats and 98 examples for Quoref and 111 examples for PubMedQA.

In each annotation task, annotators are shown a single example (question & passage) with all 5 rationale formats and their corresponding answers. This controls for annotator variance and variance across examples. To control for ordering effects, we randomize the order in which the rationale formats are presented to annotators. For each rationale format, annotators are asked to write natural language feedback to repair the rationale.

We use this natural language feedback to generate the revised rationale \(R^{\prime}\). To do this, we prompt gpt-3.5-turbo with the passage, the question, the original rationale, and human-written feedback. Finally, we generate the final answer by few-shot prompting the same model using just the revised rationale and question. Prompts and other hyperparameters are in Appendix A.

### Participants

For this study as well as the study in section 6, we recruit participants through Prolific. Participants are required to be fluent in English and are based primarily in English-speaking countries. For PubMedQA examples, they are required to be working in the healthcare sector. For more details, please see Appendix B.

### Task

To prime annotators for formulating their feedback, we ask them to first evaluate the sufficiency and faithfulness of each rationale to the context. They label these two properties for each rationale format on a Likert scale (the precise descriptions of the options in all Likert-scale questions are in Figure 6).

Sufficiency.Annotators evaluate if the rationale provides enough information to answer the question, without the context. Note that the rationale may contain inaccuracies but still be sufficient. They mark sufficiency as (_Sufficient_, _A bit insufficient_, _Entirely insufficient_).

Faithfulness to context.Next, annotators evaluate whether the rationale accurately draws conclusions from the context without misrepresenting any information. They mark faithfulness on a scale of (_Accurate_, _A bit inaccurate_, _Very inaccurate_).

#### 5.3.1 Feedback

Annotators are asked to formulate natural language feedback that would be most useful in directing the model to the reference correct answer. It could target missing or incorrect information in the rationale, but cannot explicitly reveal the correct answer. Feedback is elicited in multiple steps (examples of feedback written by annotators are in Table 4, 5):

1. **Location of error**: Annotators are required to list the step(s) (or question number) in which the error occurs.
2. **Type of error**: Annotators then identify the type of the error. We show them a few common error types that occur in rationales (for example, insufficient information, irrelevant information, incorrect inferences etc).
3. **Description of error**: Next, annotators use concrete details from the rationale, question & context to provide a description of the error.
4. **Actionable suggestion**: Finally, annotators provide an actionable edit that would repair the rationale, again using concrete details from the rationale, question & context.

Ease of repair.Using annotator feedback, we can measure how amenable each rationale format is for repair. However, this does not reflect annotators' ease of providing feedback for each format. We elicit this directly on a scale of (_Very easy_, _Somewhat easy_, _Somewhat hard_, and _Very hard_).

### Evaluation

We first evaluate the effectiveness of feedback through edit accuracy (edit_acc), where we manually label each revised rationale for whether it incorporates the feedback. Next, we compute final answer accuracy (final_acc), where we check

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & \multicolumn{2}{c}{Quoref} & PubMedQA \\ \cline{2-4}
**Rationale Format** & **EM** & **F1** & **Accuracy** \\ \hline none & 70.31 & 79.65 & 69.30 \\ \hline markup\_mask & 57.44 & 68.10 & 62.20 \\ annotated\_report & 60.26 & 70.20 & 70.20 \\ procedural & 66.09 & 77.05 & 68.30 \\ subquestions & 54.26 & 63.05 & 68.90 \\ decision\_tree & 68.61 & 77.09 & 46.70 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Initial scores using the decomposed QA pipeline (\(X2R\) + \(R2Y\)) for different rationale formats.

 

whether the final answer using the revised rationale is correct. We exclude all instances where the answer was leaked in feedback.

### Results

We first show the results on standard decomposed QA for all rationale formats as well as standard answer generation (without rationales) on both Quoref and PubMedQA in Table 2. These results show that decomposed QA models can be competitive with end-to-end models. Although they slightly underperform standard answer generation for Quoref, decomposed QA is better performing on PubMedQA. This suggests that decomposed QA is a promising modeling approach, while being predisposed to provide more faithful rationales.

Annotator labels of sufficiency and faithfulness (presented in Figure 7) indicate that annotated_report and subquestions have rationales that are most often sufficient for both datasets, while markup_mask tends to lack most with sufficiency. On the other hand, extractive rationales from markup_mask are labeled most faithful for Quoref (58%), while annotated_report is relatively faithful for both datasets.

Our main results for repairing rationales through feedback are in Table 3. For Quoref, we find that rationale formats that expose the reasoning of the model are easier to repair through feedback. Interestingly, stricter formats with well-defined operations (such as procedural) are fairly effective for Quoref. On the other hand, for PubMedQA, rationale formats with more free-text components (such as annotated_report) that can allow more flexible edits are most effective. This is likely because comprehending medical articles and making inferences based on them can involve nuances that are harder to express with strict rationale formats. For example, feedback written for one PubMedQA example (Table 5) mentioned that the rationale didn't consider the fact that the study did not consider a control group for testing their hypothesis. This is easily incorporated into the free-text nature of the annotated_report, but is harder to incorporate in the procedural format.

We also note that when feedback is used to revise rationales that contain attributions, rationales can sometimes misquote sentences from the passage by hallucinating information that does not exist in the context. Although these revisions may result in correct answers, the rationales would be unfaithful, potentially decreasing user trust in the model.

In terms of annotator ease of providing feedback, we find that markup_mask is easiest to provide feedback for, because it may be easy to verbalize when information is missing from the rationale. However, these judgements do not correlate with actual effectiveness of the feedback for rationale repair. This suggests annotator ease of providing feedback may not correlate with actual effectiveness of feedback.

## 6 Study 2: Evaluating User Perception of Rationales

### Setup

In the next study, we measure the extent to which different rationale formats enable users to _understand_ and _trust_ model responses. We sample examples where all 5 rationale formats have corresponding _correct answers_ for both datasets. 40 annotators completed this study for Quoref examples while 44 completed it for PubMedQA examples.

### Task

We collect Likert ratings of interpretability and trustworthiness of rationales. In addition, we col

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{3}{c}{Quoref} & \multicolumn{3}{c}{PubMedQA} \\ \cline{2-6}
**Rationale Format** & **edit\_acc** & **final\_acc** & **time\_taken(s)** & **edit\_acc** & **final\_acc** & **time\_taken(s)** \\ \hline markup\_mask & 50.00 & 29.69\({}^{n|*||*|*}\) & 397.38 & 71.03 & 14.95\({}^{n|*|*|o|}\) & 379.07 \\ annotated\_report & 51.67 & 38.33\({}^{n|o|*|o|*}\) & 381.52 & 62.96 & **20.37\({}^{n|*|*|*}\)** & 426.04 \\ procedural & 57.89 & **38.60\({}^{n|o|o|*}\)** & 359.75 & 55.77 & 8.65\({}^{n|*|*|*}\) & 434.24 \\ subquestions & 49.21 & 36.51\({}^{n|*|*|o}\) & 375.45 & 59.05 & 14.29\({}^{n|*|*|*}\) & 425.14 \\ decision\_tree & 56.25 & 37.50\({}^{n|o|*|o}\) & 397.56 & 69.52 & 17.14\({}^{n|*|*|*}\) & 494.42 \\ \hline \hline \end{tabular}
\end{table}
Table 3: QA accuracy after patching generated rationales with human feedback and regenerating answers. We show here the edit_acc, which is the percentage of examples for which the revised rationale successfully incorporates feedback and final_acc, which measures the final answer accuracy after regeneration with the revised rationale. Statistical significance at \(p<0.1\) is specified with \(*\) (and \(o\) if not significant) with paired bootstrap tests in the order of the remaining rows in the table.

 

lect scalar judgements of the importance of different characteristics of rationales for annotators. Descriptions of Likert-scale options are in Figure 6.

Interpretability.A rationale should facilitate in making the model's reasoning more transparent to an end user. This is measured by asking annotators how beneficial the rationale is in helping them understand the reasoning process followed by the model. It is elicited on a scale of (_Very beneficial_, _A bit beneficial_, _Not beneficial at all_).

Trustworthiness.In addition to improving user understanding, a rationale that makes a model's decision-making transparent should do so in a way that helps users trust model responses. We ask annotators how likely they are to trust the model's answer, if the rationale was provided along with the answer. The rating is elicited on a scale of (_Very likely_, _A bit likely_, _A bit unlikely_, _Not likely at all_).

Scalar judgements.Next, we directly ask annotators for characteristics they value in rationales. They rate the following predefined rationale properties on a scale of 1-5 based on their importance:

* [leftmargin=0pt]
* _Attribution_: Includes quotes from the context.
* _Depth of reasoning_: Provides detailed insight into the reasoning process.
* _Sequential reasoning_: Organized in a step-by-step manner.
* _Strictness_: Contains well-defined steps, with strict input and output formats.
* _Conciseness_: Brief and to the point.

\begin{table}
\begin{tabular}{l l l l l}
**Markup-and-mask** & **Annotated Report** & **Procedural** & **Subquestions** & **Decision Tree** \\ \hline
**Error location: Step 1** & **Error location:** Step 1 & **Error location:** Step 1 & **Error location:** Q1 & **Error location:** Q2-Yes \\
**Issue:** The information is insufficient to answer the question and the inference framework from the context is incorrect. & & **Descripton:** Charles is the second who was a bachelor & **Description:** The question asks about the first name of & **Description:** The question asks about the first name of \\
**Description:** Charles is the same of the son, and the question asks about the first instance of question as shown the first name of the person who is the son’s parent. & & **Subquestion:** The rationale needs to locate who & **objdent is not Charles** \\
**Suggestion:** The rationale needs to find out who he & & & **Charles per parents in the text** & **Spencer Super/or, and not** \\  & & & & **the first name of the \\  & & & & **reads to look at previous** \\  & & & & **then working document this?** \\  & 



### Results

Figure 3 shows the Likert distributions of judgements of interpretability and trustworthiness for all formats on both datasets. These suggest that rationales with attributions and a sufficient amount of depth (annotated_report and procedural) are most easy to understand and trust for Quoref. On the other hand, annotated_report and subquestions rate highest on both axes for PubMedQA. Our interpretation of these judgements is that to be easily understandable and trustworthy for users, rationales should provide sufficient insight into the model's reasoning process and be accompanied with attributions.

Among the rationale properties presented to annotators, we find that attribution and depth of reasoning are rated as the most important properties of rationales. Figure 4 shows averaged scalar judgements for different rationale properties. A clear conclusion from these judgements is that providing attributions in the form of extracted quotes to the context is essential to users. This is likely because the attributions ground the model's reasoning into the context. In addition, depth of reasoning is highly valuable to users, especially for PubMedQA questions, where they may value a logical and coherent description of the model's reasoning.

## 7 Related Work

Decomposed QAAlthough rationales from NLP models can be beneficial for users, there is recent evidence that shows that they are not always faithful to model responses Ye and Durrett (2022); Lyu et al. (2022); Turpin et al. (2023); Lanham et al. (2023). Decomposed question answering systems break down the QA problem into two stages, that of generating an intermediate rationale, and then using only that rationale to generate the answer Lei et al. (2016); Eisenstein et al. (2022); Radhakrishnan et al. (2023). This provides a stronger inductive bias to the model to be faithful to the rationale. Similar ideas have been pursued in other tasks such as object recognition Koh et al. (2020), image classification and text classification Yeh et al. (2020). The precise format of the intermediate rationale that is optimal for human critiquability and interpretability is understudied. Our study is dedicated towards investigating the structure of this intermediate rationale.

Human feedback in NLPProviding human feedback to NLP models has proven to be an effective way to repair models and fix model behaviors Fernandes et al. (2023). Feedback can allow users to convey example-level critiques about model predictions, which, when incorporated into models, encourage them to perform better. Prior

Figure 4: Scalar judgements of characteristics that annotators value in intermediate rationales (scale of 1-5).

Figure 3: Likert distribution of the annotator judgements of interpretability & trustworthiness for different rationale formats corresponding to correct answers (§6).



work has explored using human feedback for improving text summarization (Stiennon et al., 2020; Liu et al., 2023; Scheurer et al., 2023), question answering (Gao et al., 2022; Li et al., 2022), semantic parsing Iyer et al. (2017); Elgohary et al. (2020, 2021), dialog generation Shi et al. (2022); Ouyang et al. (2022); Xu et al. (2023), machine translation Kreutzer et al. (2018) and image captioning Fidler et al. (2017). Our work builds upon this prior work and investigates the effectiveness of human feedback for rationales provided by QA systems.

**Rationales and explanations for NLP models.** There is a large body of prior work studying explanations to supplement outputs from NLP models, both for improving models Hancock et al. (2018); Lampinen et al. (2022); Wang et al. (2022); Zelikman et al. (2022); Zhou et al. (2023b) and explaining model outputs to end users. Prior work has found that explanations can be beneficial to end users for understanding model responses DeYoung et al. (2020); Narang et al. (2020); Wiegreffe et al. (2022) as well as debugging models Lertvitayakumjorn and Toni (2021); Lamm et al. (2021). Prior work has also studied the impact of explanations on user trust Papenmeier et al. (2022) and performance of human-AI teams Bansal et al. (2021). Boyd-Graber et al. (2022); Jacovi and Goldberg (2020) provide useful guidelines to conduct human-centered and faithful evaluations of these explanations. We conduct another such evaluation that is centered on the format of model rationales presented to end users.

## 8 Conclusion

Our work analyzed how model-generated explanations or rationales should be formatted to be most amenable to repair through user feedback. We also collected qualitative judgements of how different formats enable users to understand and trust model outputs. We found that rationale formats significantly affect what rationales are amenable to be repaired through feedback. In terms of user perception of rationales, we find that some rationale formats, such as the _annotated report_, are more favorable for enabling users to understand and trust model responses. Finally, we find that among a few properties considered, attribution and depth of reasoning are the most important characteristics of rationales to users. We hope that this work can help researchers and practitioners alike make informed decisions about how to present language model responses and collect feedback from end users.

## 9 Limitations

**Rationale Formats.** The rationale formats we consider are by no means exhaustive and there could be numerous other plausible formats for intermediate rationales. We choose a set of rationales that vary according to some well-defined properties (mentioned in Table 1), that can allow us to form conclusions about the importance of those properties.

**Feedback Structure.** We choose a feedback structure that encompasses a few crucial aspects of feedback highlighted in previous work. However, it may be possible that there are other types of feedback that show different trends in effectiveness across rationale formats.

**Scope of QA problems.** We choose reading comprehension datasets where questions are formulated based on a given context. While these may not be representative of all forms of QA problems, we hope our findings can broadly inform practitioners about ways to present QA system responses to users (for instance, when deploying retrieval-augmented QA systems).

### Acknowledgements

First, we would like to thank the annotators who participated in our studies. We would also like to thank Elizabeth Sieber, Jacob Eisenstein, Weiqiu You and Eric Wong for helpful comments and discussions.

## References

* Bansal et al. (2021) Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In _Proceedings of the 2021 CHI conference on human factors in computing systems_, pages 1-16.
* Boyd-Graber et al. (2022) Jordan Boyd-Graber, Samuel Carton, Shi Feng, Q. Vera Liao, Tania Lombrozo, Alison Smith-Renner, and Chenhao Tan. 2022. Human-centered evaluation of explanations. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorial Abstracts_, pages 26-32, Seattle, United States. Association for Computational Linguistics.
* Chen et al. (2023) Xinyun Chen, Maxwell Lin, Nathanael Schirali, and Denny Zhou. 2023. Teaching large language models to self-debug. _arXiv preprint arXiv:2304.05128_ 

* Choi et al. (2021) Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. 2021. Deconestualization: Making sentences stand-alone. _Transactions of the Association for Computational Linguistics_, 9:447-461.
* Dasiigi et al. (2019) Pradeep Dasigi, Nelson F. Liu, Ana Marasovic, Noah A. Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5925-5932, Hong Kong, China. Association for Computational Linguistics.
* DeYoung et al. (2020) Jay DeYoung, Sarithak Jain, Nazreen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: A benchmark to evaluate rationalized NLP models. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4443-4458, Online. Association for Computational Linguistics.
* Dua et al. (2022) Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. 2022. Successive prompting for decomposing complex questions. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 1251-1265, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Eisenstein et al. (2022) Jacob Eisenstein, Daniel Andor, Bernd Bohnet, Michael Collins, and David Mimno. 2022. Honest students from untrusted teachers: Learning an interpretable question-answering pipeline from a pretrained language model. In _Workshop on Trustworthy and Socially Responsible Machine Learning, NeurIPS 2022_.
* Elgohary et al. (2020) Ahmed Elgohary, Saghar Hosseini, and Ahmed Hassan Awadallah. 2020. Speak to your parser: Interactive text-to-SQL with natural language feedback. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2065-2077, Online. Association for Computational Linguistics.
* Elgohary et al. (2021) Ahmed Elgohary, Christopher Meek, Matthew Richardson, Adam Fourney, Gonzalo Ramos, and Ahmed Hassan Awadallah. 2021. NL-EDIT: Correcting semantic parse errors through natural language interaction. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5599-5610, Online. Association for Computational Linguistics.
* Fernandes et al. (2023) Patrick Fernandes, Aman Madaan, Emmy Liu, Antonio Farinhas, Pedro Henrique Martins, Amanda Bertsch, Jose GC de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, et al. 2023. Bridging the gap: A survey on integrating (human) feedback for natural language generation. _arXiv preprint arXiv:2305.00955_.
* Felder et al. (2017) Sanja Felder et al. 2017. Teaching machines to describe images with natural language feedback. _Advances in Neural Information Processing Systems_, 30.
* Gao et al. (2022) Ge Gao, Eunsol Choi, and Yoav Artzi. 2022. Simulating bandit learning from user feedback for extractive question answering. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 5167-5179, Dublin, Ireland. Association for Computational Linguistics.
* Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. _Transactions of the Association for Computational Linguistics_, 9:346-361.
* Hancock et al. (2018) Braden Hancock, Paroma Varma, Stephanie Wang, Martin Bringmann, Percy Liang, and Christopher Re. 2018. Training classifiers with natural language explanations. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1884-1895, Melbourne, Australia. Association for Computational Linguistics.
* Iyer et al. (2017) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 963-973, Vancouver, Canada. Association for Computational Linguistics.
* Jacovi and Goldberg (2020) Alon Jacovi and Yov Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4198-4205, Online. Association for Computational Linguistics.
* Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2567-2577, Hong Kong, China. Association for Computational Linguistics.
* Khot et al. (2021) Tshar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2021. Text modular networks: Learning to decompose tasks in the language of existing models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1264-1279, Online. Association for Computational Linguistics.
* Koh et al. (2017) Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Munsmann, Emma Pierson, Beem Kim, and Percy 

Liang. 2020. Concept bottleneck models. In _International conference on machine learning_, pages 5338-5348. PMLR.
* Louisiana. Association for Computational Linguistics.
* Lamm et al. (2021) Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. 2021. QED: A framework and dataset for explanations in question answering. _Transactions of the Association for Computational Linguistics_, 9:790-806.
* Lampinen et al. (2022) Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Koryd Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. 2022. Can language models learn from explanations in context? _arXiv preprint arXiv:2204.02329_.
* Lanham et al. (2023) Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. 2023. Measuring faithfulness in chain-of-thought reasoning. _arXiv preprint arXiv:2307.13702_.
* Lei et al. (2016) Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 107-117, Austin, Texas. Association for Computational Linguistics.
* Lervitatayakumjorn and Toni (2021) Piyawat Letrvitatayakumjorn and Francesca Toni. 2021. Explanation-based human debugging of nlp models: A survey. _Transactions of the Association for Computational Linguistics_, 9:1508-1528.
* Li et al. (2022) Zichao Li, Prakhar Sharma, Xing Han Lu, Jackie Cheung, and Siva Reddy. 2022. Using interactive feedback to improve the accuracy and explainability of question answering systems post-deployment. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 926-937, Dublin, Ireland. Association for Computational Linguistics.
* Lievin et al. (2022) Valentin Lievin, Christoffer Egeberg Hother, and Ole Winther. 2022. Can large language models reason about medical questions? _arXiv preprint arXiv:2207.08143_.
* Liu et al. (2023) Yixin Liu, Budhaditya Deb, Milagro Teruel, Aaron Halfaker, Dragomir Radev, and Ahmed Hassan Awadalah. 2023. On improving summarization factual consistency from natural language feedback. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15144-15161, Toronto, Canada. Association for Computational Linguistics.
* Lyu et al. (2022) Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch. 2022. Towards faithful model explanation in nlp: A survey. _arXiv preprint arXiv:2209.11326_.
* Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. _arXiv preprint arXiv:2303.17651_.
* Martignon et al. (2003) Laura Martignon, Oliver Vitouch, Masanori Takezawa, and Malcolm R Forster. 2003. Naive and yet enlightened: From natural frequencies to fast and frugal decision trees. _Thinking: Psychological perspectives on reasoning, judgment and decision making_, pages 189-211.
* Narang et al. (2020) Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. Wt5?! training text-to-text models to explain their predictions. _arXiv preprint arXiv:2004.14546_.
* Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhnii Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems 35 (NeurIPS 2022)_.
* Papenmeier et al. (2022) Andrea Papenmeier, Daglmar Kern, Gwenn Englebienne, and Christian Seifert. 2022. It's complicated: The relationship between user trust, model accuracy and explanations in ai. _ACM Transactions on ComputerHuman Interaction (TOCHI)_, 29(4):1-33.
* Press et al. (2022) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. _arXiv preprint arXiv:2210.03350_.
* Radhakrishnan et al. (2023) Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamile Lukosute, et al. 2023. Question decomposition improves the faithfulness of model-generated reasoning. _arXiv preprint arXiv:2307.11768_.
* Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. _Foundations and Trends(r) in Information Retrieval_, 3(4):333-389.
* Scheurer et al. (2023) Jeremy Scheurer, Jon Anders Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2023. Training language models with language feedback at scale. _arXiv preprint arXiv:2303.16755_.
* Shi et al. (2022) Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, and Jing Xu. 2022. When life gives you lemons, make cherryade: Converting feedback from bad responses into good labels. _arXiv preprint arXiv:2210.15893_ 

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. _Advances in Neural Information Processing Systems_, 33:3008-3021.
* Sun et al. (2023) Simeng Sun, Yang Liu, Shuohang Wang, Chenguang Zhu, and Mohit Iyyer. 2023. Pearl: Prompting large language models to plan and execute actions over long documents. _arXiv preprint arXiv:2305.14564_.
* Turpin et al. (2023) Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. 2023. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. _arXiv preprint arXiv:2305.04388_.
* Wang et al. (2023) Lei Wang, Wanyu Xu, Yihui Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2609-2634, Toronto, Canada. Association for Computational Linguistics.
* Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022. Rationale-augmented ensembles in language models. _arXiv preprint arXiv:2207.00747_.
* Wiegreffe et al. (2022) Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022. Reframing human-AI collaboration for generating free-text explanations. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 632-658, Seattle, United States. Association for Computational Linguistics.
* Xu et al. (2023) Jing Xu, Megan Ung, Mojtaba Komeili, Kwshal Arora, Y-Lan Boureau, and Jason Weston. 2023. Learning new skills after deployment: Improving open-domain internet-driven dialogue with human feedback. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 13557-13572, Toronto, Canada. Association for Computational Linguistics.
* Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. _arXiv preprint arXiv:2305.10601_.
* Ye and Durrett (2022) Xi Ye and Greg Durrett. 2022. The unreliability of explanations in few-shot prompting for textual reasoning. _Advances in neural information processing systems_, 35:30378-30392.
* Yeh et al. (2020) Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. 2020. On completeness-aware concept-based explanations in deep neural networks. _Advances in neural information processing systems_, 33:20554-20565.
* Zelikman et al. (2022) Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. _Advances in Neural Information Processing Systems_, 35:15476-15488.
* Zhou et al. (2023a) Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. 2023a. Least-to-most prompting enables complex reasoning in large language models. _ICLR_.
* Zhou et al. (2023b) Yangqiacoyu Zhou, Yiming Zhang, and Chenhao Tan. 2023b. Flame: Few-shot learning from natural language explanations. _arXiv preprint arXiv:2306.08042_.

 

Experimental Details

Prompts.The prompts used to generate rationales for all formats are provided in Tables 7, 8, 9, 10 and 11, while the prompts used to generate answers are provided Tables 12, 13, 14, 15 and 16. For generating answers for PubMedQA, we modify the prompt same way as previous work (Lievin et al., 2022), by transforming it into a multiple-choice question. For revising rationales, we use a similar format as these prompts but also includes the following string in the instruction - Correct the given Rationale based on the Feedback. The Feedback first points out the Error Location, then mentions the Issue and gives a Description of the issue, and finally provides a Suggestion to correct the given Rationale. The Rationale is required to be sufficient to answer the Question on its own and faithful to the Context..

Hyperparameter settings.Rationales and answers were sampled from gpt-3.5-turbo with a temperature of 0.0 and a maximum length of 512 tokens when sampling rationales, and 64 tokens when sampling answers. For prompting models, we sample few-shot exemplars using BM25 (Robertson et al., 2009) from a set of 100 manually labeled examples with rationales. We sample as many exemplars as can fit within the maximum sequence length (4096) of the model.

## Appendix B Annotation Details

Annotator backgrounds.For both studies, annotators were recruited from Prolific3, and required to be fluent in English. They were required to have at least 100 accepted submissions and an approval rate of at least 99%. They were also required to have at least a bachelor's degree.

Footnote 3: www.prolific.co

Annotators for the Quoref task were based in UK, USA, Australia, Ireland, Canada or New Zealand. Annotators for the PubMedQA task were based in UK, USA, Ireland, Germany, France, Australia, Canada, Denmark, Netherlands, Switzerland, Norway, Portugal or Sweden. These annotators were additionally required to be employed in the healthcare/medicine sector.

Annotation costs.In both studies, annotators were compensated at the rate of $15 per hour with additional bonuses when annotators spent more time than we anticipated.

Annotation interface.Figures 5 and 6 show screenshots of our annotation interface for both Study 1 and Study 2 in the order the task was presented to annotators.

## Appendix C Additional Results

Table 6 shows the effectiveness of feedback with examples where 3 rationale formats get the answer wrong. Figure 7 shows the Likert distribution of sufficiency, faithfulness and ease of providing feedback for all rationale formats for both datasets.

 



\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{3}{c}{Quoref} & \multicolumn{3}{c}{PubMedQA} \\ \cline{2-7}
**Rationale Format** & **edit\_acc** & **final\_acc** & **time\_taken** & **edit\_acc** & **final\_acc** & **time\_taken** \\ \hline markup\_mask & 70.37 & 62.96 & 340.29 & 64.58 & 33.33 & 389.79 \\ annotated\_report & 61.29 & 58.06 & 290.17 & 58.06 & **45.16** & 447.90 \\ procedural & 52.94 & 58.82 & 345.55 & 51.28 & 33.33 & 333.80 \\ subquestions & 81.81 & **72.72** & 316.71 & 65.12 & 30.23 & 348.57 \\ decision\_tree & 66.67 & 38.10 & 340.39 & 88.64 & 13.64 & 465.39 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Results after patching generated rationales with human feedback for examples where the answer is wrong for 3 rationale formats. We show here the edit\_acc, which measures if the revised rationale successfully incorporates feedback and final\_acc, which measures the final accuracy after regeneration with the revised rationale.

\begin{table}
\begin{tabular}{l} \hline \hline
**X2R Prompt (annotated\_report)** \\ \hline Generate a rationale that is helpful and sufficient to answer the question. The rationale should contain a list of extracted phrases from the context and the conclusion drawn from each phrase. Try to include no more than 5 extracted phrases. \\ Context: [CONTEXT] \\ Question: [QUESTION] \\ Annotations: \\ \hline \hline \end{tabular}
\end{table}
Table 7: \(X2R\) prompt for the markup\_mask format.

\begin{table}
\begin{tabular}{p{199.2pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline  & \multicolumn{3}{c}{Quoref} & \multicolumn{3}{c}{PubMedQA} \\ \cline{2-7}
**Rationale Format** & **edit\_acc** & **final\_acc** & **time\_taken** & **edit\_acc** & **final\_acc** & **time\_taken** \\ \hline markup\_mask & 70.37 & 62.96 & 340.29 & 64.58 & 33.33 & 389.79 \\ annotated\_report & 61.29 & 58.06 & 290.17 & 58.06 & **45.16** & 447.90 \\procedural & 52.94 & 58.82 & 345.55 & 51.28 & 33.33 & 333.80 \\ subquestions & 81.81 & **72.72** & 316.71 & 65.12 & 30.23 & 348.57 \\ decision\_tree & 66.67 & 38.10 & 340.39 & 88.64 & 13.64 & 465.39 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Results after patching generated rationales with human feedback for examples where the answer is wrong for 3 rationale formats. We show here the edit\_acc, which measures if the revised rationale successfully incorporates feedback and final\_acc, which measures the final accuracy after regeneration with the revised rationale.





**X2R Prompt (procedural)**

Construct a structured Plan for answering the Question, that should provide a sequential process for finding the answer. The Plan should not directly answer the Question but only provide the reasoning. You can use the following operations in the plan:

- Extract-relevant-sentences: Extract relevant sentences from the passage that are sufficient to answer the question. The extracted sentences should include the necessary information to answer the question accurately.

- Disambiguate-question-entity(s): Determine the specific entity or phrase that the string s in the question refers to. Clarify any ambiguous terms or references to ensure a precise understanding.

- Disambiguate-plan-entity(s): Identify the entity or phrase that the string s in the plan refers to. Resolve any ambiguity within the plan by specifying the relevant entities explicitly.

- Locate-entity(s): Generate a subquestion s that is important to answer the original question without simply repeating the original question. Determine the exact entity or phrase that provides the answer to the subquestion s.

Context: [CONTEXT]

Question: [QUESTION]

Plan:

\begin{table} \end{table}
Table 10: \(X2R\) prompt for the subquestions format.

\begin{table} \end{table}
Table 9: \(X2R\) prompt for the procedural format.





**R2Y Prompt (annotated_report)**

You are given an annotated rationale from a passage as context. The annotations are in the format of a list of extracted phrases from the context and the conclusion drawn from each phrase. Answer the question based on the rationale alone.

Rationale: [RATIONALE] Question: [QUESTION]Answer:

\begin{table}
\begin{tabular}{p{341.7pt}} \hline \hline
**R2Y Prompt (annotated_report)** \\ You are given an annotated rationale from a passage as context. The annotations are in the format of a list of extracted phrases from the context and the conclusion drawn from each phrase. Answer the question based on the rationale alone. \\ Rationale: [RATIONALE] Question: [QUESTION]Answer: \\ \hline \end{tabular}
\end{table}
Table 13: \(R2Y\) prompt for the annotated_report format.





**R2Y Prompt (procedural)**

Answer the Question based on the Plan-based Rationale. The Plan gives a sequential process of finding the answer. The following operations can be used in a plan: <Skipped for brevity>.

Plan: [RTationale]

Question: [QUESTION]

Answer:

**R2Y Prompt (subquestions)**

Answer the given Question solely based on the Subquestions and their answers. The answer can always be found from the Subquestions so make your best guess.

Subquestions: [RATIONALE]

Question: [QUESTION]

Answer:

**R2Y Prompt (decision_tree)**

Answer the Question solely based on the Decision Tree-based Rationale.

Decision Tree: [RATIONALE]

Question: [QUESTION]

Answer:

**R3X Prompt (decision_tree)**

Answer the Question solely based on the Decision Tree-based Rationale.

Decision Tree: [RATIONALE]

Question: [QUESTION]

Answer:

**R4X Prompt (decision_tree)**

Answer the Question solely based on the Decision Tree-based Rationale.

Decision Tree: [RATIONALE]

Question: [QUESTION]

Answer:

**R5X Prompt (decision_tree)**

Answer the Question solely based on the Decision Tree-based Rationale.

Decision Tree: [RATIONALE]

Question: [QUESTION]

Answer:

**R6X Prompt (decision_tree)**

Answer the Question solely based on the Decision Tree-based Rationale.

Decision Tree: [RATIONALE]

Question: [QUESTION]

Answer:

**R7X Prompt (decision_tree)**

Answer the Question solely based on the Decision Tree-based Rationale.

Decision Tree: [RATIONALE]

Question: [QUESTION]

Answer:

**R8X Prompt (decision_tree)**

Answer the Question solely based on the Decision Tree-based Rationale.

Decision Tree: [RATIONALE]

Question: [QUESTION]

Answer:

**R9X Prompt (decision_tree)**

Answer the Question solely based on the Decision Tree-based Rationale.

Decision Tree: [RATIONALE]

Question: [QUESTION]

Question: [QUESTION]

Answer:

**R10X Prompt (decision_tree)**

Answer the Question solely based on the Decision Tree-based Rationale.

Decision Tree: [RATIONALE]

Question: [QUESTION]

Question: [QUESTION]

Answer:

**R11**

Answer the question solely based on the Decision Tree-based Rationale.

Decision Tree: [RATIONALE]

[MISSING_PAGE_POST]





## 1. (Study 1 + Study 2) Context & Question + Rationale + Generated Correct Answer

**Context:**

**RQMCOR

2. (Study 1) Sufficiency + Faithfulness

2. (Study 2) Interpretability + Trustworthiness

3. (Study 1) Feedback: Instructions + Location of Error + Type of Error + Description of Error + Actionable Suggestion

4. (Study 1) Case of Providing Feedback 

* [25] M. C. Bader, A. D. Kessler, and J. S. Murray, "A new approach to the identification of a set of attributes in an image-based fashion," _IEEE Transactions on Image Processing_**10**(4):639-678, 2000.
* [26] R. F. Barnett, P. N. G. Smith, and T. W. L. Jones, "The role of attribute attribute attribute for attribute attribute attribute," _Journal of Machine Learning Research_**10**(1):1-10, 2004.
* [27] E. B. Chiarello, M. C. Bader, and J. S. Murray, "Feature-based attribute attribute attribute for attribute attribute attribute," _arXiv preprint arXiv:1905.05587_ (2019).
* [28] H. Chen, Y. Li, X. Zhang, and Z. Sun, "Feature-based attribute attribute attribute for attribute attribute attribute," _arXiv preprint arXiv:1905.05587_ (2019).
* [29] H. Chen, Y. Li, and Z. Sun, "Feature-based attribute attribute attribute attribute for attribute attribute attribute attribute," _arXiv preprint arXiv:1905.05587_ (2019).
* [30] H. Chen, Y. Li, and Z. Sun, "Feature-based attribute attribute attribute attribute for attribute attribute attribute attribute," _arXiv preprint arXiv:1905.05587_ (2019).
* [31] H. Chen, Y. Li, and Z. Sun, "Feature-based attribute attribute attribute attribute for attribute attribute attribute attribute," _arXiv preprint arXiv:1905.05587_ (2019).
* [32] H. Chen, Y. Li, and Z. Sun, "Feature-based attribute attribute attribute attribute for attribute attribute attribute attribute," _arXiv preprint arXiv:1905.05587_ (2019).
* [33] H. Chen, Y. Li, and Z. Sun, "Feature-based attribute attribute attribute attribute attribute for attribute attribute attribute attribute attribute," _arXiv preprint arXiv:1905.05587_ (2019).
* [34] H. Chen, Y. Li, and Z. Sun, "Feature-based attribute attribute attribute attribute attribute attribute attribute," _arXiv preprint arXiv:1905.05587_ (2019).
* [35] H. Chen, Y. Li, and Z. Sun, "Feature-based