Enhancing LLM Faithfulness in Rationale Generation via Dual-Reward Probabilistic Inference

Authors: Hanqi Yan, Jiazheng Li, Yulan He
Primary area: interpretability and explainable AI
Keywords: interpretability, faithfulness, Large language model, constrained generation

Abstract:
As large language models (LLMs) are increasingly applied to complex reasoning tasks, achieving both accurate task performance and faithful explanations becomes crucial. However, LLMs often generate unfaithful explanations, partly because they do not consistently adhere closely to the provided context. Existing approaches address this problem either rely on superficial calibration, such as decomposed Chain-of-Thought prompting, or require costly retraining to improve model faithfulness. In this work, we propose a probabilistic inference paradigm that provides fine-grained and lookahead rewards to ensure that LLM-generated rationales are logically coherent and comprehensive. These rewards are derived from a domain-specific proposal distribution, allowing for optimised sequential Monte Carlo approximations. Our evaluations across three different reasoning tasks show that this method, which allows for controllable generation during inference, improves both accuracy and faithfulness of LLMs while keeping computational costs similar to those of existing decoding techniques. This method offers a promising path towards making LLMs more reliable for reasoning tasks without sacrificing performance or efficiency.

Decision: reject
Reviewer scores: 5, 1, 3, 5

Reviews and discussion:
<thread>
creation date:
1729499305436

thread_type:
official_review

<thread_object>
creator: reviewer_1tj5

<content>
summary:
They tackle the rationale generation tasks in LLMs' reasoning process. Specifically, they propose a probabilistic inference paradigm that provides fine-grained and lookahead rewards to instruct LLMs to generate good rationale. The key problem addressed is that LLMs often produce unfaithful explanations, especially when they fail to incorporate essential contextual information. 

+ **Local Reward**:  this component ensures coherence with the immediate context, often by using a domain-specific expert model.
+ **Global reward**: This assesses the plausibility of the current token in relation to desirable future attributes

The search algorithm, especially for lookahead reweight seems interesting.

Please forgive me if I misunderstand something. I spent much time for reading the paper but to be honest, I am not an expert in this area. I will available on the rebuttal time for author's response and will read their response. I am also open to other reviewers' opinions.

strengths:
1. The paper introduces a novel probabilistic inference method with a dual-reward mechanism, combining local and global reward. This is a very novel solution. 
2. The paper is well-written. I am not an expert in this domain but I can get their core contributions. 
3. The experiment design is clear: they design the ablation study in Section 5.1 to justify the local and global rewards for the final performance. Although I suggest authors could do better by choosing more LLMs in different model size to better support their experimental design.

weaknesses:
1. There are several related works that are missing or less discussed: 
        + Evaluating Human Alignment and Model Faithfulness of LLM Rationale
        +  On Measuring Faithfulness or Self-consistency of Natural Language Explanations
2. Figure 2 about the distribution of domain-specific words is unclear to me. "showing that our method can respond more actively to those domain-specific words" Why does this part matters to the experimental results.

questions:
1. The overall experiments are conducted on LLaMA3. I think more backbone LLMs and other sizes of LLMs are needed to justify the proposed inference paradigm.
2. More experiments on more related datasets is needed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730441273467

thread_type:
official_review

<thread_object>
creator: reviewer_msqh

<content>
summary:
This work proposes an inference-time method to improve the performance and faithfulness of general (instruction-tuned) large language models (LLMs). Specifically, the method uses expert models to provide fine-grained and lookahead rewards to search and reweight possible tokens or continuations proposed by the LLM. With the help of expert models trained on the target task or domain, the proposed method can improve both the accuracy and faithfulness of the zero-shot answers of two instruction-tuned models on three reasoning tasks.

strengths:
The direction this paper explored has been receiving increasing interest recently: improving the quality of LLM answers at inference time without modifying the model weights directly. The proposed method improves the zero-shot accuracy and faithfulness of two strong general instruction-tuned models (Llama-3-8B and Mistral-7b-Instruct-v0.3) on three reasoning tasks. The experiment showing the benefits of going beyond local/token-level rewards and taking into account the global/lookahead reward is interesting.

weaknesses:
- There needs to be more details explaining the proposed method, the motivation of each part, the equations and variables, the relation to related work, and the implementation details. Specifically:
  - Section 3.3: how does the Feynman-Kac Formulae model inspire the faithfulness-seaking search framework? The connection is not straightforward. The notation of eq 1 is ambiguous. What does posterior P_t(st) mean exactly? How is it used in the proposed method? Also, the equation itself needs more explanations on what it is computing and why in this way.
  - Section 3.4 (Local constraint): line 179 I find it hard to follow the motivation. How "certain attributes can be implicitly conveyed over longer spans rather than the individual token" is connected to "Instead, domain-specific experts tend to demonstrate better accuracy in knowledge-rich tasks."? If the domain expert has better accuracy why not just use the expert to predict the scores? Why bother to use them to improve the backbone LLM? In lines 180-181, it says "we introduce a set of classification label words C from these expert models ...", how is C constructed? What is the motivation behind token masking?
  - Section 3.4 (Lookahead Reweight): Equation 3 is hard to understand without proper explanations. $m$ and $x_i$ are not explained in the texts. $s_{t+l}=s_{t-1}||w_t$ is more confusing: $s_{t+l}$ has $t+l$ tokens while $s_{t-1}||w_t$ has $t$ tokens. What does equality mean here?
- Many experimental details are missing, and important experiments are missing.
  - Missing baselines: the performance and faithfulness of the expert models alone. If the faithfulness or accuracy of the expert models are better than the backbone LLM, why do we even need to use the expert models to improve the backbone LLM?
  - Evaluation details: how is the original model evaluated? If it is a zero-shot evaluation. What is the exact prompt and task format used? How to extract answers from the outputs to calculate the accuracy? The backbone LLMs are state-of-the-art instruction-tuned models. However, the task performance as well as the faithfulness are quite low, so the authors need to provide more details on the evaluation.
  - What is the choice of hyperparameter n (number of rollouts) and how is it chosen?
- The writing of the paper could be improved for better readability. First, the paper is not properly scoped. For example, in lines 16-18, it says "... to ensure that LLM-generated rationales are logically coherent and comprehensive." However, there is no result discussing the logical coherence or comprehensiveness of answers in the paper. Another example is line 108: it says "We firstly introduce the faithfulness definition in our context,", but there is no clear definition in section 3.2.

questions:
1. If the expert model is as big as the base model, how can the computational cost be similar to beam search?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705308942

thread_type:
official_review

<thread_object>
creator: reviewer_msmv

<content>
summary:
The work aims to improve the faithfulness of the LLM-generated rationales for reasoning tasks. They propose an inference-based method where an LLM is guided to generate more faithful rationales by both local and global rewards. Both rewards are provided by additional expert models which are trained on the downstream tasks. Experiments demonstrate the effectiveness of the method in achieving higher accuracy and faithfulness.

strengths:
1. Faithful rationales are important for explainability and model control, which makes this work well-motivated.
2. The proposed method is training-free (although with reliance on trained expert models), making their method portable.
3. A comprehensive set of experiments is conducted to showcase the effectiveness of their proposed method.

weaknesses:
1. The method requires the model to generate the answer prior to the rationale, which provides no guarantee that the decision is made based on the rationale. The model could still suffer from inherent biases.
2. The method is limited to reasoning tasks with constrained answer space, limiting its generalization to more open-ended tasks.
3. The method is poorly introduced. It would be very helpful if the authors could explain what exactly Eq.1-3 are doing in plain words.

questions:
Could this method generalize to the setting where the rationale is generated before the answer?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730759571303

thread_type:
official_review

<thread_object>
creator: reviewer_tvlg

<content>
summary:
The paper proposes an approach to do faithful rationale generation in LLMs. It uses a steering-based approach to make the outputs more faithful to the reasoning of the llm in classification. The idea is to weight token logits using 2 kinds of reward models: A "local" one that tries to match tokens to those suggested by a domain-specific expert model and a "lookahead" one that does an MTCS type search and re-weights logits based on rewards from unrolled sequences.  

Experiments are performed on a couple of QA type datasets, demonstrating that each method makes improvements in classification accuracy and faithfulness of rationales. Some qualitative analyses are also presented.

strengths:
1. MTCS type inference is a hot topic right now, and it is indeed an important frontier for LLMs to improve on.
2. At a surface level, experimental results seem to show large gains.

weaknesses:
Section 3 is pretty badly written, it is pretty hard to get the details of the approach. Instead of invoking irrelevant sophisticated-sounding terminology like "Feynman-Kac" formulas it would be better to describe the method in more detail. The math especially is confusing, see below.  

The paper seems to show some positive experimental results, but I am concerned about whether we are looking at a meaningful comparison. The proposed methods rely on domain experts. Looking at table 8 in the appendix these are generally models that have been fine-tuned for the task in some way (and not just on the validation sets as the main section claims, some have access to external datasets). So it shouldn't be that surprising that a method that is given access to an expert which has more signal will do better than the backbone pre-trained model. A fair comparison would have to be with an approach that does vanilla fine-tuning of LLama or mixtral model. 

In terms of novelty: The authors have not really cited relevant work in the controlled decoding space:

https://arxiv.org/abs/2310.17022

https://sea-snell.github.io/ILQL_site/

These works already do something more sophisticated than just token reweighting by a reward score. So what is the novel contribution here? 2 possibities:
1. Focusing on the faithfulness problem.
2. The "lookahead" idea of the reward model. I dont recall having seen this before, but it feels like a simplification of a full-blown MCTS. I would also call this a poor man's version of ILQL.

So we are just left with #1 then, unless I missed something. And this is something I consider of limited novelty (more like an application for a particular problem, though one with interesting implications from the steering perspective).

questions:
1. what is "t \wedge T" ?
 2. sec 3.3, what is P(s_t) a posterior over?
 3. In what sense is \pi_t a "potential" function?
 4. I cannot make any sense of eq 2. Is w \in V the same as w_t?  why cant you simply remove the indicator function and write it as \sum_w \in C ? why is the indicator function in the deminator as well? is the intent to have a logit distribution that only puts mass on the tokens in C?
 5. are the rollouts done on the backbone model or the expert model? have we considered /measured the inference time cost? this is an important consideration in a paper about mtcs type methods.

6. Does q_\phi simply reward completions of the output that have tokens in C?

7. intro: "in contrast, an expert model....." : this is an interesting claim (does seem plausible). is there a citation for evidence?

8. line 140: tend to generate similar token....": what does this mean?

9. i am not up to date on the faithfulness literature, but the kind of interventations that the paper describe as standard ways of evaluation i.e. word inclusion and perturbation just seem to be likely to be noise-prone, leading to unreliable evals?

10. GenExpert =?  lookahead?

11. comment: the discussion between 329-342 helped understanding a bit and should be earlier in the paper.

12. sec 5.2.2: the NLI example is a bad one i think. Submergible only means it is something that can be submerged. which doesnt automatically mean it is submerged.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732234649167

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of revision in pdf

comment:
Thank you very much for all the reviewers' feedback. We have explicitly incorporated your initial feedback into our revised PDF (in blue):

- Summarised our three-fold **contributions in the Introduction.**
- Revision on **method introduction:**
  - **Algorithm Description In Section 3.3**: we introduced a step-by-step description of our method in Algorithm 1.
  - **Feynman-Kac Model Explanation in Section 3.3**: we reorganised the introduction of the model and added a more thorough explanation of Eq. (1).
  - **Local and Global Reward Formulations**: we elaborate how the rewards are calculate by referring to Algorithm 1.
- Update the **ablation study results** We updated the performances of vanilla expert models in **Table 6**.
- Replace the **case for NLI in Section 5.3 Case Studies.**
- Add detailed **evaluation** and **hyper-parameter** in **Appendix A.**
- Add **faithfulness evaluation results** on three tasks based on the **Mistral-7B backbone in Appendix B.**
- In **Appendix C.1**, we elaborated on how our framework can be generalized to tasks where the **answer space is unconstrained.**
</content>
</thread_object>

</thread>
<thread>
creation date:
1732237388573

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to all reviewers

comment:
We thank all reviewers for their valuable feedback and dedicated time! As some of the reviewers are concerned about the method details, here we give a brief explanations about our motivation and method:

**Motivation of leveraging the expert model for faithful rationale generation**

As demonstrated in Table 1, we first identified the importance of domain-specific words in enhancing context adherence. This insight motivated us to increase the generation probability of domain-specific words, leading to incorporation of expert model trained on specific domains for faithfulness enhancement. To the best of our knowledge, this is the first study to improve faithfulness by explicitly encouraging the generation of domain-specific tokens. 

**The advantages of our overall probabilistic inference framework by comparing with other MCTS-like methods**

Our faithfulness-seeking model is techinically distinguished in *computation efficiency* and *lookahead rewards*. 
- Unlike existing explicit MCTS [1,2] requiring expensive rollouts or simulations to evaluate potential actions, it computes expected rewards in a more integrated and efficient way, streamlining the inference process. 
- The incorporated lookahead is achieved on cumulative rewards across multiple steps, rather than overly prioritising short-term gains or rely on heuristics, e.g., the normalised average that do not model future states effectively.

**Reward**

**Local reward**: Inspired by the literature that domain-specific experts tend to demonstrate better accuracy in knowledge-rich tasks. Specifically, we introduce a set of classification label words and penalise label words that are not included in the expert model’s predictions. The output probabilities are then renormalised to ensure validity.

**Global reward**:  As a faithful rationale requires coherence with the surrounding context. We use a expert model that is trained on domain-specific corpus to provide **lookahead** reward when generating rationales. Specifically, the expert model scores the current state $x_t$ by evaluating the generated $(x_t, x_{t+1})$ from backbone LLMs. It is expected that text spans faithful and coherent with the domain-sensitive context are preferred, as they better align with the expert's fine-tuned distribution.

**Contribution and novelty**

- We investigate the challenge of faithful rationale generation by highlighting the limitations of general LLMs in producing domain-specific responses. To the best of our knowledge, this is the first study to enhance faithfulness by explicitly encouraging the generation of domain-specific tokens.
- We propose two novel reward mechanisms, namely local and lookahead rewards, tailored for the rationale generation problem. These are integrated into a efficient probabilistic inference framework to achieve a trade-off between task accuracy and rationale faithfulness.
- Empirical results show both enhancement in accuracy and faithfulness over seven tasks, with an absolute accuracy improvement of 33\% over the seven datasets, along with a 10\% improvements in faithfulness evaluation, while maintaining a computation cost similar to beam search (1.3$\times$).

References:

[1] Don’t throw away your value model! generating more preferable text with value-guided monte-carlo tree search decoding. 

[2] Pairwise optimization for o1-like olympiad-level mathematical reasoning.
</content>
</thread_object>

</thread>


Metareview: