RAPPER: Reinforced Rationale-Prompted Paradigm for Natural Language Explanation in Visual Question Answering
Authors: Kai-Po Chang
Chi-Pin Huang
Wei-Yuan Cheng
Fu-En Yang
Chien-Yi Wang
Yung-Hsuan Lai
Yu-Chiang Frank Wang
Venue: ICLR 2024 poster (2024)
Keywords: Vision Question Answering Natural Language Explanation (VQA-NLE)
Primary area: generative models
TL;DR: We propose a novel approach, RAPPER, designed to tackle the challenges of implausibility and hallucination in producing natural language explanations (NLEs) for vision question answering (VQA) tasks.

Abstract: Natural Language Explanation (NLE) in vision and language tasks aims to provide human-understandable explanations for the associated decision-making process. In practice, one might encounter explanations which lack informativeness or contradict visual-grounded facts, known as implausibility and hallucination problems, respectively. To tackle these challenging issues, we consider the task of visual question answering (VQA) and introduce Rapper, a two-stage Reinforced Rationale-Prompted Paradigm. By knowledge distillation, the former stage of Rapper infuses rationale-prompting via large language models (LLMs), encouraging the rationales supported by language-based facts. As for the latter stage, a unique Reinforcement Learning from NLE Feedback (RLNF) is introduced for injecting visual facts into NLE generation. Finally, quantitative and qualitative experiments on two VL-NLE benchmarks show that Rapper surpasses state-of-the-art VQA-NLE methods while providing plausible and faithful NLE.

Published as a conference paper at ICLR 2024
RAPPER: REINFORCED RATIONALE-PROMPTER PARADIGM FOR NATURAL LANGUAGE EXPLANATION
IN VISUAL QUESTION ANSWERING
Kai-Po Chang'
Chi-Pin Huang'
Wei-Yuan Cheng!
Fu-En Yang
Chien-Yi Wang'
Yung-Hsuan Lai'
Yu-Chiang Frank Wang!2
'Gaaduate Institute of Communication Engineering, National Taiwan University, Taiwan
2NVIDIA, Taiwan
r11 942097, 008500022 entu.edu..uu.............
(fredy, frankwang, chienyiw @nvidia. com
ABSTRACT
Natural Language Explanation (NLE) in vision and language tasks aims to provide human-understandable explanations for the associated decision-making process. In practice, one might encounter explanations which lack informativeness or contra- dict visual-grounded facts, known as implausibility and hallucination problems, respectively. To tackle these challenging issues, we consider the task of visual ques- tion answering (VQA) and introduce Rapper, a two-stage Reinforced Rationale- Prompted Paradigm. By knowledge distillation, the former stage of Rapper infuses rationale- proooppinp via large language models (LLMs), encouraging the rationales supported by language-bel baaed facts. As for the latter stage, a unique Reinforcement Learning from NLE Feedback (RLNF) is introduced for injecting visual facts into NLE generation. Finally, quantitative and qualitative experiments two VL-NLE benchmarks show that Rapper surpasses state-of-the-art VQA-NLE methods while providing plausible and faithful NLE.
INTRODUCTION
Deep learning has achieved remarkable success vision-language (VI.) tasks such as visual reasoning (Suhr et al. 2017), visual question answering (VQA, VVA,,Goyal et al. 2017). and visual entailment (Xie) al. 2019). Take VQA as an example, while these models exhibit impressive ability in inferring answer descriptions from the given image-question pairs, its decision-making process remains an unsolved problem. As a result, such a black-box manner severely restricts their applicability in certain real-world scenarios (e.g., medical VQA, Lin et al. 2023). where the interpretability of the learning model is crucial for establishing trustworthy systems. To tackle this long-standing challenge, some approaches adopt attention mechanisms (Anderson et al.) 2018) gradient-based activations (Selvaraju et al.) 2017), focusing on highlighting image regions which are relevant to the associated prediction. However, such visual explanations might be desirable for VL tasks (c.g.. those beyond classification) due to the lack of reasoning process (Kayser et al. 2021 Sammani et al. 2022). As result, Natural Language Explanation (NLE) has emerged as a potential alternative, which aims to interpret the underlying reasoning process by natural language descriptions.
To extend NLE for lision-language tasks (i.e., VL-NLE), Park al., (2018) and Kayser et 2021 introduced the benchmarks for explaining the decision-making process with NL.Ss for VQA and visual entailment tasks, respectively. Subsequent VI.-NLE works have evolved into two research lines. The first research line (Park et al., (2088) Marasović et al. 2020) focuses how to improve their pipeline from an architecture perspective for training NLE generators within a fully supervised learning manner. On the other hand, Sammann al. (2022) and Suo al. (2023) emphasize the utilization of unlabeled pre-training data to enhance the language models' NLE capability.
Despite significant advancements, most existing VL-NLE works require training in a full supervised manner. They might encounter problems where the explanations are irrelevant to the questions or contradictory to the established supporting facts (Majumder et al. 2021). The other potential concern

---

Published as a conference paper at ICLR 2024
Model Input
(a) Previous VQA-NLE paradigm
Image
Model Output
Answer
Explonation
Implausibility
Sunny
tower.
Model Output
Answer
Explonation
Hallucination
Sunny
Becouse the sun is big.
(b) Reinforced Rationale- Prompte VQA-NLE paradigm (Rapper, ours)
Rationale
Model Output
The presence of a clear blue suggests that the weather is sunny
Explonation
Plausibility & Faithfulness
Question
What is the weather like?
Sunny
Becouse the sky blue.
Figure I: Comparison between (a) previous VQA-NLE paradigm and (b) our proposed reinforced vationale-prompter VQA-NLE paradigm (Rapper). Instead of directly generating answer or explanation, Rapper learns plausible and faithful explanations which prompt the VQA model with improved performance.
is that the explanation is not related to the visual image (Ji et al. former problem referred to as implausibility, while the latter known hallucination. Take visual input and question in Fig. D as an example, "Because there is a tower.' is an implausible explanation since it is irrelevant question, and "Because the sun is big." is hallucinated one since the sun is not visible in the image. Although these issues have been recently studied in the NLE community (Zhao al., 2023; Turpin et al. 2023), they remain unexplored in the field of VL-NLE. As result, generating plausible yet faithful NLES for elucidating vision- language models continues to pose a crucial challenge.
2023)
More specifically, the
Recently, rationale- based prompting techniques have been manifested to improve the capability of Large Language Models (LLMs) on complex reasoning tasks (Wei et al., [022 Liu et al. 20226).
Such techniques involve elicitation of rationales from LLMs, producing knowledge-riched or fact- based intermediate to facilitate the reasoning capability language model. Thus, these prompting
manners emerging promising solutions for NLE (Zhao et al. 2023 Krishna et al. 2023).
These rationale- prompting paradigms have been further extended to multi-modal regimes such as mm-CoT (Zhang al. 2023) and mm- me-ReeAtt (Yang et al. 2023). However, mm-CoT (Zhang et al.) 2023) relies on the ground-truth rationales for training, while mm-ReAct (Yang et al. (023) have potential hallucinated outputs due the information loss when converting visual signals into text for ChatGPT API call understanding.
attionale-prompting
In this paper, we propose Reinforced Rationale-Promptee Paradigm (Rapper) for providing accurate answers for VQA with sufficient NLE, which are plausible and faithful. As depicted in Fig. (b), our Rapper learns exploit knowledge learned from LLM and incorporate the corresponding visual content from input images into rationales through two stages. Without observing any ground truth rationale during training, the first stage utilizes a knowledge distillation process to introduce L.M for enriching the rationales with supporting facts, encouraging NLE to be factual and plausible. The subsequent stage of Reinforcement Learning from NLE Feedback (RLNF) further exploits the answer-explanation feedback to enforce the produced rationales associated with both question and visual inputs, allowing faithful NLE.
We now summarize the contributions of this work below:
reinforced rationale-prov prompeed paradigm, Rapper, proposed for plausible and faithful NLE generation in VQA. This is achieved through two proposed stages: knowledge distillation process from LLM and Reinforcement Learning from NLE Feedback (RLNF).
In Rapper, we first advance LLM and perform knowledge distillation. This results in predicted rationales being based on language-basse facts, which prompt the VQA model for plausible NLE,
To align NLE with the visual input, introduce Reinforcement Learning from NLE Feedback (RLNF) to Rapper, which utilizes the answer-explane explanatnnnnnioon feedback as rewards and prompts the VQA model with predicted rationales for faithful NLE..
2

---

Published as a conference paper at ICLR 2024
Our Rapper achieves new sate-of--hhear performance for both VQA-X X Park et al. 2018) and SNLI-VE (Kayser et al. 2021) on NLE generation. We also demonstrate that Rapper outperforms existing VQA-NLE works with reduced implausibility and hallucination.
RELATED WORK
Plausible and Faithful Natural Language Explanation Research on plausibility and faithfulness in NLE (Majunder et al., 2021 King et al. [022 Gou et al. 2023 Stacey et al., (2023) has garnered wide attention, particularly due the evolution of Large Language Models (LLMs) and chain-of- thought (CoT) prompting techniques (Wei et al. 2022). Notably, the method of integrating external knowledge databases for fact generation or retrieval has been proven effective in enhancing the plausibility and faithfulness of NLEs (Majumder et al.) 2021] Stacey et al. 2023). Based on this advancement, some recent approaches, such as the verify-then-corree pipeline by Gou et al. (2023) and novel decoding strategies proposed by Lan et al. (2023) and King et al. (2022), ) aim to mitigate hallucination in textual outputs. However, these works typically focus on isolated single text modality or rely static external knowledge databases, limiting its scalability to multimodal data.
Natural Language Explanation for Vision-Language Tasks Most existing VL-NLE works (Wu & Mooney 2018 Park et al. [0188] Marasović et al. [020] Kayser et al., 2021) generate explanations in a predict-then-explann fashion. Specifically, an answer is first predicted by a pre-trained VI
model .. UNITER (Chen et al. (020) or Oscar ([i et al. 2020p, followed by the generation of the corresponding explanation via separate language decoder (e.g.. GPT2 (Radford et al., (019)). As the answer and explanation are predicted separately, the explanation often contains irrelevant or contradictory descriptions of the given visual information, struggling to faithfully represent the underlying reasoning process. Recently, NLX-GPT (Sammani et al., 2022) proposes to jointly generate the answer and explanation by a unified sequence- to-squeence model, while S3C (Suo et al., 2023) further enforces the explanation to be consistent with the predicted answer. Although the above approaches have been shown to mitigate the hallucination issue, it is not clear how their NLE is established upon supporting facts or taking the visual input into consideration. Therefore, how to tackle the potential implausibile or hallucinated NLE remains challenging task.
Reinforcement Learning for Language Models Several research works have explored RL and view it as the key component to enhance models across vision-languace tasks such as image caption- ing (Rennic et al. 2017), novel object captioning (NOC) (Yang et al. 2022), and VQA (Lu et al.) 2018 Liu et al. 2018). There has been a concentrated effort to align LMs with
2022a Fan et al.
natural language (NL) feedback (Akyürek al. 2023 Yang et al.) 2022 Liu al. 2022a) well as non-NL feedback (Bai et al. [022 Lu et al. 2022b). For example, Liu et al. (2022a) utilizes the
probability of the correct answer as reward to stimulate an auxiliary module produce beneficial knowledge, thereby enhancing QA-task performance. Similarly, Yang el al. (2022) employs CIDEr optimization strategy to enhance the caption with sufficiently visual fidelity in the task of novel object
captioning. Despite of their effectiveness, their RL framework or NL-feedback approaches cannot be easily applied for VI.-NE tasks.
3 PROPOSED METHOD
3.1 PROBLEM FORMULATION
Given VQA input X = (V, consisting of an input image V and textual input Q (i.e., question), our goal to predict the answer Å and the corresponding explanation É (denoted as f= (Á. E)) via a reasoning module M. In order to encourage M to provide correct answer with plausible and faithful explanation, we propose a Reinforced Patiionll-Proomyee an additional rationale generator G to jointly exploit the supporting facts from LLMS and the visual content observed from the conditioned image into rationales. Note that only the ground truth A and E are available during training, not the rationales. As depicted in Fig. 2 the learning of Rapper is decomposed into: (A) Knowledge Distillation from LLM (Sec. (.2). and (B) Reinforcement learning
Parudigm (Rapper) scheme, which learns
from NLE Feedback (RLNF) (Sec. (.3)) which trains rationale generator G for providing auxiliary intermediates when predicting Y (A, R).
3

---

Published as a conference paper at ICLR 2024
Multimodal Training Data
(A) Knowledge Distillation from LLM
Discard Á is wrony
The sky is cloudy when rainy. which couses the ground wet
B
RS
Pretrained QA Model F
R
R
is
Filter
À
Rationale Generator
.... L Meseeeeee
Reasoning Module
L (Reasening Loss)
(Distillation Less]
V.
(B) Reinforcement Learning from NLE Feedback
rainy?
Penalty
the presence of clouds sky rainy
It'
becouse there clouds the sky
0.2
Because the ground is wet and the girls are using umbrellas
Rationale Generator
Reasoning Module M
V2Q
aans
Fexp
G
presence
becouse the ground people. hold umbrellos
R: ground and umbre suggest it is rainy
Yes
0.8
Ves
A
Reward
Yes
Figure 2: Overview of Rapper. Rapper involves two training stages: (A) Knowledge distillation
introduce the rationales Ry from LLM by offering established facts, facilitating the generation of plausible NLEs from the reasoning module M. (B) Reinforcement learning from NLE feedback (RLNF) further refines the rationales from R' to R by incorporating visual information, encouraging generation of faithful NLEs from M.
3.2 PLAUSIBLE NLE GENERATION
Since VQA-NLE models typically rely on ground truth answers and explanations for training, it not clear whether the underlying visual and language knowledge are exploited to support the predicted outputs. In the first stage of Rapper, we propose to leverage powerful reasoning capability inherent
LLM for plausible NLE generation. As depicted in Fig. A(), we propose learn rationale generator G by utilizing knowledge distillation from LLM (e.g., LLaMA-655 (Touvron et al. (023)) ))
This would have the reasoning module M elaborate the conditioned rationales before answering and explaining and encourage plausible NLE. We now detail this learning stage.
3.2.1 KNOWLEDGE DISTILLATION FOR SOCTED-BASER RATIONALE GENERATION
With the recent success of LLMMs showing great capability for generating rationale prompts as intermediate reasoning steps and knowledge (Wei et al.] 2022, Kojima et al.) 2022, Liu al. 2022b) for reasoning task, propose to advance the guidance of pre-trained LLMs acquire such knowledge, so that supporting facts or knowledge can be exploited and serve as rationales for VL-NLE. Since no ground-truth rationales available, we leverage the LLM to produce rationales as pseudo ground truth for training our rationale generator G. Inspired by Liu et al. (2022a b) and Min al. (2022), elicit pseudo rationale T from LLM with task-specific set of few-shot demonstrations (see Sec. A.5 for details) as follows:
R2 trp Ir, Pum(y.g),,
where y is the ground-truth answer-explanation pair, 9 is question, PLLM denotes the LLM in autoregressive manner, Tp is the sampled pseudo rationale from PLLM. and thus R, is the set of all rp-
However, the above pseudo rationales may be redundant, noisy lengthy, which would not be desirable for subsequent NLE tasks (Li et al. 20236). Thus, we apply post-processing mechanism filter pseudo rationales R2 to R2 To be specific, apply round-trip consistency by answering
4

---

Published as a conference paper at ICLR 2024
the input question the pseudo rationales with pre-trained wawteooinnnnwwiing..gm..o. (QA) model F The pseudo rationale is retained when the matching score between the ground-truth answer and the answer predicted by F exceeds a predetermined threshold т. This matching score is quantified with the token-level Fl score (Wang et al. 2020). Thus, the process of collecting the filtered pseudo rationales P is formulated follows:
; (rpp F1-score(āãa) 2 T, à ~ rp( E Rp):
(2)
where is the ground truth answer, is the answer predicted by F based on the pseudo rationale, and Py denotes the pre-trained QA model F in an autoregressive fashion.
With the above RA serving as psuedo ground truth, we are able train the rationale generator G with the distillation loss LG described below:
T LG = Σ
(3)
t=1
where r E 12,, T - and x = { EX.
3.2.2 PROMPTING BY FACT-BASED RATIONALE FOR PLAUSIBLE NLE
With rationales Ry better aligned with the facts, we can proceed to the training of the reasoning module M for NLE generation. We note that, since rationales Ri are in the form of natural language, our the reasoning module M (which is also based on visual- visual-language model) would be able to interpret them. Thus, in addition to the image-question pair X as the inputs to the reasoning module M, the derived pseudo rationales R2, are further viewed as input prompts, which provide fact-supporting conditions when training to perform VQA-NLE. As a result, we train M by calculating the reasoning loss Las follows:
CM Σ
log(pa((ueloo--1rrk 2)).
In the above cross-entropy loss, y = |a; e] Y is the concatenation of the ground-truth answer and explanation e.
3.3 FAITHFUL NLE GENERATION
Although the above knowledge distillation process based on LLM introduces plausibility into our rationale generation, the predicted rationales might not be related to the visual input and thus encounter the hallucination problem. To tackle this issue, we introduce a novel technique of Reinforcemens Learning from NLE Feedback (RLNF). This learning strategy is to encourage the rationale generator G to fully exploit multimodal input data, so that the output rationales are not only plausible but also faithful. Once G produces faithful rationales, can fine-tune the reasoning module M for plausible yet faithful NLE.
3.3.1 RLNF FOR INJECTING VISUAL FACTS
To address the potential hallucination issue, we propose Reinforcement Learning from NLE Feedback (RLNF) by enforcing rationale generator G to derive the visual facts from the input image into rationales. To achieve this, we define a reward function via RL that penalizes the fact-based but hallucinated rationales R', while rewarding the rationales R that contain both established facts and visual content, as depicted in Fig. B). To achieve this, we design our reward Voout to be the addition of answer scores aans and the explanation score Texp which the average predicted probability of the ground-truit answer and CIDEr score (Vedantam et al. 2015),, respectively. For the answer score, inspired by and following Kadavath al. (2022), we maximize the answer score to assess the faithfulness of the predicted explanation. This maximization enforces the rationale
In the implementation, follow (Changpinyo et al. 2022 and use UnifiedQA (Khahhabi al. 2022 the pre-trained QA model.
5

---

Published as a conference paper at ICLR 2024
Algorithm Training RAPPER
Input: Rationale generator G. reasoning module M, LLM PIum and pre- vre-trained QA model Pe Data: Image-question pairs (x') Ji=1, and answer-explanation pairs Y {
1 Stage(A): j: KD for Plausible NLE Generation */
R + Collect pscudo rationales (Eq. equation T
R + Get filtered pseudo rationales from Rp (Eq. equation
Section 3.2
G <- Update G with Le (Eq. equation 3
M Update M with Lm (Eq. equation
D Section 3.2.2
Stage(B); RLNF for Faithful NLE Generation
< Update with Rannt (Eq, equation 8
Section 3.3.1
M +- Update M with Lus (Eq. equation 10
Section 3.3.2
Output: Go, M
generator G to inject more visual content into the rationale because the reasoning module M need more visual clues to correctly answer the question. Therefore, this process transform R' to R, and simultaneously provide the M with more visual fact-based rationale R enable the explanation with sufficient faithfulness. On the other hand, the explanation score Tep is (i.e., specifically CIDEr score) to maintain the plausibility of NLE after the first training stage. As result, the reward Fooal formulated as follows:
Footal (I, a, C,,,,,,r) T (a, x, + (e, Vans (ans(a, x, r) Z(Par, (a x r)), Kepp (e, é) = 2(CIDEr(e, é))
(5)
(6)
(7)
where x = {v,g) g) is the input image-question pair, denotes the ground-truth answer, denotes the
ground-trutt truth explanation, is the predicted explanation from M, and r E is the sampled rationales
from G.Notably, Z is input-specific normalization function that follows Deng et al. (2022) to
normalize reward for stabilizing the RI. training process.
RLNF Formulation Our RLNF employs Proximal Policy Optimization (PPO) (Schulman et al. 2017) as the RI. algorithm. As the policy model updated, the rationale generator G is maximize the following reward Reoeal
max Brona (,,,,
II Pa(w:lwea),
where r= [w: 12 T T |rl. and {v. g). However, need to ensure the generated rationales are understandable by humans and do not deviate too from the distilled knowledge. To achieve this, we add KL penalty term between the learned policy 0 and the initial policy Ouit after the knowledge distillation phase. Therefore, the overall reward is defined as:
Rioaa(I ê, = (x a, e,e,,,, e) - a log PG(r|2;0) (r(cr O
where Rioral (I, ê, r) is the reward in Eq.
3.3.2 PROMPTING BY IISUALFFAC-BBAE RATIONALE FOR FAITHFUL NLE
Once the rationale generator trained with the introduced RLNF, it is encouraged to produce visual fact-based rationales R that are encapsulated with established facts and visual content from visual input. Again, since R are natural language prompts, they are inherently interpretable by our reasoning module M. Therefore, for the given imagc-question pairs X. utilize R part of input prompts during the reasoning process of M. This ensures the NLEs from M retain plausibility because of the established supporting facts lies in R, together with the enhanced faithfulness because

---

Published as a conference paper at ICLR 2024
of the derived visual content embedded in R. We optimize M to achieve this with the reasoning loss L defined as follows:
LM Σ
(10)
where E R. == and explanation sequence.
= {v,9} E X, and y = (a; e] Y, which is the concatenated ground-truth answer
Therefore, through the complete Rapper training process as outlined in Algorithm I
VQA-NLE tasks
would be successfully enabled with adequate plausibility and faithfulness.
3.4 INFERENCE
At inference time, for given input image-question pair E X, we first generate rationale on the fly from the rationale generator G:
(ws wi Pa(wes, 2); . n),
where y } the sampled rationale, = |r], and x {u, q). Subsequently, we prompt the reasoning module M by concatenating the predicted rationale i with the image-question pair x for outputting the final answer and explanation sequence j. This can be formulated as:
ÿ = [a; él { 2; (M(zes 2, r); = 0, o,..,,
where = |0 and
{z = the concatenated answer and explanation, denoted as [á; E].
4 EXPERIMENTS
4.1 DATASET AND SETUP
We follow (Kayser et al., 2021 Sammani et al. 2022 Suo et al. 2023) and consider two VL-NLE
datasets. VQA-X (Park al. 2018) builds upon VQAv2 dataset (Goyal et al. 2017). It is composed of 32.3K samples, divided into 29K for training. 1.4K for validation, and 19K for testing. e-SNLI- VE (Kayser al., 2021) builds upon e-SNLI dataset (Camburu et al. 2018). consisting of 43K mmage-hypothesis pairs, divided into 40K for training, 1.4K for validation, and 1.6K for testing.
Rapper is consists of rationale generator G and a reasoning module M, are both initialized from the pretrained image captioning model (Li et al. 2023a). The LLM for knowledge distillation during stage(A) is LLaMA-65B (Touvron al. 2023). More implementation details are shown in Sec. A.]
4.2 EVALUATION METRICS
For NLE evaluation, we use BLEU@N (Papineni al., 2002), METEOR (Banerjee & Lavie 2005)
ROUGE-L (Lin 2004), CIDEr (Vedantam et al. 2013), and SPICE (Anderson et al. 2016) as the metrics, while using VQA accuracy evaluate predicted answers. To evaluate the degree of plausibil- ity and faithfulness of explanations, we measure them with CIDEr/SPICE and RefCLIPScore Hessel al. (021), respectively. In addition, build human evaluation for explanation plausibility and faithfulness since automatic metric measures not always reflect the correctness and logicality. Please refer Appendix A.3 for the details of our human evaluation process.
Plausibility To quantitatively evaluate explanation plausibility, we employ CIDEr and SPICE scores. CIDEr measures the similarity between the generated explanation and human -written ground truth sentences, capturing human consensus by introducing tf-idf weight (Vedantam et al. 2015). On the other hand, SPICE converts sentences into semantic scene graphs, allowing evaluation to break grammatical constraints and thus closely resembling human judgment (Anderson et al. 2016).
Faithfulness We adopt RefCLIPScore, which computes the harmonic mean of CLIPScore (Hessel al. (021) and maximal reference cosine similarity, thereby encapsulating the correlation between the explanation and its reference. As noted by Hessel et al. (202T), RefCLIPScore surpasses prior metrics in correlating with human judgment for hallucination detection.
7

---

Published as conference paper at ICLR 2024
VQA-X
Method
B@ 1
B@2
B@3
B@4
METEOR
ROUGE-L
CIDEr
SPICE
Accuracy
PJ-X PPrk et al.
2018
57.4
42.4
30.9
22.7
19.7
46.0
82.7
17.1
76.4
FME
Mooney,
20.4
87.0
18.4
RVT Mardsovic
2020
51.9
37.0
25.6
17.4
19.2
42.1
52.5
15.8
68.6
QA-only Kayser et T
51.0
36.4
25.3
17.3
18.6
41.9
49.9
14.9
eUUG
Kayser al
74.1
NLX-GPT
Samman
2022
64.2
49.5
37.6
28.5
23.1
51.5
110.6
22.1
83.07
S3C
Suo oi al 2023
64.7
50.5
38.8
30.7
23.9
52.1
116.7
23.0
85.6
Rapper (ours)
65.5
51.6
40.5
31.8
24.3
$2,9
124.0
24.5
87.25
e-SNLI-VE
Method
B@2
B@3
B@4
METEOR
ROUGE-L
CIDEr
SPICE
Accuracy
PJ-X [Park 2018
29.4
18.0
11.3
14.7
28.6
72.5
24.3
69.2
FME
[WD Moobey
30.6
19.2
15.6
29.9
83.6
26.9
73.7
RVT
Marasovic u..
2020
29.9
19.8
13.6
9.6
18.8
27.3
81.7
32.5
72.0
QA-only Kayser el al
2
29.8
19.7
13.5
18.7
27.0
80.4
32.1
c-UG
Kayser
2041
30.1
19.9
19.6
27.8
85,9
34.5
79.5
NLX GPT
(Samman sl 2022
37.0
25.3
17.9
12.9
18.8
34.2
117.4
33.6
73.91
Rupper (ours)
40.5
28.1
20.2
14.7
20.8
35,9
128.6
34,9
75.73
Table 1: Quantitative NLE comparisons of filtered results (i.e., NIE evaluation conditioned on correct. answers) on VQA-X and --LLLVVE
Unfiltered
Filtered
Method
6
METEOR
ROUGE-L
CIDEr
SPICE
B@4
METBOR
ROUGE-L
CIDEr
SPICE
Accuracy
Rupper
30.0
23.3
51.3
116.0
23.2
31.8
52.9
124.0
87.3
RLNN
29.4
23.6
51.2
113.0
23.0
31.2
24.5
52.5
120.2
24.2
86.6
RLNF
KD
27.1
21,8
49.7
103.2
20.7
29.3
23.0
51.6
112.1
22.3
85.0
Unfiltered ROUGE-L CIDRr
Filtered
Method
B64
METEOR
SPICE
B64
METFOR
ROUGEL CIDr
SPICE
Accuracy
Rapper
30.0
23.3
51.3
116.0
23.2
31.8
24.3
52,9
124.0
24.5
87.3
Rapper w/o filtering
28.5
22.7
50.8
110.6
22.2
30.1
23.4
52.1
116.7
23.4
86.4
Table 2: Ablation studies the proposed training schemes (up) and the filtering mechanism for
knowledge distillation (bottom). We compare the performances in both filtered and unfiltered settings.
4.3 QUANTITATIVE ANALYSIS
Method
RefCLIPScore(t)
NLE evaluation. Table Table and Table we demonstrate that Rapper outperform previous state-of-the-art methods NLE-related metrics on both VQA-X and e-SNLIV-VE datasets, with filtered and unfiltered settings. The filtered setting in Table [ considers the explanations that are associated with correct answers. Conversely, the unfiltered setting in Table 5 and Table 6 in Appendix A.2 indicates evaluations of explanations without considering the correctness of the corresponding answers.
Madrmont VL-
NLX-GPT S3C
65.09
Our abhand spprrucher
Rapper (wfo KD and w/o RLNF) Rapper (wlo RINF) Rapper
65.66
Table 3: Faithfulness evaluation on the VQA X dataset under filtered setting. Note that higher RefCLIPScore indicates less ballucination.
Plausibility & faithfulness of NLE. We assess the plausibility and faithfulness in NLE through CIDEr/SPICE (in Table D.. RefCLIPScore (in Ta- ble 3), and human evaluation (in Table 4 In ble [ we demonstrate that Rapper outperforms previ- ous state-of-the-art methods NLG metrics both VQA-X and c-SNLI-VE benchmarks, underscoring its superiority in generating plausible explanations. On the other hand, in table Rapper's superior RefCLIPScore indicates fewer hallucinations and increased faithfulness over other VQA-NLE works, although the RefCLIPScore of Rapper (w/o RLNF) is lower due to the hallucinations introduced by knowledge distillation from LLM. Nonethe- less, Rapper still successfully reduce hallucination after the RLNF training. This demonstrates the effectiveness of proposed RLNF enable the model generate faithful NLEs. Lastly, through human evaluation in Table 4 we provide further human- -ennan-eeeeeved evidence for the effectiveness of Rapper for improved NL.B generation.
Method
Plausibility (t)
Faithfulness (t)
NEX-GPT S3C
0.771
0.795
0.797
0.811
Rapper
0.845
0.859
Table 4: Human evaluation on plausibility and faithfulness on VQA-X in the filtered setting,
8

---

Published as a conference paper at ICLR 2024
Multimodal Input
(b)
cluttered?
Q: this an asian country?
What kind animal this?
Methods
GT A: Yes
Sheep
GT
There single with flowers on it
language text
The animal covered wool
public
No
Yes
Sheep
NLX-GPT
There are no objects the table
There is train on the tracks
It has a long face and long nose
SCC
A
No
Yes
Sheep
There are only few itemss
There train the stations
It
and white
R
The table not dutttered because
The presence of asian writing on the train suggests that asion country
A sheep is type animal that has wool its body
only object
Rapper
No
Sheep
There only one object: on
There is asian writing on the train
Its has wool its body
Visualization of output answers and explanations predicted by different methods. Note that words in red denote hallucinated explanations, and those orange denote implausible ones. Words in blue denote faithful and plausible explanations to the input image-question pair.
Figure 3:
evaluate our two-stage approach: (A) KD from LLM and (B) RLNF. Compared to the Rapper baseline without KD and RLNF, our method enhances explanation plausibility and faithfulness, highlighting the importance of both stages.
Ablation on the proposed stages. In top of Table 2
Ablation on the "Filter" mechanism. In bottom of Table 2 our filtering mechanism in knowledge distillation stage outperforms the baseline Rapper without filtering, by effectively removing overly redundant and noisy pseudo rationales that could impair model performance.
VQA-X
78.
Test Accuracy (%)
76-
74.48%
Ablation studies of derived rationales. In Fig. we
demonstrate that introducing two proposed stages im-
72
proves the quality of derived rationales, benefiting the VQA performance of rision-language large model. Specif- ically, we test whether mPLUG-Owl (Ye et al. 2023) can answer accurately when given pair (image, question, and (None, R', R)), where None indicates no ratio- nales as imput, 2 R' indicates the rationales are from Rapper with KD training. and R indicates the ratio- nales are from Rapper with both KD and RLNF training. Notably, find that rationale quality improves progres- sively as we implement the stages we have proposed. This underscores the effectiveness of our designed stages in enhancing rationale quality.
69.84%
68.30%
None
x=R' (rationale w/ only KD)
x=R (rationale w/ KD and RLNF)
Figure 4: Ablation studies of derived ratio- nales. Note the VQA accuracy on the VQA- X dataset is evaluated.
4.4 QUALITATIVE EVALUATION
al.) 2022), S3C (Suo et al., 2023). and our Rapper on the VQA-X dataset. Rapper consistently produces more plausible explanations. For example, Fig 3) highlights ability of Rapper to derive visual facts, such as identifying a single object on the table, surpassing previous methods that might produce hallucinated explanations. Similarly, in Fig Rapper offers plausible explanations like recognizing Asian writing, contrasting with the implausible outputs of prior methods. Additional results and ablation studies are in Appendix A.4
In Fig we compare LEX-GPT(Saannn
5 CONCLUSION
In this paper, we proposed Rapper, two-stage Reinforced Rationale- Rationale-Prompted Paradigm for enabling NLE with sufficient plausible and faithful properties. Our Rapper uniquely distills language- based knowledge from LLM and utilizes RL with natural language feedback from the VQA task, so that the designed rationale generator is able to produce rationales with the aforementioned desirable properties. By prompting such predicted rationales into the reasoning module, we demonstrated that satisfactory VQA performances can be achieved. Compared to SOTA VQA-NLE methods, possible implausible hallucinated explanations be mitigated by Rapper.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
This work is supported in part by the National Science and Technology Council via grant NSCT 12- 2634-F-002-007 and Center of Data Intelligence: Technologies, Applications, and Systems via grant NTU- LLU-1133.00002. We also thank the National Center for High-performance Computing (NCHC) for providing computational and storage resources.
REFERENCES
Afra Feyza Akyürek, Ekin Akyürek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wijaya, and Niket Tandon. RI4f: Generating natural language feedback with reinforcement learning for repairing model outputs. arXiv preprint arXiv:2305.08844 2023.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Ams- terdam, The Netherlands, October 1-14, 2016, Proceedings, Part V 14, pp. 382-398. Springer, 2016.
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6077-6086, 2018.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint 1xXiv:22/2.00073: 2022.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Natural language inference with natural language explanations. Advances Neural Information Processing Systems, 31, 2018.
Soravit Changpinyo, Doron Kukliansky, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut, All you may need for vqa are image captions. arXiv preprint arXiv:2205.0/883, 2022.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, pp. 104-120. Springer, 2020.
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. RLPrompt: Optimizing discrete text prompts with reinforcement learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3369-339) Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URJAstpes//aclaattccooo... orgp2222...../22...... emnlp-main
222
Zhihao Fan, Zhongyu Wei. Siyuan Wang, Yang Liu, and Xuan-Jing Huang. A reinforcement learning framework for natural question generation using bi-discriminators. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 1763-1774 2018
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.1 / 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Star Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6904-69133 2017.
10

---

Published as a conference paper at ICLR 2024
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. arXiv preprint arXi:22044.8718, 2021.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know, arXiv preprint arXiv:2207.05221, 2022.
Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do, Zeynep Akata, and Thomas Lukasiewicz. e-vil: A dataset and benchmark for natural language explanations in Aision-lanuuen tasks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1244-12544 2021.
Daniel Khashabi, Yeganeh Kordi, and Hannaneh Hajishirzi. Unifiedqa-v2: Stronger generalization via broader cross-format training. arXiv preprint arXiv:2202.12355, 2022.
Daniel King, Zejiang Shen, Nishant Subramani, Daniel Weld, Iz Beltagy, and Doug Downey. Don't say what you don't t know: Improving the consistency of abstractive summarization by constraining beam search. arXiv preprint arXiv:2203.068. 084366, 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 2022.
Satyapriya Krishna, Jiaqi Ma, Dylan Slack, Asma Ghandeharioun, Sameer Singh, and Himabindu Lakkaraju. Post hoc explanations language models can improve language models. arXiv preprint arXiv:2305. 1426, 2023.
Zhibin Lan, Wei Li, Jinsong Su, Xinyan Xiao, Jiachen Liu, Wenhao Wu, and Yajuan Lyu. Fact- gen: Faithful text generation by factuality-awan pre-training and contrastive ranking fine-tuning. Journal of Artificial Intelligence Research, 1281281-11 1303, 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint a7X55 2023a.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang. Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-trane pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pp. 121-137 Springer, 2020.
Yuang Li, Yu Wu, Jinyu Li, and Shujie Liu. Prompting large language models for zero-shot domain adaptation in speech recognition. arXiv preprint arXiv:2306. 16002776666007, 2023b.
Chin Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches oul, pp. 74-81, 2004.
Zhihong Lin, Donghao Zhang, Qingyi Tao, Danli Shi, Gholamreza Haffari, Qi Wu, Mingguang He, and Zongyuan Ge. Medical visual question answering: A survey. Artificial Intelligence in Medicine, pp. 102611, 2023.
Feng Liu, Tao Xiang, Timothy M Hospedales, Wankou Yang, and Changyin Sun. ivqa: Inverse visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 161199 2018.
Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannanch Hajishirzi, and Yejin Choi. Rainier: Reinforced knowledge introspector for commonsense question answering. arXiv preprint arXiv:22100030778 2022a.
11

---

Published as a conference paper at ICLR 2024
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Han- naneh Hajishirzi. Generated knowledge prompting for commonsense reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pp. 3154-3169, Dublin, Ireland, May 2022b. Association for Computational Linguistics. doi: 10.18653/v1/2222.ach 225.. URL hwt.t:ttttaaaaat http:////...aaaoooooooooooooe wrw.oogg..rgg//2/22// aeee2aacllloo long
225
Jiaying Lu, Xin Ye, Yi Ren, and Yezhou Yang. Good, better, best: Textual distractors generation for multiple-choice visual question answering via reinforcement learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 4921-4930, 2022a.
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27600 2022b.
Bodhisattwa Prasad Majumder, Oana-Maria Camburu, Thomas Lukasiewicz, and Julian McAuley. Knowledge-grounde grounded rationaliaaiioonnnnn via extractive and natural language explanations. arXiv preprint arXiv:2106. 100006.1876 1376, 2021.
Ana Marasović, Chandra Bhagavatula, Jae sung Park, Ronan Le Bras, Noah A. Smith, and Yejin Choi. Natural language rationales with full-stack visual reasoning: From pixels to semantic frames to commonsense graphs. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.128.77 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. Multimodal explanations: Justifying decisions and pointing the evidence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8779-8788, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
Steven Rennie, Etienne Marcheret, Youssef Mrouch, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7024, 2017.
Fawaz Sammani, Tanmoy Mukherjee, and Nikos Deligiannis. NIx-gpt: A model for natural language explanations in vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8322-8332, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17Xiv:1707.06347, 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruy Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEFE international conference computer vision, pp. 618-626, 2017.
Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Oana-Maria Camburu, and Marek Rei. Log- ical reasoning for natural language inference using generated facts as atoms. arXiv preprint arXiv:2305. 13214, 2023.
Alane Suhr, Mike Lewis. James Yeh, and Yoav Artzi A corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 217-223, 2017.
12

---

Published as a conference paper at ICLR 2024
Wei Suo, Mengyang Sun, Weisong Liu, Yiqi Gao, Peng Wang, Yanning Zhang, and Qi Wu. S3c: Semi-supervised vqa natural language explanation via self-critical learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2646-2656, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023,
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint arXiv:2305. 04388, 2023.
Ramakrishna Vedantam, C Lawrence Zitnick and Devi Parikh. Cider: Consensus-baad based image description evaluation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4566-4575, 2015.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, and Nathan Lambert. Trl: Transformer reinforcement learning. htttes///gtttuub..cmm/vwweerrrre 2020.
Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions evaluate the factual consistency summaries. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. Linguistics. doi: 10.18653/v1 00.553/v12020..ee a0a.n//2222..mmmm.5555000 URL
Online, July 2020. Association for Computational
2020..cl-
450
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 2022.
Jialin Wu and Raymond J. Mooney. Faithful multimodal explanation for visual question answering. ArXiv, abs/1809 809.02805, 2018a.
Jialin Wu and Raymond J Mooney. Faithful multimodal explanation for visual question answering. arXiv preprint arXiv:/809.00 1809.02805, 2018h.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706. 2019.
Cheng-Fu Yang. Yao-Hung Hubert Tsai, Wan-Cyuan Fan, Russ Salakhutdinov, Louis-Philippe Morency, and Frank Wang. Paraphrasing is all you need for novel object captioning. Advances in Neural Information Processing Systems, 35:6492-653 6504, 2022.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303. 2212.33333188 1381, 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14000.4.......... 2023.
Zhuosheng Zhang, Aston Zhang, Mu Li. Hai Zhao, George Karypis, and Alex Smola. Multimodal thain-of-thought reasoning in language models. arXiv preprint arXiv:2302.009223 2023.
Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey, arXiv preprint arXiv:2309.01029 2023.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 IMPLEMENTATION DETAILS
Given an image-question pair x = (v. q) E X, the rationale generator G first generate rationales R that contains both rich knowledge and visul-grounder facts. The ground-trute answer- explanation pair y = (a, e) E Y is available. Subsequently, conditioned on these rationales as well as the image-question pair, the reasoning module is enable generate the answer á A as well as NLEs E E that are sufficiently plausible and faithful. Specifically, our approach follows a series of steps outlined in Algorithm We use 8 V100 GPUs to perform the above training algorithms.
Stage(A): KD from LLM In Sec. 3.2.1 we gather pseudo rationales for each
pairs using in-context learning to prompt LLaMA-65B (Touvron et al. 2023). To ensure the quality of these pseudo rationales,
employ UnifiedQA (Khashabi al., 2022) for filtering, keeping only those rationales whose predicted answers have a token-level FT score surpassing a threshold T (follow Changpinyo et al. (2022) to set to 0.54 (annuully).Nexx,, proceeding Sec. (.2.2) we train rationale generator G for 10 epochs using the distillation loss Lg. The input contains a image and input textual template, formed by concatenating the question with the filtered pseudo rationale, represented as [Question: {4} Rationale: {r The ground-truth label template is [ra]] Training settings include a total batch size of 128, a learning rate of 3e-5, and a weight decay of 0.95.
Proceeding to Sec. 3.2.2 we train the reasoning module M with the reasoning loss L. for 15 epochs. Similar to the rationale generator, the input template includes the concatenated question and pseudo rationale, which is formulated as [Question: {q} Rationale: (r/} Answer: {á; ê}}, and the ground truth label template is {aze).
Stage(B): RL for NLE feedback In Sec. 3.3.1 we apply RLNF to continue to train the rationale generator G. For the RL experimental settings. we follow von Werra et al.) (2020), and use their default PPO hyperparameter setting to train for 10 epochs with a batch size of 128. The rationale generator G and reasoning module M both use greedy search to sample the rationales, answers, and explanations for RL optimization.
Finally, in Sec. 3.3.2 we continue to train the reasoning module M for 10 epochs with the same loss LM and similar training parameters. The input contains a image and input template which involves the concatenated question y, our predicted rationale f, and ground-truth answer e and explanation e, which is formulated as: [Question: {4} Rationale: {f} Answer: {a;e}}, e}]. and the ground truth label template is e) During the training period, the rationale generator G samples the rationales on the fly by beam search decoding with beam size of 5. During the evaluation period, both G and M generate rationales and answer-explaaattin pairs by beam search decoding with a beam size of 5.
A.2 UNFILTERED QUANTATIVE RESULTS
VQA-X
Method
BG4
M
R
C
S
&-NLLLVEE
CAPS (ark
2018
5.9
12.6
26.3
71.3
35.2
11.9
Method
B64
M
R
$
PME
& Mirely Nh
24.4
19.5
47.7
88.8
17.9
NLX-GPT Sammani
2022
11.9
18.2
32.5
109.0
33,0
NLX GPT
Stammot e 2022
25.6
21.5
48.7
97.2
20.2
Rapper (puEs)
13.9
20.1
3466
34,9
S3C
27.8
22.5
50.7
104.4
21.5
Ropper (OLES
51.3
116.0
23.2
Table 5: Quantitative comparisons of unfiltered scores on VQA-X datasct.
Table 6: Quantitative comparisons of unfiltered scores c-SNLI-VE dataset.
As demonstrated in Table 5 and Table 6 RAPPER significantly outperforms existing VL-NLE methods across all metrics on both VQA-X and e-SNLI-VE datasets. It's notable that Rapper surpasses the second- best results by 11.6 and 12.6 in CIDEr on VQA-X and e-SNLI-VE datasets, representing a relative improvement of 11.1 % and 11.6%, respectively.
14

---

Published as a conference paper at ICLR 2024
A.3 HUMAN EVALUATION PROCESS
We follow the evaluation setting/process applied in NLXGPT (Sammani et al., 2022) and S3C (Suo al., 2023) (i.e., Iwo SOTAs on VQA-NLE), randomly select 200 test samples from the VQA- X (Park et al. 2018) dataset with correctly predicted answers. Then, subjective evaluation is performed by different annotators. Note that each annotator has select one out of 4 choices: yes, weak yes, weak no, and no, as response to whether the explanation justifies the answer. And, these 4 decisions are numerically mapped to 1, 2/3, 1/3, and respectively. With averaged results obtained for each method, we present the performance comparisons in the following table. From this table, we see our proposed Rapper is preferable by the users in terms of subjective plausibility and faithfulness assessment. This conclusion also aligns with the objective quantification evaluation of Table 1 presented in the main paper.
A.4 MORE QUALITATIVE RESULTS
Multimodal
Input
(b)
(c)
Hypo: Girls are gessiping GT A: Contradictinn
The shooting with bow GT Contradicton
Hypo: motortyciitt wearing
Methods
Contradictin
boys
GT E: The man can't using rifle and bow the same time
GT E: Yellow and blue are different colors
Neutral
Entallient
Entallment
NLX-GPT
doesn't mean they are gossiping
same as man is shooting with bow
motorcyclist wearing
matorcyclist
R
image shows boys, contradicts the hypothests Contradiction
The image shows using which contradicts the hypothesis Contradictin
The image shows matorcyclit wearing yellow, which contradicts the hypothesks. Contradiction
Rapper
Å
The cannot shooting and rifee at the same time
The motorcyclist cannot wearing and blue at same time.
Bays and girls
same
Figure Visualization output answers and explanations predicted by different methods on e-SNLI- VE dataset. Note that words in red denote hallucinated explanations and those in blue denote faithful and plausible explanations to image.
Multimodal
Input
smartphane7
Q: cioudy?
Q: Should the men be weering helmet? GT A: Yes
Methods
phone has black
completely
fall
white screan
get
A smartphone device has screen, a camera, and a web browser
The presence dloud sky: suggests that it is cloudy
Wearing helmnet riding motorcydle not recommendedd
Rapper (w/o RLNF)
No
It has touch screen,
and web
There clouds the
He riding scooter
R
The presence a physicca keyped on the device that it not smartphone
The abseece of clouds in the sky suggests that it not cloudy No
The men is riding motorcycle, which is known to be dangerous, he should wearing helmet Yes
Rapper
No
phone
The clear blue
He riding matorcycle
Figure 6: Visualization of output rationale, answer, and explanations predicted by the different training stages (A) and (B). Note that words in red denote hallucinated rationales, while those in blue are rationales that derive visual facts from the input image.
In Fig. 5 we present some qualitative results comparing NLX-GPT (Sammani et al. 2022) and our Rapper the e-SNLI-VE dataset. The results demonstrate that Rapper generates more precise
15

---

Published as a conference paper at ICLR 2024
answers accompanied by more faithful explanations. For instance, seen in Fig. (J)), NLX-GPT mistakenly identifies the rifle in the image as a bow. On the other hand, Rapper accurately answers the question, supported by the visually evident fact contained within the prompting rationale that the man is holding a rifle. As shown Fig. 6, we present some qualitative results comparing Rapper without RLNF and our Rapper the VQA-X dataset. Through the comparison, we can observe that RLNF help lower our model's probability of generating hallucinated rationales. As a result, our model can generate more accurate answers and reasonable explanations with the help of RLNF.
We show more qualitative results in Fig.
Image
Question
Rationale
Answer/Explanane GT Answer/GT Explanation
green light indicates that cars allowed to proceed through the intersection, and light to procoed through the intersection
Should the ight?
Yes
light color
the presence suggr the
Pitching throw a baseboll
Pichiin throwing basebal
2
What this man doing?
the man pitching bascball
graduation caps gowns suggests graduation taking place
Graduation weening greduation caps and gowns
Graduation They wearing graduation gowns
heppering?
kind
room suggests
Bathroom
bathroom
Does the look happy?
expression suggests guy happy
No he has an angry look his face
No he ien't smiling and look annoyed
What doling?
he presence of cooking suggests person cooking
Cooking pen food
Cooliing using
What they paparticipeting
The skis are participating skünng
that
Skiing They have skis their poles
Skiing There under
What aind of sandwich is this?
The presercce hot dogs in sandwich suggests that it hot sandwich
Hot dog There are hot dogs it
Hot dog Thoir is hot dog mean in bun
O
What sport being plaed??
The presence of tennis racket that the being
Tennis The men is holding tennis racket
Tennis / The player holding racket
Figure More qualitative results of Rapper compares ground truth answers and explanations.
A.5 FEW-SHOT DEMONS CRTOONTTTAIONSS FOR ELICITING LLAMA TO GENERATE PSEUDO RATIONALES
In Fig. and Fig. 9 we show the task-specific few-shot demonstrations for the VQA-X and e-SNLI-
VE tasks. We these demonstrations prompt LLaMA (Touvron et al. 2023 with in-context learning.
16

---

Published as a conference paper at ICLR 2024
Please rationale that justifies the following explanation
question:
foreground run
answer: no
explanation: bicycles can not be ridden on the water
retionale: the rationale is that bicycles are designed to be operated on solid ground, so unikkely that would run on water
quection: yes
look excited
explaration:
face retionale: the rationale is that arms in the air and smile are indications of excitemenn
question: this recent picture
answer:
explanation: because the tennis player has modem haircut and is holding a modem racket hairstyfes equipment not typkally
rationale: the
photos, suggesting
picture
question: should this cake be esten by only one person
answer: no
explanation: because large cake party
retionale: the rationale is that large cakes are typically intended to be shared and enjoyed by muttple people, so they should not be eaten by only one person
Figure 8: Few-shot demonstrations for prompting LLaMA (Touvron al. 2023 generate pseudo rationales in VQA-X task.
17

---

Published as a conference paper at ICLR 2024
Please the retionale eccording to the hypothesis, and explenation.
play mud rebinnale: rattonale is
hypothesks:
controdiction
explanatton: boy
dog not
image, so statement contradict.
hypothesis: A clean shaven men is standing in comniield
antwert contradittion
explanation: a man with a goatee is not deen shaven men
rafionale: the rationale is that the clean shaven man doesn't have any facial hair who doesn't have goatee in the image which is contradicted to the hypotheses.
hypothesis: The people are sating the kitchen
answer contradiction
explenation: conversing and eeting different actions
retionale: retionale the people conserving insteed estiing the which
contradkited hypothesis.
hypothesis:
outside
answor:
explaration: men dean
are rationale: retionale is that men clean the outside windows, impling they are outside,
vindows
hypothesls: Two men are sitting on dhairs.
anowor: contradiction
explanation: people who are shiting then can not be in marathon at the same time. rabonale: the rationale is that the image shows people running marathon and there are no chairs in the image, which imples they are not sitting chairs.
hypothesis: person outside. entallment.
explanation: you
outside
rationala: intoonales
typically outdoars, which implics outside.
hypothesis: group of people are welking work.
explaration: just becouse walking down
does not moan they are walking to relionale: the retionale is that there no clear indicetion of people's destination the image, the
hypotheesi remains neutral.
hypothesis: The three young men are riding their own privatto sallboat.
enswer: contradiciion
explanation: they connot simultancously be riding their own privte boat and taking public transporation.
retionele: retionale is thet the image shows the three
taking public trensportation, which
contredicts hypothesis,
hypothesis:
birds steps.
contradition
explanation: there woman image shows women dathing standing near hirds the
you say women
rotionale:
stops, which contradicts hypothesi..
hypothesis: The men stends on the ladder while someone holds it steedy. arower entailinent
explanation: person hokiing ladder means that someone is haking the ladder steady rationale: the intionale is that the image shows someona holding the laddos, which implies that the ladder being hold steady, and therefore supports the hypothesis.
hypothesis: The people are friends.
neutrel
explanetion just because two girls with boy doesn't imply they friends.
rationala: iationale
dear indication rolationship betwoon people the
image, so hypothesss remains
hypotheris: Girl plays an
answor entaliment
explenation: the drums are instrument giths are children therefore they are young.
retionale: the rationale is that the image shows airl playing drums, which supports the hypothess.
hypothesis: boy with blue towel and white hat and goggles getting ready to go home for the day.
enswer: contraditiion
explanation: the boy holding googles cannat wear them at the sa zame time.
rationale: the rationale is that the image shows boy holding goocles on his head, which contradicts the
hypothesis.
hypothesis: sketeboard riding down Uhe street his house is
neutrel
that dear indication image the strect located, the hypothesis remeins neubral.
house
rationale: inttonaleis
hypothesks: Warkers answer: neutrel
up after bomb explodes.
explanation: there may only be the crane openetor, not multiple workers the creme could be in midst. a build, not necessarlyy a clean up rubble occurs for more reasons than when a bomb explodes rationale:: the mitionale is that there no dler indication of numbers of wor kers, worker's activity and evidonon bombing the image.
hypothesis: The male child is on beach.
entaliment
explanation: asian with spiky wearing yellow shirt smilles beach. shows mele child with sokky hair weering yellow shirt and
retionale: retionale is thet smilnng beach, supports the hypothesis.
Figure 9: Few-shot demonstrations for prompting LLaMA Touvron et al. 2023 to generate pscudo rationales in e-SNLI-VE task.
18


Reviews:
[{"review_id": "QA1x10oM1W", "reviewer": "Reviewer_EbLA", "summary": "The paper is about mitigating implausibility and hallucination problems (non-informative or contradicting visual context) for generating natural language explanation (NLE) under VQA problems. To mitigate the issue, the authors introduced a notion of “rationale” which is similar to chain-of-thought prompting. To combat the issue of generating rationale without training data, the authors distill rationale from LLMs into the rationale generator. To penalize hallucinated rationale, “Reinforcement Learning from NLE Feedback” is used. The combination of proposed method brings a marginal improvement on benchmarks.", "questions": "- Please answer my questions in “weaknesses” section. I may raise score if the rebuttal is satisfactory.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is largely well-written and easy to understand.\n-  The function of each component in the method is clear and sound.\n- The method achieves SOTA on NLE benchmarks.", "weaknesses": "- The role of using rationale to improve implausibility and hallucination is unclear. It is well-known that chain-of-thought can improve reasoning. However, it is unclear to me if adding one more step, i.e., rationale could really mitigate hallucination.\n- While the method is sound, I’m not very convinced that we cannot just use large vision language models and perform a chain-of-thought style prompting (which was actually the inspiration of this method)? How does large vision language models (e.g. BLIP-family or LLAVA models) perform?\n- In ablation study table, the impact of proposed method is small. Especially, RLNF effect is small.\n- A clear definition of “rationale” is not presented in the paper. Only mentioned that it is like an “intermediate” just like in chain-of-thought prompting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PPLWE91UJz", "reviewer": "Reviewer_iCAp", "summary": "The authors propose a reinforced rationale-prompted paradigm (Rapper) for natural language explanation (NLE) in visual question answering (VQA). They aim to generate plausible and faithful NLEs to address issues like implausibility and hallucination in existing VQA-NLE methods. Rapper has two stages - knowledge distillation from large language models (LLMs) and reinforcement learning from NLE feedback (RLNF). In stage 1, it elicits pseudo rationales from LLM to encourage plausibility and filters rationales using QA model for quality.\nIn stage 2, it uses RLNF with answer and explanation scores as rewards to inject visual facts into rationales, improving faithfulness.\nRAPPER, when evaluated on VQA-X and e-SNLI-VE datasets, achieves new SOTA on both for NLE metrics. It shows better plausibility via higher CIDEr and SPICE scores compared to prior VQA-NLE methods and demonstrates improved faithfulness through higher RefCLIPScore than previous methods. The approach reduces hallucination and implausibility qualitatively over other approaches.", "questions": "How did you determine the optimal hyperparameters (e.g. threshold τ) for filtering pseudo rationales from the LLM? Was any tuning or analysis done to validate these settings?\nDid you experiment with different LLMs for knowledge distillation? If so, how did the choice of LLM impact the quality of the distilled rationales?\nYou mention the potential to extend Rapper to other vision-language tasks. What challenges do you anticipate in adapting the approach to other datasets and tasks?\nThe elicitation process for pseudo rationales is a key component of your approach but is not analyzed in depth. Can you provide more details on this process and how the prompts were designed?\nCan you discuss any trade-offs between plausibility and faithfulness you observed? Does optimizing one tend to hurt the other?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper offers a novel two-stage approach to inject both language-based facts and visual content into rationales.\nIt leverages powerful knowledge and reasoning capabilities of LLMs through distillation. RLNF provides a way to align rationales with visual input for faithfulness. Rationale prompting is interpretable and improves reasoning module's NLE. Training is end-to-end, does not need ground truth rationales.\nMoreover, the qualitative results show more precise and reasonable NLEs: if achieves new SOTA on VQA-X and e-SNLI-VE for all NLE metrics in both filtered and unfiltered settings. It also shows higher CIDEr and SPICE scores demonstrating enhanced plausibility of NLEs.\nImproved RefCLIPScore indicates increased faithfulness and reduced hallucination.\nAblations validate importance of both knowledge distillation and RLNF stages and analysis of derived rationales indicates progressively better quality.\nQualitative examples exhibit more visually grounded and plausible NLEs than prior methods. It also reduces cases of implausible and hallucinated explanations over other VQA-NLE approaches.\nThe claims seem reasonably supported by the quantitative and qualitative results on the standard benchmarks. The improved performance across NLE metrics substantiates the effectiveness of the Rapper approach for plausible and faithful explanation generation. The ablation studies validate the contribution of the individual components. The qualitative examples provide additional evidence that Rapper produces more precise and reasonable rationales and explanations.", "weaknesses": "Some potential weaknesses include:\nThe approach relies on eliciting high-quality pseudo rationales from the LLM, but the process for doing so is not extensively analyzed. In fact LLMs, especially smaller ones  (relative to e.g. GPT4) are prone to hallucinations. \nThe impact of different choices of LLM for knowledge distillation is not addressed.\nEvaluation is limited to VQA; extending Rapper to other VL tasks may reveal additional challenges.\nMore human evaluations on plausibility and faithfulness could further validate the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JtMcluxKGN", "reviewer": "Reviewer_iXW4", "summary": "The paper introduces RAPPER, a two-stage Reinforced Rationale-Prompted Paradigm designed to improve Natural Language Explanation (NLE) in Visual Question Answering (VQA) tasks. The first stage employs knowledge distillation from large language models (LLMs) to generate rationales that are fact-based. The second stage introduces a unique Reinforcement Learning from NLE Feedback (RLNF) to incorporate visual facts into the NLE generation. The paper claims that RAPPER outperforms existing state-of-the-art methods in VQA-NLE on two benchmarks, providing more plausible and faithful explanations.", "questions": "How does the complexity of the two-stage approach impact the computational efficiency of the model?\n\nCould you elaborate on how the RLNF stage specifically tackles the hallucination problem in NLE?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper addresses the problem of implausibility and hallucination in NLE for VQA, which is a novel contribution. The two-stage approach combining knowledge distillation and reinforcement learning is also unique and the RLNF technique for incorporating visual facts into NLE is particularly noteworthy.\n\nThe paper is well-organized and clearly articulates the problem, the proposed solution, and its advantages. The use of figures to illustrate the model architecture and the comparison with existing methods is helpful.\n\nImproving the plausibility and faithfulness of NLE in VQA has important implications for real-world applications, such as medical VQA, where interpretability is crucial.", "weaknesses": "The two-stage approach, while novel, adds complexity to the model. It would be beneficial to see a discussion on the trade-offs involved, such as computational cost.\n\nThe paper focuses on VQA tasks, and it's not clear how's performance of the proposed method when it was adapted to other vision-language tasks.\n\nNo human evaluation is conducted regarding the generation quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)