From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation
Authors: Jiaxin Ge
Sanjay Subramanian
Trevor Darrell
Boyi Li
Venue: EMNLP 2023 Main (2023)
Keywords: Vision-Language Models
Visual Reasoning
Vision-Language Explanation
Self Training

Abstract: Addressing the challenge of adapting pre-trained vision-language models for generating insightful explanations for visual reasoning tasks with limited annotations, we present ReVisE: a Recursive Visual Explanation algorithm.  Our method iteratively computes visual features (conditioned on the text input), an answer, and an explanation, to improve the explanation quality step by step until the answer converges. We find that this multi-step approach guides the model to correct its own answers and outperforms single-step explanation generation. Furthermore, explanations generated by ReVisE also serve as valuable annotations for few-shot self-training. Our approach outperforms previous methods while utilizing merely 5\% of the human-annotated explanations across 10 metrics, demonstrating up to a 4.2 and 1.3 increase in BLEU-1 score on the VCR and VQA-X datasets, underscoring the efficacy and data-efficiency of our method.

From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation
Jiaxin 2e and Sanjay Subramanian ! and Trevor Darrell and Boyi Lift
UC Berkeley, CA, USA
Peking University, Beijing, China {gejiaxin, sanjayss, trevordarrell,
edu
Abstract
human-annotated explanations (Sammani al., 2022; Plüster et al., 2022). Nonetheless, such tech- niques, reliant on extensive fine-tuning, fall short in the face of limited annotations. Thus, we propose an approach to amplify the model's own reason- ing capabilities during inference to generate high- quality explanations. Recent research has demon- strated the efficacy of step-by-step reasoning in language and multimodal reasoning, particularly in contexts where samples are limited (Wei et al., 2022b; Lu et al., 2022; Zhang et al., 2023; Ge al.. 2023). As such, we adopt a phased approach, inte- grating visual and linguistic components for step- by-step vision-language explanation.
Addressing the challenge of adapting pre- trained vision-language models for generat- ing insightful explanations for visual reason- ing tasks with limited annotations, we present VisE: a Recursive Visual Explanation algo- rithm. Our method iteratively computes visual features (conditioned on the text input), an an- swer, and an explanation, improve the expla- nation quality step by step until the answer con- verges. We find that this multi-step approach guides the model to correct its own answers and outperforms single-step explanation gen- eration. Furthermore, explanations generated by ReVisE also for few-shot self- training Our approach outper- forms previous methods while utilizing merely 5% of the human- annotated explanations across 10 metrics, demonstrating up to a 4.2 and 1.3 in- crease BLEU-I score on the VCR and VQA- X datasets, underscoring the efficacy and data- efficiency of our method.
valuable annotations
In this work, we introduce the Recursive Visual Explanation (ReVisE) method for generat- ing visual reasoning explanations that surpasses previous methods while using merely 5% of the human-annotated explanations. Initially, we fine- tunc BLIP-v2 (Li et al., 2023) to generate expla- nations on 5% of the dataset. During inference, we generate an initial explanation, then iteratively generate new explanations based on the preced- ing one. Each step involves computing new visual features, guided by the preceding sentence. This sentence and the as inputs to generate a new sentence in the next step. Crucially, Re VisE serves as a dynamic, self- correcting mechanism by progressively redircct- ing visual attention on the image and regenerat- ing the explanation over steps. Additionally, Re- VisE generates pseudo-ground truth explanations for few-shot self-training, producing pscudo-label that considerably aid seff-improvement compared to traditional pseudo-labels.
1 Introduction
Explanations for visual reasoning are important in real-world applications (Anderson et al., 2018; Hendricks et al., 2016) like assistive technolo- gies (Dognin et al., 2020) and interactive learning (Misra et al., 2018), but collecting human annota- tions for these explanations is expensive. The use of language models (LMs) and pre-trained vision- language models (VLMs) have shown promise in explanation generation (Sammani et al., 2022; Plüster et al., 2022). However, generating high quality explanations remains considerable chal- lenge when annotations are scarce (Bayoudh et al., 2021; Suzuki and Matsuo, 2022).
visual features then serve
We evaluate Re VisE on four vision-language nat- ural language explanation (VL-NLE) tasks e- E-SNLI-VE (Do et al., 2020), VQA-X (Park et al., 2018). AOK-VQA (Schwenk et al., 2022), and VCR (Zellers et al., 2019). Our results show improvements across ten evaluation metrics,
Previous work has aimed to ameliorate this is- sue by focusing on enhancing model architecture and subsequent finetuning using large amounts of
Work done while visiting UC Berkeley: Code Available at betps://gitthe cwm/ Equul advising.

---

Encogled ovenes
reglon's ccoupator?
Guiput Rasionale n+1
Image Encoder
Q-Former
Encoder
LMM-Decoder
persond red region is fight
persont region barber Ansver
Answar
Ansser
Input Image
Anower Answeer n13
Ansiver
person] standing (n al barber chast
perseni
unitem slundon dies. Rationale p+2
32 Prior Learned Queries
Rationale
Figure 1: The pipeline of VisE. In each step, QFormer receives the concatenated input, consisting of = 32 pre-trained queries and the explanation generated from the previous step, calculate cross-altention with the encoded image. The output from QFormer, processed further, pairs with the question to guide the frozen LLM in generating next-step explanation.
with enhancements of up to 4.2 and 1.3 in the BLEU-1 score VCR and VQA-X respectively. Furthermore, self teff-trannngg using our method's pseudo-ground truth explanations shows consis- tent progress compared with traditional generation- based self self-training.
downstream tasks.
Vision-Language Natural Language Explana- tion (VL-NLE) VL-NLE tasks demand compre- hensive understanding and reasoning across both vision and language modalities (Kim et al., 2021). There are two prevailing strategies: the first is a modular approach integrating two separate mod- ules-one for predicting an answer, and another for generating an explanation represented by works such e-UG (Kayser et al., 2021), PJ-X (Park et al., 2018), FME (Wu and Mooney, 2018), RTV (Marasovic al., 2022), and QA-only (Kayser et al., 2021). The second approach is unified one that uses single model to generate an answer and explanation simultancously; relevant works include NLX-GPT (Sammani et al., 2022) and OFA-XMT (Pltister et al., 2022). Our work utilizes the more efficient and training-effectivy unified approach. However, these methods fall short in effectively integrating the reasoning aligmment between vision and language, gap we address with ReVisE.
Further in in-depth ablation studies elucidate the im- pact of ReVisE and the insights behind it, indi- cating that sentences can effectively guide visual attention, and that sentence structure and phrasing style are pivotal for few-shot self-training. Our contribution summarized follows:
peplaaation-reprseenttee
We demonstrate that recent pre-trained mod- els require only fraction of the annotations used by previous explanation generation ap- proaches reach the same quality.
We proposed and implemented the Recur- sive Recursive Visual Explanation (ReVisE). method that iteratively refines explanations by re-computing visual features.
Vision-Language Reasoning Vision-language rea- soning cornerstone of vision-language expla- nation generation. (Hendricks et al., 2016) spear- headed the field by deriving visual explanations from deep networks. (Anderson et al., 2022) fo- cused on while (Park et al., 2019) applied temporal reason- ing visual tasks. Recently, chain-of-thought (Wei al., 2021, 2022b,a) has been harnessed to approach tasks using step-by-step reasoning methodology, effectively enhancing the coherence and logical flow of language reasoning (Wei et al., 2022b: Wang et al., 2022a), self-consistency (Wang et al., 2022b; Lyu et al., 2023) and multimodal rea- soning. (Lu et al., 2022; Zhang et al., 2023; Ge et al., 2023)
We show that self-training using ReVisE to produce pscudo-ground truth annotations fur- ther improves the quality of explanations.
2 Related Work
navigation instructions,
Vision-Language Models (VLM) Large Vision-
Language models have showcased significant po- tential in vision-language tasks, including VQA, image captioning, and image-text retrieval retrivval (Li et al., 2023; Alayrac et al., 2022; Bao et al., 2021; Chen et al., 2022; Li et al., 2021, 2020; Wang et al., 2021; Kim al., 2021; Bao al., 2022). Re- cently, BLIPv2 (Li al., 2023) was proposed. This model aligns vision with language through light- weighted transformer architecture (QFormer), ren- dering computationally efficient for training on

---

Few-Shot Self Training Self-training, a technique which uses trained model to generate pseudo- labels for unlabeled data for further model train- ing, improves the model's s robustness (Chen et al., 2020; Hendrycks et al., 2019) and benefits vision- language tasks (Baevski et al., 2022; Zhu et al., 2020; Wu and Mooney. 2019). Few-shot self- training, which trains on small number of samples with pseudo-labels, is used enhance model per- formance when data resources are scarce or train- ing costs are high (Li al., 2019; Mukherjee and Awadallah, 2020; Chen et al., 2021). However, the quality of self self-generated pseudo labels greatly in- fluences the effectiveness of few-shot self-training (Zou et al., 2019; Xie et al., 2020; Li and Zhou, 2005). In this work, we demonstrate that Re VesE generate robust pseudo-exii exprrddttxlnnnnionns beneficial for few-shot vision-language self self-training.
This strategy enhances the integration between vi- sual and language components, leading to better task comprehension.
Given an image denoted as I. We use the image encoder Fimage to encode the image to get the im- age features Ff. We denote K BLIPv2 pretrained tokens as T. These tokens, together with F,, are passed to the QFormer QF, which processes F1 and ' to produce the image queries Q₁:
Qi = [F(msegee (T).T)
(1)
We denote the tokenized prompt as P. with the format "Answer the question by reasoning step by step. Question: {) Answer:". We concatenate Qr and P to form the full input F to the language model I,, then we feed it into the language model and obtain the output generated by language model O:
Iterative computation of visual features The iter- ative computation of visual features based on text have been applied in previous works to refine visual grounding of an image (Yang et al., 2020), which showed that re-computing visual feature can benefit the visual attention. However, how re-computing benefits text generation remains unexplored. Our work focuses the text-generation task and shows that the iterative approach can simultaneously ben- efit the grounding of an image and the quality of the generated text.
O [(concat(Q.,))
(2)
We calculate cross-entropy loss LcF between the generated output O and the ground truth sentence G, which is constructed in the format "[answer] because [explanation]"..
LeE = sSm * log(())
(3)
The model parameters are updated to minimize this loss. Only the parameters of the QFormer and the vision encoder are updated while the parameters of the language model are kept frozen.
3 Method
this section, we first provide an overview of the architecture of BLIPv2 (Li et al., 2023) and how we trained BLIPv2 for VL-NLE tasks. Then, we provide a detailed introduction and pseudo code for ReVisE. Finally, we discuss how RiVisE is employed for self training.
3.2 Recursive Visual Explanation (ReVisE)
Al Algorithm 1 Pseudo Code for RiVEsE
I: Input: Image I, Question Q
2: Output: Final Answer An, Explanation E..
3: Fi (image(!)
4: n
5: while An An-1 do
3.1 Finetuning BLIPv2 on VL-NLE
6:
E, = Tokenize(AA.)
7:
Enjemboitted Embeed(E.)
BLIPv2 a generative vision- lisguaelanguage model that provides a powerful tool for bridging the divide between vision and language. Its architecture fea- tures a lightweight, multi-layer transformer, the QFormer, which computes cross-attention between K = 32 pretrained query tokens and encoded im- age features. Instead of jointly training a text en- coder and vision encoder as in traditional models, BLIPy2 takes novel approach by freezing the lan- guage model parameters and only train the vision encoder and QFormer to convert image features into tokens interpretable by the language model.
8:
Concat,, concat( EennattEE,emmmeeee emberdded. T)
QI. QF(Connat Fr)
10: Ant1, En-1 L(Qi,))
end while
return An, En
Given image I and question Q, we first en- code the image into feature set Fy through the image encoder F1 Emaage (I) and obtain ini- tial image queries using the pretrained K queries through the QFormer Q₁ = QF(Fi, We then initialize our iterative steps indexed by n = 0. At

---

the very first step n O, we feed the image queries Qr and question Q into the model generate an initial answer Ag and explanation Eo.
the language model and the vision encoder, leaving only the QFormer for further finetuning.
Oer arg minGQr ECA Rieee QQF)
Ao, Eo L(concat(Q, Fi))
(4)
(8)
where denotes the loss function, OQF represents the parameters of the QFormer, and &QF are the updated parameters. This finetuning procedure is designed to bolster the model's 's ability to generate accurate and explanatory responses.
For each following iteration n > 0, the output On is of the form "[answer] because [explanation]". We tokenize the explanation part of Oz, denoted En. The tokenized explanation En is then fed through embedding layer.
We contrast this self-training strategy with a tradi- tional approach. In the traditional approach, the model is given the correct answer directly to gener- ate an explanation Figen whereas in our approach Ej is generated through recursive querying. In the traditional self seff-training approach, the model param- eters are updated as follows:
En.embeddded = Emmee((E,)
(5)
We then concatenate En.embedded with the K BLIPv2 pretrained tokens T to create Concat, = cTncal(T, embeddedde))- This concatenated struc- ture Concat, is then passed into the QForerr to calculate a cross attention with the image feature set Fr, which then generates new image query QI, based on the explanation En, T, and F,
gnew argmineqr QF
E(A F Egen (Qr),
By juxtaposing these two self-training strategies, we aim to assess the potential benefits of our pro- posed method, where explanations generated by Re- VisE serve as a corrective mechanism, over the con- ventional approach that relies solely on the model's ability to
Qr = [C(Concttz Fr)
(6)
This new image query Qı, is then used as input to the language model L, which regenerates an explanation and an answer for the next step n + 1. denoted as An+1 and En+1
explanations from provided answers. A pseudo code is in Appendix C.
1nt1, Ex+1 = I(Qmn)
(7)
4 Experiments
This process is repeated recursively until the model converges in its answer.In practice, we limit the maximum iteration number 5 to prevent potential non-convergence. We provide a pseudo code in Algorithm and method pipeline Figure
In this section, we first introduce the basic set- tings including the task formulation, training de- tails, baselines, and metries. Then, we provide detailed experiment results and in-depth analysis for the results.
3.3 VeVi for Self Training
4.1 Settings
ReVisE's recursive querying process allows the model to correct its own answers, which could lead further performance improvement. Leveraging this, we propose few-shot self-training mecha- nism using the explanations generated by Resi, Suppose we have a set of samples S for which we have the ground-truth answers but lack annotated explanations. Initially, we randomly select a few- shot subset S' C S such that the model originally incorrectly answers these instances, but corrects its answers through Re VisE. Let Açorr denote the correct answer and RRVVisE the explanation gen- erated by Visi for the ith sample in S'. We then use these pairs, (Acerr, E.RVVss)) to further finc- tune the model. During this phase, we freeze both
Task Formulation Our focus is Vision- Language Natural Language Explanation (VL- NLE) tasks which demand generating an answer and high-quality explanation given an image- question pair. We test our method on three es- tablished VL-NLE datasets (VQA-X (Park et al., 2018), e-SNLIVVE (Do et al., 2020), and VCR (Zellers et al., 2019)), and provide additional re- sults for AOK-VQA (Schwenk et al., 2022). Ap- pendix E provides detailed dataset descriptions. Implementation Details For finetuning BLIPv2 on VL-NLE tasks, we maintain language model frozen and concurrently fine-tune the vision encoder with the QFormer, adopting a learning rate of le - 5. We use the entirety VQA-X while only selecting

---

Table 1: Filtered Scores comparison for VCR, e-SNLI-VE, and VQA-X against state-of-the-ar models. Our BLIPv2 model is fine-tuned 5% of the VCR and e-SNLI-VE datasets and the complete dataset for VQA-X while others are all finetuned on the full dataset.
B1
B2
B3
B4
M
R-L
C
BS
VCR
PJ-X
21.8
11.0
5.9
3,4
16.4
20.5
19.0
4.5
78.4
FME
23.0
12.5
7.2
4.4
17.3
22.7
27.7
24.2
79.4
e-UG
20.7
11.6
6.9
4.3
11.8
22.5
32.7
12.6
79.0
QA-Only
18.0
10.2
6.0
3.8
11.2
22.0
30.6
11.6
78.9
RTV
18.0
10.2
6.0
3.8
11.2
21.9
30.1
11.7
78.9
OFA-XMT XMT
22.3
13.0
8.0
5.2
11.3
24.3
44.6
17.8
NLX-GPT
24.7
15.0
9.6
6.6
12.2
26.4
46.9
18.8
80.3
VisE (Ours)
28.9
21.7
17.6
14.4
15.3
29.5
27.9
82.2
e-SNLI-VE -VE
PJ-X
29.4
18.0
11.3
14.7
28.6
72.5
24.3
79.1
FME
30.6
12.4
29.9
26.8
RVT
29.9
19.8
13.6
18.8
27.3
32.5
QA-only
29.8
19.7
13.5
27.0
32.1
e-UG
30.1
19.9
13.7
19.6
27.8
34.5
OFA-XMT
32.4
21.8
15.2
10.8
17.9
31.4
108.2
32.8
80.4
NLX-GPT
37.0
17.9
12.9
18.8
34.2
117.4
33.6
Re (Ours)
38.3
26.5
19.0
13.8
19.7
34.7
126.7
34.2
81.5
VQA-X
PJ-X
57.4
42.4
30.9
22.7
19.7
46.0
82.7
17.1
84.6
FME
59.1
43.4
31.7
23.1
20.4
47.1
87.0
18.4
85.2
e-UG
57.3
42.7
31.4
23.2
22.1
45.7
74.1
20.1
87.0
QA-Only
51.0
36.4
25.3
17.3
18.6
41.9
49.9
14.9
85.3
RTV
51.9
37.0
25.6
17.4
19.2
42.1
52.5
15.8
85.7
OFA-XMT XMT
64.0
49.4
37.6
28.6
23.1
51.0
110.2
22.6
86.8
NLX-GPT
64.2
49.5
37.6
28.5
23.1
51.5
110.6
22.1
86.9
Re VisE (Ours)
64.6
50.0
37.7
28.2
23.2
51.8
108.9
22.6
88.1
a random 5% subset from e-SNLI-VE and VCR and AOK-VQA. Under the few-shot self-training scenario, we use 32 examples and exclusively fine- tune the QFormer, applying a learning rate of le 66. More implementation details provided in Ap- pendix B.
human evaluations. Details of these metrics are available in Appendix D. In accordance with estab- lished methods, we present filtered scores that rep- resent results for explanations accompanied by cor- rect answers. Additionally, for instances where incorrect answers were given, providing comprehensive view of the model's performance.
also report scores
Baselines For finetuned BLIPv2, we compare with previous state of the art models that uses ci- ther unified approach or modular approach on the three VL-NLE datasets, incluing e-UG (Kayser et al., 2021), PJ-X (Park et al., 2018), FME (Wu and Mooney, 2018), RTV (Marasovic al., 2022), QA-only (Kayser et al., 2021), NLX-GPT (Sam- mani al., 2022), OFA-XMT (Plüster al., 2022). We provide backbone information in Appendix A. Evaluation Metries keeping with established practices, we employ N-gram scores, including BLEU (Papineni et al., 2002), METEOR (Baner- jee and Lavie, 2005), ROUGE (Lin, 2004), CIDEr (Vedantam et al., 2015). SPICE (Anderson al., 2016), and BERTScore (Zhang et al., 2019). We also use
4.2 Finetuned BLIPv2
In Table 1, we present our method's performance against other state-of-the-an models using filtered scores for consistency. Leveraging only 5% the VCR and e-SNLI-VE datasets and the en- tire VQA-X dataset, we managed to match or ex- ceed benchmark scores with substantially less data. This highlights that advanced pre-trained models like BLIPv2 can achieve comparable performance using fewer annotations. The unique design of BLIPv2, which preserves the language model while transforming visual features into language model- interpretable tokens, offers promising avenue for future vasiun-language model architecture research.
recent metric, G-Eval (Liu al., 2023), which uses GPT4 (Bubeck al., 2023) and Auto- Aito-Chain-Goe Of-Thoughh (Zhang et al., 2022) for evaluation that has been shown to align better with

---

Question and Answer
Q: Why does person1 in red region have his back person2 yellow region and person3 in blue region? A/He is talking on phone.
Q: What game is the woman playing?
A: frisbee because plastic disk is being thrown and caught air
He is standing in front of person2 yellow region and person3 in blue region and he trying to make barrier between them.
Step1
She about to a tennis ball with racket
She is halding tennis racket and about to tennis ball
Step2
is trying to make barrier between him and person3 blue region.
He talkknng on the phone and is looking away from persion2 yellow region and person3 blue region
She holding frisbee in her hand
Step3
Figure 2: We provide case study of the ViVisE process. We use grad-cam to visualize how the visual attention changes along with how language explanation changes over steps.
Table Re VisE Improvement Scores for VQA-X, eSNLI-VE, AOKVQA, and VCR. Our approach was evaluated samples initially misinterpreted by the BLIPv2 model. The score of state of the art model NLX-GPT the same metric is also provided for reference.
B1
B2
B3
B4
M
R-L
S
BS
G-Eval
V-
NLX-GPT
31.9
20.3
13.3
9.0
16.4
27.8
85.1
31.2
79.52
5.42
Ours (w/o ReVisE)
35.0
22.7
15.2
10.3
17.9
29.9
101.0
30.6
79.30
6.21
Ours(w/ ReVisE)
36.7
23.7
15.8
10.8
18.2
31.1
104.0
31.4
79.90
6.64
VQA-X
NLX-GPT
51.7
35.3
23.6
16.1
16.9
40.3
59.1
14.5
83.77
2.62
Ours(w/o ReVisE)
51.1
34.3
22.6
14.6
15.8
39.6
51.7
12.6
83.28
Ours(w/ ReVisE)
54.8
37.3
25.0
16.2
17.6
41.3
62.6
15.0
83.52
4.24
AOK-VQA
NLX-GPT
55.1
38.3
27.1
18.1
16.2
44.0
57.4
14.3
85.43
4.12
Ours(w/o ReVisE)
57.5
39.9
28.1
19.0
16.5
15.3
86.36
Ours(w/ ReVisE)
59.7
41.5
28.9
19.7
17.7
44.6
60.4
85,86
4.82
VCR
NLX-GPT
9.7
3.2
20.1
24.5
12.6
73.64
Ours(w/o ReVisF)
26.7
19.4
15.6
12.7
13.1
24.6
19.6
21.7
79.25
3.65
Ours(w/ ReVisE)
27.2
20.3
16.4
13.4
14.1
26.2
28.7
23.7
79.35
3.97
4.3 Recursive Visual Explanation VRsV)
elucidate how ReVisE guides the model's ss attention allocation over steps. While initially, the atten- tion maps are broad or focus on areas irrelevant to the question, Re VisE'ss language-guided proce- dure redirects the model's attention towards areas pertinent to the question at hand, suggesting im- provement in the model's interpretability.
In Table 2, we showcase Re VisE's impact aug- menting model performance. As our approach aims at self-correcting and refining initially poor-quality explanations, we evaluate ReVisE on samples ini- tially misinterpreted by the BLIPv2 model. The process involves using recursive language querying to extract pertinent image features, progressively refining the model's 's output. We find that ReVisE persistently enhances the quality of the generated cxplanations, underscoring the critical role of lan- guage as guide for image feature extraction. Figure 2 exhibits representative examples from the VL-NLE datasets, clearly demonstrating the self- correcting mechanism of VisE. Employing grad- CAM visualizations (Selvaraju et al., 2017), we
By taking the explanation from one iteration and using as input for the next, the model refines its interpretation and visual attention. Conceptually, it's analogous to person rephrasing a statement repeatedly to enhance clarity.
4.4 Few-Shot Self- Training
In Table3, we show results for few-shot self- training. We use explanations generated by Re-

---

Table Performance comparison Re VisE in a few-shot self-training context for e-SNLI-VE, VQA-X, AOKVQA. and VCR. The table depicts results without self-training with traditional self-training, and with Re self-training. We use 32-shot in all these experiments.
B1
B2
B3
B4
M
R-L
C
BS
G-Eval
e-SNLI-VE
No Self-train
35.0
22.7
15.2
10.3
17.9
29.9
101.0
30.6
79.30
6.21
w/o ReVisE
34.9
22.7
15.3
10.4
17.9
29.8
100.7
30.5
79.21
6.49
wiReVisE
36.2
23.5
15.8
10.9
18.2
30.5
103.2
30.7
79.61
6.75
VQA-X
No Self-train
51.1
34.3
22.6
14.6
15.8
39.6
51.7
12.6
83.28
2.98
w/o ReVisF
51.2
34.1
22.4
14.3
15.9
39.6
50.6
12.6
83.00
3.21
WisE
53.5
36.6
24.8
16.2
16.9
40.7
58.9
13.8
83.65
4.41
AOK- VQA
No Sclf-train
57.5
39.9
28.1
19.0
16.5
44.4
59.1
15.3
86.36
4,46
w/o ReVisE
57.3
40.2
28.4
19.2
16.6
44.8
61.1
15.7
86.44
4.44
w/ReVish
60.0
41.1
28.7
19.6
18.6
45.1
62,4
18.1
85.28
4.71
VCR
No Scif-train
26.7
19.4
15.6
12.7
13.1
24.6
19.6
21.7
79.25
3.65
w/o ReVisE
26.9
19.6
15.7
12.7
13.3
25.2
21.1
21.9
79.35
3.99
w/ReVish
27.1
20.1
16.2
13.3
13.7
25.4
21.6
23.1
79.55
4.14
There clock wall above stairs
There clock on the wall
squirrel jumping the attacking small child
Just because a squireel jumps does not meen it attacking a small child
Figure 3: We display the distribution of convergence steps, indicating the percentage of samples that reach convergence each respective step. We show results of C-SNLI-VE, VQA-X, AOKVQA and VCR and found that most samples converge by step2 and at least 90% samples converge by step3.
They are looking down the balcony:
They are standing on the balcony looking down.
Figure 4: Comparison between pseudo-explanationn generated by ReVisE(in the box above) and pseudo- explanations generated directly providing groundtruth answers(in the box below).
VisE to self-train on samples that are initially incor- rect but self-corrected during the Re VisE process. When compared to providing the model with the correct answers directly let it generate an ex- planation the same samples, self-training with Re VisE explanations led better performance, in- dicating the model benefits more from its own rea- soning process than simply digesting the correct answers.
Additionally, we explored the impact of varying the number of self-training samples. As shown in Table 5, while any addition of few-shot samples en- hances self-training, even as few as 8-shot samples can improve the model's performance.
4.5 Ablation Study
Qualitative results in Figure reveal minor seman- tic differences between ReVisE-generre generated pseudo- explanations and explanations crafted with pro- vided ground-truth answers. The variations typ- ically lie in phrasing style sentence struc- tures, suggesting that attention sentence struc- ture or phrasing pattern could prove crucial for high-quality pseudo-exphe explanttxplanniionss sho-shot self- training.
Implicit VS Explicit Language In our approach, we forward both the integrated K queries and the language queries after cross-attention. We compare this procedure with forwarding only K queries after cross-attention, as illustrated in Figure 6. The K queries integrate language information implicitly through the cross-attention process but do not forward the encoded text directly.

---

Table 4: Ablation study examining the impact of lim- iting VeVisE iterations to 2 and 3. Since up 90% of samples converge by the third step, constraining itera- tion steps yields strong results.
Table 6: Ablation Study for three VL-NLE datasets. ' refers incorporating language signal implicitly and 'E' refers to incorporating language signal explicitly. Generally, 'E' outperforms T..
B1
B4
M
R-L.
C
S
B1
B4
M
R-L.
C
S
C-SNLI-VE
e e-SNLI-VE
VE
2Steps
36.6
10.6
18.1
31.0
103.0
31.0
35.0
10.4
17.9
29.9
100.8
30.6
3Steps
36.7
10.8
18.2
31.1
103.2
31.4
E 36.7
10.8
18.2
31,1
104.0
31.4
VQA-X
VQA-X
2Steps
54.6
15.7
17.5
41.0
59.6
14.9
51.0
14.8
16.0
39.9
52.4
12.5
3Steps
54.8
16.2
17.6
41.3
60.6
15.0
E
54.8
16.2
17.6
41.3
62.6
15.0
VCR
VCR
2Steps
27.0
13.4
14.0
26.0
28.3
23.6
24.1
11.1
12.0
24.3
18.7
19.4
3Steps
27.2
13.4
14.1
26.2
28.7
23.7
E 25.5
11.6
12,4
24.5
20.0
19.8
Table Ablation study investigating the effect of vary- ing sample sizes for few-shot self-training. We present results for 8-shot, 16-shot, and 32-shot self-training approaches, using pseudo-explanatioos generated by Visis
What kind of enemy are person2 in yellow region and the rest against? A: A very large and dangerous one.
They ara wearing sunglasses and holding
Step3
B1
B4
M
R-L
They are wearing dark sunglasses and lot of black clothing.
--SNLIIVE
8-shot
35.5
10.5
17.9
30.1
101.5
30.6
Step2
16-shot
35.7
10.6
18.0
30.2
101.8
30.5
They are wearing sunglasses are all wearing suiks.
Case: Explanation Gets Worse Over Stepa
32-shot
36.2
10.9
18.2
30.5
103.2
30.7
Stepa
VQA-X
O: Why person1 in red region's mouth ajar?
8-shot
53.3
15.8
16.7
40.6
57.4
13.2
A: person1 In red region is surprised joke person2 yellow region made.
16-shot
53.3
15.7
16.7
40.3
56.9
13.6
32-shot
53.5
16.2
16,9
40.7
58.9
13.8
VCR
person2 in yellow regian is moving to kiss personi red region.
8-shot
26.9
12.9
13.3
25.2
20.7
22.3
16-shot
27.1
13.1
13.5
25.3
21.4
22.6
32-shot
27.1
13.3
13.7
25.4
21.6
23.1
Step1, Step3.
person1 in red region is smiling person2 in yellow region laughing.
The ablation study results in Table indicate that implicit language integration through the K queries alone does not significantly enhance per- formance. Explicitly combining language queries offer crucial semantically-gronnde context which cannot be captured by the K learned queries alone, thus providing more substantial advantage in refining the model's image comprehension.
Fallure Case: Explanation Never Converges
Step2, Step4,
Figure 5: Failure cases when iterations doesn't 't converge or adding more iterations worsens the performance.
Failure Cases We notice certain instances (2%) where additional iterations negatively affect the quality of the generates explanations, as illustrated in Figure 5. For most failure cases, the model enters into recursive loop. In some others, the model ini- tially generates explanations closely aligning with the ground truth but diverged with subsequent iter- ations. This reveals the importance for a balance between the depth of reasoning and model certainty in recursive reasoning.
Limit Iteration Steps Recursive querying may be time-consuming, so we limit the maximum number of steps to 2 and 3 to investigate its impact. As shown in Table 4, limiting the steps to achieved performance nearly on par with that of unrestricted steps. Furthermore, Figure 3 presents the percentage of samples that reach convergence each respective step, indicating that most samples converge by the second step and 90% of the samples converge by step3. The e-SNLI-VE samples exhibit the fastest convergence, potentially due to the simplicity their answer options.
Data Efficiency We provide further ablation on the amount of training data used. On the e-SNLI- VE dataset, we tried 1%, 3%, and 5% of the dataset and report the filtered score. The results are shown in Table 7. This illustrates that our model, leverag-

---

Ethics Statement
Radionale n
Audionaee
Quaries
Encoded Rational
The proposed methods, rooted in the principles of transparency and interpretability, promote the ethical goal of developing Al systems that are easily comprehensible and verifiable. By enabling Al to generate more coherent explanations, we contribute to the objective of trustworthy AI.
Qormme
Q-Former
.
Rationale
Rationate
Learned Querles
32 Prier Learnet Querles
Prier
Forward encoded querie and the ancoded Rationale
Forward queries
Figure 6: Comparison between forwarding all the en- coded queries with forwarding only the queries after cross-attention.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-2371
ing recent advancements in pre-trained models, can deliver high-quality explanations even with sub- stantially fewer annotations than traditional meth- ods.
Nathan Anderson, Caleb Wilson, and Stephen D. Richardson. 2022. Lingua: Addressing scenarios for live interpretation and automatic dubbing. In Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas (Volume 2: Users and Providers Track and Govern- ment Track), pages 202-209, Orlando, USA. Associ- ation for Machine Translation in the Americas.
Table 7: Ablation Study for the amount of data used on e-SNLI-VE dataset. We tried using 1%, 3% and 5% of the training data and reported the filtered scores.
B4
M
R-1.
1%
37.0
13.4
19.4
33.9
122.3
33.5
3%
38.0
13.6
19.7
34.5
126.7
34.0
5%
38.3
13.8
19.7
34.7
126.7
34.2
Peter Anderson, Basura Fernando, Mark Johnson, and Spice: Semantic proposi- tional image caption evaluation. In Computer Vision- ECCY 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 382-398. Springer.
Stephen Gould. 2016.
5
Conclusion
this paper, we introduce ReVisE, a method for generating natural language explanations for visual reasoning tasks with a small number of annotations. We demonstrate its high performance in generating language explanations using significantly less data compared to existing models, enabling the model's self-correction mechanism, and effectively facili- tating few-shot self-training by generating robust pseudo-epplanations. Our work raises the question whether recursive procedures like Re can be used to improve performance in other multimodal domains as well.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauff, lan Reid, Stephen Gould, and Anton Van Den Hengel. 2018. Vision- and Lad-lang navigation: Interpreting visually- grounded navigation instructions in real environ- ments. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3674- 3683.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433.
Limitations
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. 2022. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, pages 1298-1312. PMLR
Althought BLIPv2 has a large potential for vision- language explanation, it might encode social bias. As (Chuang et al., 2023) illustrated, vision- language models have been shown to inherit biases from their training datasets. Conducting thorough investigation of the potential bias of BLIPv2 and addressing would be an important future work. Also, further enhancing the method identify and address the failure cases also a future work to improve this method.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved cor- relation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summariza- tion, pages 65-72.

---

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2021. Beit: Bert pre-training of image transformers. arXiv preprint arXiv: 204iv:2206.000 08254.
Jiaxin Ge, Hongyin Luo, Siyuan Qian, Yulu Gan, Jie Fu, and Shanghang Zhan. 2023. Chain of thought prompt tuning in vision language models. arXiv preprint 230:02300.07911.
Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal. Subho- jit Som, Songhao Piao, and Furu Wei. 2022. Vlmo: Unified vision-language pre-training with mixtrr-of- modality-expertt, Advances in Neural Information Processing Systems, 35:32897-32911
Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. 2016. Generating visual explanations. In Computer Vision-ECCV 2016: 14th European Conference, Am- sterdam, The Netherlands. October 11-14, 2016, Pro- ceedings, Part IV 14, pages 3-19, Springer.
Khaled Bayoudh, Raja Knani, Fayçal Hamdaoui, and Abdellatif Mtibaa. 2021. A survey on deep multi- modal learning for computer vision: advances, trends, applications, and datasets. The Visual Computer, pages 1-32.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. 2019. Using self-supervised learn- ing can improve model robustness and uncertainty. Advances in neural information processing systems, 32.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1900
Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do, Zeynep Akata, and Thomas Lukasiewicz. 2021. e-vil: A dataset and benchmark for natural language expla- nations in vision- Isionnlanguage tasks. Proceedings of the IEEE/CVF international conference on computer vision, pages 1244-1254.
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- herg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint arXiv:2303.1271 12712.
Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-languaae transformer without convolu- tion or region supervision. In International Con- ference on Machine Learning, pages 5583-5594, PMLR.
Ting Chen, Simon Kornblith, Kevin Swersky, Moham- mad Norouzi, and Geoffrey E Hinton. 2020. Big supe-suue models are strong semi-supervised learners. Advances in neural information processing systems, 53:22243-22255.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-ima -iggge pre- training with frozen image encoders and large lan- guage models. arXiv preprini arXiv:2301. 125997.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Pier- giovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. 2022. Pali: A jointly-scaled mul- tilingual language-image model. arXiv preprint 200.69209
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty. Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems. 97-5999-7775..
Yiming Chen, Yan Zhang, Chen Zhang, Grandee Lee, Ran Cheng, and Haizhou Li. 2021. Revisiting self- training for few-shot learning of language model. arXiv preprint
Ming Li and Zhi-Hua Zhou, 2005. Setred: Self-training with editing. In Pacific-Asia Conference on Knowl- edge Discovery and Data Mining, pages 611 12-12. Springer.
Ching- Yao Chuang, Varun Jampani, Yuanzhen Li, Anto- nio Torralba, and Stefanie Jegelka. 2023. Debiasing vision-language models via biased prompts. arXiv preprint 2rxiv:2302.000770
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2020. Unimo: Towards unified-modal understanding and generation via cross- contrastive learning. arXiv preprint arxiv:2012.15409
Virginie Do, Oana-Maria Camburu, Zeynep Akata, and Thomas Lukasiewicz. 2020. e-snli-ve: Corrected visual-textual entailment with natural language ex- planations. arXiv preprint 20X1::000..0344..
Xinzhe Li, Qianru Sun, Yaoyao Liu, Qin Zhou, Shibao Zheng, Tat-Seng Chua, and Bernt Schiele. 2019. Learning to self-train for sepi-super few-shot classification. Advances in neural information pro- cessing systems, 32.
Pierre Dognin, Igor Melnyk, Youssef Mrouch, Inkit Padhi, Mattia Rigotti. Jarret Ross, Yair Schiff, Richard A Young, and Brian Belgodere. 2020. Im- assistive technology: Lessons
Chin-Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Sunmariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
age captioning learned from vizwiz 2020 challenge. arXiv preprint arXiv:2012.1166 1699..

---

Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer.
international conference on computer vision, pages (641.
Bjöm Plister, Jakob Ambsdorf, Lukas Braach, Jae Hee Lee, and Stefan Wermter. 2022. Harnessing the power of multi-task pretraining for ground-truth level natural language explanations. arXiv preprint
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. Gpteval: Nig evaluation using gpt-4 with better human align- ment. arXiv preprint arXiv:2303. 16634.
Fawaz Sammani, Tanmoy Mukherjee, and Nikos Deli- giannis. 2022. Nix-gpt: A model for natural language explanations in vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 8322- 8332.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances Neural Information Processing Systems, 35:2507-2555
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022, A-okvqa: A benchmark for visual question answering using world knowledge. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part VIII, pages 146-162. Springer.
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of- thought reasoning, arXiv preprint arXiv:2301.13377
Ramprasaath R Selvaraju. Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra, 2017. Grad-cam: Visual explanations from deep networks via gradient- bassed localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626.
Ana Marasovic, Iz Beltagy, Doug Downey, and Matthew Peters. 2022. Few-shot self- rationalization with nat- ural language prompts. Findings of the Associa- tion for Computational Linguistics: NAACL 2022, pages 410-424, Seattle, United States. Association for Computational Linguistics.
Masahiro Suzuki and Yutaka Matsuo. 2022. A survey of multimodal deep generative models. Advanced Robotics, 36(5-6))261-278.
Ishan Misra, Ross Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta, and Laurens Van Der Maaten. 2018. Learning by usking questions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11 11-20.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image de- scription evaluation. In Proceedings of the IEEE conference computer vision and pattern recogni- tion, pages 4575.
Subhabrata Mukherjee and Ahmed Awadallah. 2020. Uncertainty- aware self-training for few-shot text clas- sification. Advances in Neural Information Process- ing Systems, 01:21199-21229
Boshi Wang. Xiang Deng. and Huan Sun. 2022a. Itera- tively prompt pre-trained language models for chain of thought. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 27 2714-2730.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 31 111-318.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Lee, Fd Chi, and Denny Zhou. 2022b. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.
Dong Huk Park, Trevor Darrell, and Anna Rohrbach. 2019. Robust change captioning. In Proceedings of the IEEF/CVF International Conference on Com- puter Vision. pages 4624-4633.
Zirui Wang. Jiahui Yu, Adams Wei Yu, Zihang Dai, Yu- lia Tsvetkov, and Yuan Cao. 2021. Simvlm: Simple visual language model pretraining with weak super- vision. arXiv preprint arXiv:2108,
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. 2018 Multimodal explanations: Justifying decisions and pointing to the evidence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8779-8788.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoe Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint 1rXiv:2109.01655
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- to-sentence models. In Proceedings of the ICEE
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama. Maarten Bosma, Denny Zhou, Donald Metzler, et al.

---

2022a. Emergent abilities of large language models. arXiv preprint arXiv:2206.07668
A Model Backbone
We present model parameters and vision trans-
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten
former backbone for the different models in Table
Bosma, Edd Chi, Quoc Le, and Denny Zhou. 2022b.
Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1900.
8
Models FME
Backbone
Trainable Params
Jialin Wu and Raymond Mooney. 2019. Self-critical reasoning for robust visual question answering. Ad- vances in Neural Information Processing Systems, 32.
ResNet-101
142M
RVT
ResNet- 101
277M
e-UG
ResNet 101
277M
OFA-X
ResNet152
472M
Jialin Wu and Raymond Mooney. 2018. Faithful mul- timodal explanation for visual question answering.
NLX-GPT
VIT
182M
arXiv preprint prxivv:809.02880
Ours
VIT
108M
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quae V Le. 2020. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698.
Table The vision backbone for FME, RVT, e-UG, OFA, NLX-GPT and Ours.
B Additional Implementation Details
Zhengyuan Yang, Tianlang Chen, Liwei Wang, and Jiebo I Luo. 2020. Improving one-stage visual ground- ing by recursive sub-query construction. In Com- puter Vision-ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part XIV 16, pages 387-404 Springer.
We provide additional implementation details. When training on BLIPv2 we use beam search with num beam = 5 during decoding. For AOK-VQA- wc also set length penalty to -1, consistent with the original BLIPv2. During training, we use co- sine annealing sceduler and Adam optimizer and train for 6 epochs. Since BLIPv2 does not have any regional proposals, we followed (Zellers et al., 2021) and add colored bounding boxes around the people/objects referred and refer to them as "per- son 1 in red region" or "person2 in yellow region". As (Zellers et al., 2021) demonstrates. through fine- tuning, the model learns a matching between the color referred to in language and the color denoted in the image.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Vi- sual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pat- lern recognition, pages 6731..
Rowan 7ellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34:23634-236511.
Tianyi Zhang, Varsha Kishore. Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint
C Pseudo Algorithm For ReVisE self-training
a7Xiv:1904.0965.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompt- ing in large language models. arXiv preprint arXiv:2210.03493.
We also provide pseudo code for self-training in algorithm 2, which is a pscudo code description of the self-training process in the Method section.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multi- modal chain-of-thouuhh reasoning in language mod- cls. arXiv preprint 000.0033.
D Detailed Metrics
We provide details of the recent new metric G- Eval (Liu et al., 2023). This metric commences by formulating task and employs GPT3.5 (Brown et al., 2020) and GPT4 (Bubeck et al., 2023) to autonomously generate evaluation steps using the Auto Chain-of-Thought (AutoCoT) (Zhang et al., 2022). Subsequently, the task instruction along with the AutoCoT evaluation steps and the sample under consideration are fed to the GPT model to- gether to obtain a comprehensive score from 1-10.
Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. 2020. Vision-language navigation with self- supervised auxiliary reasoning tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10012- 10022.
Yang Zou, Zhiding Yu. Xiaofeng Liu, BVK Kumar. and Jinsong Wang. 2019. Confidence regularized self-training, In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 5982-5991.

---

Algorithm 2 VisE for Self Training
1: Input: Model M, Samples S. Few-shot size
2: Output: Finetumed Model M'
3: Initialize: TrainingSet 6- {
4: for each sample E S do
5: (Aold: Hild))
M. M.generate Anawer WithontReViss((ammpn
6:
(Auew, Bnew)
M. wwwtte/wwwwwtttteiiiiieiiie..
7:
if M.check Answer (Aoaa) is False and
M. M.checkAAswerr ) is True then SediningSet..ddd(aaannn 9: end if
8:
Egeneraten))
10: end for
11: Randomly select & samples from l'rainingSet to form
SewShotSet for few-shot self training
12: M' A. W..ineeuneeHormeey Few:Shot.ell
13: return M'
This metric has been shown to align better with human evaluations than previous metrics.
E
Data Details
VQA-X and A-OKVQA both augments the VQAv2 dataset (Antol et al., 2015) with explanations for each answer. The images in VQA-X are sourced from the COCO dataset (Lin et al., 2014), and comprises 33K QA pairs drawn from 28K images, e-SNLI-VE provides explanations for the Visual Entailment Prediction task, which involves answer- ing whether given image and hypothesis are in tailment, contradiction, or neutral relationship. The images for this dataset are drawn from Flickr30k (Plummer al., 2015), and contains over 430K examples. VCR a dataset that presents a model with an image, a question, and list of objects that are annotated with bounding boxes, and requires the model to first select answer and then explain
it. VCR includes 290K samples of questions, swers, and rationales. For each of the dataset, use the original train set of each dataset and their own test set.


Reviews:
[{"review_id": "dhBphAA2OD", "reviewer": "Reviewer_UABU", "summary": "", "questions": "My main concern is the convergence of such recursion. First, the answer and explanation generation involves a  sampling, with which $A_n = A_{n-1}$ doesn't necessarily lead to convergence. Second, why would the model tend to converge? There is no source of \"improvement\" at each step. Providing clear proof might be challenging given the complexity of LLM, but it would be better to give a persuasive motivation for the possible step-wise improvement that leads to convergence.\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and insightful feedback on our paper! We deeply appreciate the time and effort you've dedicated to understanding our work. We'd like to address your concerns:\n\n1. **Convergence Issue**: As you rightly highlighted, a small percentage of samples do not converge. We observed this in the \"Failure Case Analysis\" section, where approximately 2% of samples enter an infinite loop (refer to the second case in Figure 5). To provide more insight into this issue, we analyzed the non-convergence rates for the three datasets: VQA-X (1.31%), e-SNLI-VE (1.84%), and VCR (1.67%). Investigating the root causes behind this behavior in a subset of data is indeed a compelling direction for future work.\n\n2. **Source of Improvement**: The iterative feedback mechanism is central to our model's step-wise improvement. By taking the explanation from one iteration and using it as input for the next, the model refines its interpretation and visual attention. Conceptually, it's analogous to a person rephrasing a statement repeatedly to enhance clarity. While this isn't a formal proof of convergence, it offers a reasonable justification for our model's iterative improvement.\n\n3. **Computational Considerations**: We recognize the computational overhead introduced by recursion. However, as evidenced in Table 6, we found that limiting the iterations to just three steps provides optimal performance. This suggests that a modest number of iterations is sufficient for our model, mitigating excessive computational costs.\n\nWe hope these explanations address your concerns. We're grateful for the opportunity to clarify and improve our paper's presentation. Your feedback has been invaluable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Recursive revising the answer and explanation is a promising direction towards explanation AI.\nThe self-training mechanism is interesting.\nThe manuscript is well organized, and the writing is easy to follow.", "reasons_to_reject": "The convergence of the algorithm is questionable.\nThe improvement of most metrics is somewhat marginal, given times of recursion as the price.", "questions_to_authors": "My main concern is the convergence of such recursion. First, the answer and explanation generation involves a  sampling, with which $A_n = A_{n-1}$ doesn't necessarily lead to convergence. Second, why would the model tend to converge? There is no source of \"improvement\" at each step. Providing clear proof might be challenging given the complexity of LLM, but it would be better to give a persuasive motivation for the possible step-wise improvement that leads to convergence.\n", "paper_topic_and_main_contributions": "In this manuscript, the authors address the visual question answering problem, aiming to provide insightful explanations together with the answers. The proposed approach, namely the Recursive Visual Explanation algorithm (ReVisE), recursively generates an answer, together with an explanation, based on the encoded image. The generated answer and explanation will be used for the next step in the recursion. The motivation is to allow the model to improve the answer and explanation step by step in the recursion until it converges. The authors also proposed a few-shot self-training mechanism to train the model with low annotated explanation resources. Experimental results are shown to support the proposed method.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z5Gq2d9i43", "reviewer": "Reviewer_yJ8S", "summary": "", "questions": "A. Are the reported metrics evaluated on the answer or the explanation?\n\nB. Have you looked into how explanation quality varies in the iteration process?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your thoughtful review and the acknowledgment of our work's strengths! We've provided further clarifications to address your concerns:\n\n1.**Evaluation Target**: The metrics we reported are evaluated on the explanations.\n\n2.**Clarifying Metrics**:  \n\nTo assist in understanding the diverse metrics, we've categorized them into two primary groups:\n\n1. Traditional metrics from \"Microsoft COCO Caption Evaluation\" and BertScore.\n2. GEval: A novel metric leveraging models like GPT3.5/GPT4 and chain-of-thought reasoning\n\nHere's a succinct breakdown:\n\n|Metric  |\tDescription |\n|:-------------:|:-------:|\n|Bleu\t|Measures the overlap of n-grams between the generated explanations and the reference explanations.|\n|Meteor\t|Harmonizes precision and recall by considering synonymy, stemming, and word order in the explanations.|\n|Rouge\t|Focuses on the overlap of n-grams, specifically the recall between generated and reference explanations.|\n|Cider\t|Considers the consensus of terms in the generated explanations relative to multiple reference explanations.|\n|Spice\t|Evaluates semantic propositions within the generated explanations and contrasts them with the references.|\n|BertScore |Measures the similarity between the generated and reference explanations using contextual embeddings from BERT.|\n|G-Eval|Evaluates the logic and reasoning of the generated explanations using GPT3.5/GPT4 and take the references as examples.|\n\nIn the realm of vision-language explanation generation, no singular metric can comprehensively evaluate the quality and correctness of the generated text. Each metric we used serves a distinct purpose and examines a different aspect of the generated explanations.\n\n3.**Iterative Explanation Quality**:\n\nWe provide a table for the e-SNLI-VE dataset, We tested the quality of the samples that stops at the first step (referenced as \"step1\"), the quality of the samples that stops at the second step (referenced as \"step2\"), and the quality of the samples that stops at the third step (referenced as \"step3\")\n\n​​The table illustrates the consistency in explanation quality across steps. This suggests that even for samples that require deeper iterations for correctness, the explanation quality remains commendably robust.\n\n| | B1         | B2     | B3 | B4 | M | R-L | CIDEr | S |\n|-------------|-------|-------|----------|-----|-----------|---------|-----|-----------|\n| step1                    | 36.7|23.7|15.8|10.8|18.2|31.1|104.0|31.4|\n| step2                    | 36.5|23.5|15.7|10.7|18.1|31.0|102.9|31.3|\n| step3                    | 37.7|25.8|17.6|12.2|18.7|31.6|110.9|34.7|\n\n4.Thank you for your detailed efforts in pointing out the areas of improvement regarding typos, style, and presentation. We really appreciate it and will certainly act on them for improvement.\n\nWe hope these explanations provide clarity on your concerns. Thank you again for the time and effort you dedicated to our paper's review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Paper is written with clarity and easy to follow.\n- Extensive experiments to show the effectiveness of ReVise against previous methods and also its use in self-training.\n- Method is data efficient, compared to existing approaches.\n- This method also allows for few-shot self-training, further addresses the issue of scarce annotations.", "reasons_to_reject": "- I had a hard time understanding the results because of the various metrics. While the different metrics show ReVise performs the best, it would be nice if the authors can give a better idea of what metrics to look at and how to interpret them. \n- Unclear evaluation: not sure whether the evaluation is conducted on answers or explanations.", "questions_to_authors": "A. Are the reported metrics evaluated on the answer or the explanation?\n\nB. Have you looked into how explanation quality varies in the iteration process?", "paper_topic_and_main_contributions": "This paper proposes ReVise, a method that recursively generate explanations for vision-language tasks by recomputing visual features. The method is data efficient and achieves on-par performance with current methods for VL-NLE. They also showed the effectiveness of ReVise in self-training by generating pseudo ground truth annotations for explanations and self-train on these generated explanations. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Too many tables / figures (especially in page 6, 7, and 8.) Some can be moved to appendix.\n- Some citations should use \\citet"}, "extra_text": {}}, {"review_id": "N890jVPGLf", "reviewer": "Reviewer_86Pj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review and insightful comments. We read your comments very carefully and here are some efforts we made that could hopefully address some of your concerns:\n\n1.**Clarification on the First Contribution and Data Efficiency**\n\nFor the first contribution of our experiments, we used **5%** of the training set and tested on the full test set while the other methods are all trained on the full training set and tested on the full test set. We provide further ablation on the amount of training data used. On the e-SNLI-VE dataset, we tried 1%, 3%, and 5% of the dataset and report the filtered score. the results are shown below:\n| | B1         | B2     | B3 | B4 | M | R-L | CIDEr | S |\n|-------------|-------|-------|----------|-----|-----------|---------|-----|-----------|\n| NLX-GPT             |      37.0 |25.3| 17.9 |12.9 |18.8 |34.2| 117.4| 33.6|\n| 1%                    | 37.0 | 25.7| 18.4 | 13.4 | 19.4 | 33.9 | 122.3 | 33.5 |\n| 3%                    | 38.0 | 26.7| 18.9 | 13.6 | 19.7 | 34.5 | 126.7 | 34.0 |\n| 5%                    | 38.3 | 26.5| 19.0 | 13.8 | 19.7 | 34.7 | 126.7 | 34.2 |\n\nRemarkably, using merely 1% of the dataset, our method achieves a performance comparable to the state-of-the-art NLX-GPT. As we scale to 3% and 5% of the data, we observe performance gains beyond the benchmark set by NLX-GPT. This illustrates that our model, leveraging recent advancements in pre-trained models, can deliver high-quality explanations even with substantially fewer annotations than traditional methods.\n\nAdditionally, we've elaborated on how both the recent vision-language model BLIPv2 and our proposed method ReVisE contributed to these gains in Table2 and Table3, showing that our improvement gain is caused by both the usage of the recent vision-language model BLIPv2 and also our proposed method ReVisE.\n\n In addressing the need for a clearer differentiation from existing methods, we've furnished a more detailed breakdown:\n| Method | Vision Backbone         | Language Model     | Training Method | \n|:-------------:|:-------:|:-------:|:----------:|\n| FME                    | ResNet-101 |Up-Down LSTM | Modular | \n| RVT                   | ResNet-101 | GPT-2| Modular | \n| e-UG                    | ResNet-101 | GPT-2| Modular | \n| OFA-X                    | ResNet-152 | OFA[1] | Unified | \n| NLX-GPT                    | ViT | Distilled GPT-2 | Unified | \n| Ours                    | ViT | FlanT5-xxl | Unified | \n\nWhere unified model generates answer and explanation together and modular models use two separate modules to first generate the answer and then generate an explanation.\n[1] The text generation architecture of OFA is similar to GPT\n\nFurthermore, we've furnished a detailed experiment setting table for different methods, specifying original settings from each respective paper:\n| Method |  Training Data  | Learning Rate |Optimizer  | Scheduler | Batch Size|\n|:-------------:|:-------:|:-------:|:----------:|:-----:|:-----------:|\n| FMT                    | 100% | 5e-4 | Adam | Step Decay |  128 | \n| RVT                   | 100%|  5e-5 | AdamW | Linear | 32|\n| e-UG                    | 100%| 2e-5 | AdamW | Linear  | 64 |\n| OFA-X                    | 100% | 5e-5 | AdamW | Polynomial |  128 |\n| NLX-GPT                    | 100% | 1e-5 |Adam | Linear |    64 |\n| Ours                    | 5% | 1e-5 | AdamW | Cosine | 2 |\n \nWe use a batch size of only 2 due to GPU constraint, we assume that a larger batch size(>=32) would be more beneficial for stable training and potentially further enhance the performance.\n\n2.**Comparison and Differentiation from Previous Works**\n\n**Task Differences: Text vs. Visual Grounding**:  we focus on the text-generation task which is substantially different from the visual-grounding task seen in earlier works. In our case, we use the step-wise recalculated visual feature and the previous text explanation to generate a new text at each step. In the previous works, the recursive method is used to ground a text to the image region. Even if recursive techniques have been used before, like in the ECCV paper or with RNNs, applying them to our kind of task is not straightforward.\n\n**Simplicity**: Our method is much simpler. We don't require any additional detection models(as in the visual grounding) and we don't need to decompose the question down into sub-queries.\n\n**Dynamic Text Generation**:  One key part of our approach is generating a new text at every step. This is different from just pulling different text features from existing text, which is what existing methods do.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a multi-step explanation generation method, and the experimental results indicate that the model's performance improves with more steps.\n- The paper proposes a few-shot self-training method, enabling training with generated pseudo-explanations.", "reasons_to_reject": "- Lack of detailed description of the first contribution. 5% of the human-annotated explanations are insufficient to understand the significance of this aspect. The paper should provide more comprehensive information regarding the comparison methods, experimental settings, and detailed analyses to establish a clear contribution.\n- It seems that the novelty of the proposed recursive method is relatively incremental, as the iterative computation of visual features conditioned on text input is a common technique in the multimodal domain, as seen in previous work (e.g., Paper [1]). The paper should address how the proposed method extends beyond existing techniques to justify its contribution.\n- The paper should include a thorough analysis of the significant performance gap observed between the proposed method and state-of-the-art methods, which are essential for a comprehensive evaluation of the proposed method's effectiveness.\n\nReferences:  \n[1] Improving one-stage visual grounding by recursive sub-query construction. ECCV, 2020.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a recursive visual explanation method for step-by-step vision-language explanation generation. The proposed method appears to build upon the powerful BLIP-2 model. Experiments on three datasets demonstrate the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a recursive visual explanation method for step-by-step vision-language explanation generation. The authors have done a nice job of rebuttal. In general, all the reviewers are positive (or neural) about the paper, commenting that (1) the paper is well written, (2) experiments are sufficient, and (3) the few-shot self-training mechanism is interesting. One reviewer questioned about the novelty of the method, which the AC also agrees. Overall, the AC thinks that the merits of the paper outweigh the flaws, and is positive about the paper.
Decision: Accept-Main