Unified Reinforcement and Imitation Learning for Vision-Language Models
Authors: Byung-Kwan Lee
Ryo Hachiuma
Yong Man Ro
Yu-Chiang Frank Wang
Yueh-Hua Wu
Venue: NeurIPS 2025 poster ()
Keywords: Vision-Language Models
Primary area: applications

Abstract: Vision-Language Models (VLMs) have achieved remarkable progress, yet their large scale often renders them impractical for resource-constrained environments. This paper introduces Unified Reinforcement and Imitation Learning (RIL), a novel and efficient training algorithm designed to create powerful, lightweight VLMs. RIL distinctively combines the strengths of reinforcement learning with adversarial imitation learning. This enables smaller student VLMs not only to mimic the sophisticated text generation of large teacher models but also to systematically improve their generative capabilities through reinforcement signals. Key to our imitation framework is a LLM-based discriminator that adeptly distinguishes between student and teacher outputs, complemented by guidance from multiple large teacher VLMs to ensure diverse learning. This unified learning strategy, leveraging both reinforcement and imitation, empowers student models to achieve significant performance gains, making them competitive with leading closed-source VLMs. Extensive experiments on diverse vision-language benchmarks demonstrate that RIL significantly narrows the performance gap with state-of-the-art open- and closed-source VLMs and, in several instances, surpasses them.

Unified Reinforcement and Imitation Learning for Vision-Languagg Models
Byung-Kwan Leef NVIDIA, KAIST byungkwanl@nve com
Ryo Hachiuma NVIDIA rhachiuma@nvidia.cm com
Yong Man KAIST
ymro@kaist.c ac
Yu-Chiang Frank Wang
Yueh-Hua Wu
NVIDIA, National Taiwan University
NVIDIA
frankwanggnvidia.com
kriswenvidia. com
Abstract
Vision-Language Models (VLMs) have achieved remarkable progress, yet their large scale often renders them impractical for resource-constrained environments. This paper introduces Unified Reinforcement and Imitation Learning (RIL), novel and efficient training algorithm designed to create powerful, lightweight VLMs. RIL distinctively combines the strengths of reinforcement learning with adversarial imitation learning. This enables smaller student VLMs not only to mimic the sophisticated text generation of large teacher models but also to systematically improve their generative capabilities through reinforcement signals. Key to our imitation framework is L..M-based discriminator that adeptly distinguishes between student and teacher outputs, complemented by guidance from multiple large teacher VLMMs ensure diverse learning. This unified learning strategy, leveraging both reinforcement and imitation, empowers student models to achieve significant performance gains, making them competitive with leading closed-source VLMs. Extensive experiments on diverse vision- lision-language benchmarks demonstrate that RIL significantly narrows the performance gap with state-of-the-an open- and closed-source VLMs and, in several instances, surpasses them.
Performance Improvements (%) across Evaluation Benchmarks
(GRPO)
20
Rl, (DNGRRO)
Stagle Laqge
Chan
15
RIL-Mutt @Durs
Curs
10
BLINK Figure Showing the performance improvements (%) of WW..2.55VVLL-7B. 1 across vision-language evaluation benchmarks for AI2D [2] Math Vista 3
A12D
MathVista MM-Vet
MMMU
14 14-Avg Bench
MMMU [5), BLINK 6 and Note that, we conduct RL on GRPO [7] and advanced GRPO, Dr.GRPO [8], with only answer rewards from LLM-as-a-Judge 9 (see Algorithm [}, and we present RIL based on similarity rewards from single or multi large teacher VLMs and simultaneously answer rewards (see Algorithm 2).
MM-Ver Vet 4
the average scores for evaluation benchmarks used in Table I
Work Done during Internship.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

7B
BB InternVL 3 (Ours)
7018
75
3B
Wwn2.5-VL-BR VL-RRIL (Ours)
3
72B
+78
GFI-60
Average: Pefoorruutt
70
TnterVi15
Clande355
2B
000
TaternVT
G Gemini-15
65
58
3.68
&-ulttmodal
1B
BB
Qum2VV
60
Owen2-VL
Owen2
26
Owen2. 5-VL
Pri-s-Multimdal
18
InterVLL2
Phi 3. Vision
18
InterWVL2.5
GPT-4g (24.05.13)
50
LnternVLL-RIL (Ours)
InternLL3
Gamini-1.S-B
Cleude-J 3. 5-5onnet
20
40
60
Parameters (B)
Figure 2: Comparing RII-appliid VLMs based on multi large VLMS with diverse open- and closed-
source VLMs, under average performance of numerous vision-language evaluation benchmarks:
A12D p ChartQA [991, Vista [3), MMB [20]. MM-Vet 4. MMMU (5), MMMU-Pro [21].
MMStar (22]. BLINK [6). SEED 23 SEED2+
24, and RealWorldQA (RWQA).
1
Introduction
The pursuit of artificial general intelligence (AGI) has gained momentum, with much of the initial progress driven by instruction-cuned large language models (LLMs) (10-15). Vision-Language Models (VLMs) [16] 18 represent a significant step forward, extending the LLM framework to integrate visual data and thereby achieve multimodal understanding. Their ability to generate textual responses that increasingly emulate human- like comprehension has made VLMs focal point of considerable attention. Recognizing their transformative potential, numerous organizations are heavily investing to push the boundaries of what VLMs can achieve.
insightful,
Historically, the enhancements in vision- performance have primarily relied traditional strategies such as scaling up model sizes [13] 25, 26 and expanding visual instruction tuning datasets (27-30 [8]. More recent advancements have explored architectural modifications for 32] or integrating auxiliary vision [33
instance, directly altering model structures (31
27
34
and reasoning modules 355 Alongside these efforts, the 'think-answer' paradigm 36
46
has
gained prominence following DeepSeek-RII ( application of reinforcement learning (RL) via GRPO 433. However, such architectural changes and the verbose 'think' responses preceding answers can significantly increase inference latency and computational memory requirements. These challenges render many powerful VLMs impractical for resource-limited settings like smartphones and augmented reality (AR) devices. As result, there is a growing imperative within the research community to develop VLM designs that achieve a compelling balance between strong vision- language capabilities, low inference latency, and lightweight model footprint.
To overcome these challenges, we introduce an approach for developing high-performing, efficient VLMs that avoids architectural modifications and the need for lengthy 'think" responses. Our method, termed Unified Reinforcement and Imitation Learning (RIL). an efficient training algorithm integrating principles from both GRPO [43 and the GAIL framework [44]. The central objective of RIL to enable smaller VLMs (c.g.. 7B models) to effectively mimic the text generation style of significantly larger VLMs (e.g., 72B models). Prior works 45 46) have argued that natural language response-basee based distillation more effective than high digh-diimnsional feature distillation, emphasizing the importance of utilizing the language head referree to as the verbalization effect. Motivated by this insight, we began by leveraging the similarity natural language responses between teacher and student models. Drawing inspiration from how models articulate responses differently, RIL employs an LLM-bbsed discriminator trained to distinguish between the text responses generated by the student (small VLMs) and the teacher (large VL.Ms). This discriminator's 's output provides similarity reward signal, guiding the student VLMs, which act as a generator, to produce responses increasingly akin to those of the teacher. To maintain training stability and prevent the generator or discriminator from overpowering the other, a common issue known as the balance problem [47 48) the discriminator's 's architecture initially mirrors that of the generator.
However, the practical implementation of RIL for VLMs entails several specific challenges. Firstly, relying on continuous discriminator scores (ranging from zero to one) to assess similarity to large VLM outputs can introduce ambiguity into the learning signal. To ensure clearer and more decisive
reward, and drawing inspiration from prior works that binarize answer rewards 7 43 we convert the discriminator's similarity score into a binary value. Secondly, the discriminator, by design, focuses stylistic similarity and does not inherently verify factual correctness against ground truth

---

You are discriminator assistant You are provided with R Response in <response> </response> tag. tauk is to determine whether the Response was gemerated by teacher or student model.
SYSTEM
Retur 0 if the Response was generated by the teacher model.
Return if the Response was generated by the student model.
USER <response> <response-freppxveeeee frespanree-
Determine the source of the Response.
Provide your answer we as one of the following: (Teacher model) (Student model)
Figure 3: Input prompt for Discriminator
answers, which could otherwise lead performance degradation. We mitigate this by incorporating an LLLM-as-a-Judge [9] to provide a separate binary reward signal based on the accuracy of the generated response. A third challenge arises during the GRPO [43] phase: if the student VLMs, due to their more limited knowledge, fail to generate any correct responses for a given task, they struggle to identify effective learning pathways. To provide clearer guidance, especially when the student VLM lacks sufficient lomain-speciffe knowledge compared to larger models, we incorporate text responses from both student and teacher VLMs during this GRPO training step. This strategy not only stabilizes RIL training but also creates opportunities for the student VLM to potentially surpass the performance of larger teachers.
Several key aspects of RII. contribute to its effectiveness. Notably, we observe that sourcing text responses from multiple large teacher VLMs significantly boosts student performance beyond what a single teacher can achieve, owing to the richer diversity of responses and the consequently more robust discriminator. This enriched learning environment further enhanced during GRPO [43] step: by incorporating text responses from both student and teacher VLMs, we provide the student with clearer exemplars for reaching correct answers. RIL also demonstrates particular synergy with distillation based VLMs; these models show marked performance improvements because their inherent feature alignment through distillation complements RIL's objectives. Beyond these learning dynamics, crucial practical advantage of RIL is its inference efficiency: trained models do not require an explicit 'think' phase before generating answers, thereby maintaining the same inference speed as the original student VLM.
Furthermore, RIL's reliance on LLM-as-a-Judge g for accuracy evaluation offers substantial ad- vantages in applicability. Unlike methods such as DeepSeek-RI [7] and its variants [39 49 41 (3], which often use conventional answer parsing for reward computation and are thus limited domains with predefined metrics (e.g., math or image grounding), RIL transcends these limitations. For example, conventional parsing struggles with open- general visual questions, such as "How to cook this dish?" based on a food image, or "Summarize this movie" from its poster. In addition, the parsing poses a challenge to flexibly compare different types of answers, if a response "The answer is twenty percent" but its ground truth is "20". The versatile evaluation power of I.LM-as-a-Judge [9] enables RIL's successful application to a broad spectrum of visual question answering tasks, fostering strong performance in areas like recognition, OCR, common-sense reasoning, mathematical problem- solving, and chart/document understanding. Finally, because RIL operates purely on generated text responses, it is agnostic to the specific image embedding strategies or language tokenizers used by the student or teacher VLMs, ensuring broad compatibility.
To validate our approach, we conduct extensive experiments across diverse vision- laiiuage bench- marks. The results compellingly demonstrate that RIL not only significantly narrows the performance gap to state-of-the-an the-art open- and closed-soure VLMs but, in several instances, surpasses them. The primary contributions of this work are:
Unified Reinforcement and Imitation Learning (RIL): We introduce RIL, an efficient and novel training framework for VLMs. It empowers smaller models to emulate the text generation patterns of significantly larger VLMs, leading to substantial enhancements in their overall performance.
Broad Applicability and Flexibility: RIL exhibits wide applicability, functioning effec- tively with diverse VLMs irrespective of their underlying image embedding strategies or language tokenizers. Furthermore, it preserves the original inference speed by avoiding lengthy intermediate reasoning steps and capably addresses general visual question answer- ing tasks through its integration with LLM-as-a-Judge
3

---

SYSTEM You are an evaluation assistant.
Q Question in <question> <kquestion> tag.
Groud Truth is <ground truth> </ground truth> tag.
Generated Text in <generated text> </genorated text> lag.
A After reading
Question, compare Gemerated Text against Ground Truth summary: If the Generated Text fully and correctly captures core point
If it is incorrect irrelevant
Output the evaluation score (0 after giving brief explanation
The evaluation score should be wrapped in <answer> <Janswer> logs
USER
<question> (prestion(qquetioonn <qquestion>
<ground truth (ground truth) truth>
<yeneraled lext> (ggeerrree text>
Provide the evaluation score after giving brief explanation.
The evaluation score should be wrapped in <answer> </answer> tag.
Figure 4: Input prompt for LLMdas-a-Judge 9
Extensive Validation and Significant Performance Gains: Our comprehensive experi- mental results consistently show that RIL delivers substantial performance improvements across variety of vision-language benchmarks. These gains position RIL-trained models as highly competitive with, and sometimes superior to, existing state-of-tte-art VL.ss.
Related Works
Evolution of Vision- Lingua- Models (VLMs). Conventional remedies for advancing VLMs have largely centered scaling up model sizes and expanding datasets to push performance boundaries.
Consequently, promising VLMs have emerged, for example, closed-source VLMs: GPT-4o [50],
Gemini [51]. and Claude-3.5 Sonnet [52]. and open-source VLMs: Molmo-72B
[53] VA-
One Vision-722 72B (28), NVLM-72B [54), Swen2.5-VL-722 [T], and Intern VL3-78B
[8] have fol-
lowed this paradigm, incorporating large- language models such Qwen22/wwen255-725 (13)
However, the sheer size and computational demands of these models present significant barriers to deployment in cesuurce-onnstrained environments such mobile and embedded devices. To address these challenges, CoLLaVO (33) and MoAI (34) utilize computer vision models directly for visual capability, and Mini-Gemini 277,, MoVA
[55) and Eagle (56) employ multiple vision encoders such
as CLIP [57]. ConvNeXt (58), DINO-v2
559 and SAM (60) In parallel, Meteor [35] explores the
efficient way of learning complex reasoning abilities, and TroL. 31 and Phantom 332 investigate
propagation modification for how we can embed vision lsigun-a knowledge as much as possible despite using the same architectures. More recently, since DeepSeek-RI (7) introduces think-answer process and shows its dramatic performance improvements by using reinforeement learning (RL) such as GRPO 43 many variant models for VLMss has presented LMM-RI 61] Vision-RI (37]. R1-V 42), V VThhinke [62), R1-One
63, R1-Zero (64, and MM-Eureka (65) They
employ think think- answer process to VLMs in specific areas such as math problems and object counting
tasks. In addition, NoisyRollout [40] injects noise into clean images for distortion and train VLMs to
strongly improve visual robustness, thereby understanding visual properties more than before.
Imitation Learning (IL). It originates from the concept of learning expert behavior in robotics. Generative adversarial imitation learning (GAIL) 44] is foundational framework in this domain. employs a generator and a discriminator where the generator (e.g., small student VLMs) attempts to replicate expert behavior (e.g., large teacher VLMs) by producing outputs such as actions trajec- tories (e.g., text responses) that are indistinguishable from those generated by experts. Meanwhile, the discriminator aims to distinguish between the generator and the expert. By framing imitation learning (IL) as a minimax game, GAIL [44] leverages ad adversarial training to align the behavior of the generator with that of the expert without explicitly defining a reward function. Instead, GAIL [4] only relies on discriminator scores to evaluate how the generator similarly produces outputs of an expert. This approach has demonstrated strong performance by directly learning from expert data, making it a widely adopted strategy in IL [66]. Based on its effectiveness, we integrate reinforcement and imitation learning (RIL) for VLMs with four key modifications: (1) combining GRPO [3] and GAIL 44 framework with explicitly reward design, (2) making the output scores of the discriminator binary to stabilize training, (3) incorporating LLM-as-a-Judge (9] to evaluate answer rewards that assess whether the generated text responses from
complex, high-dimension
tasks

---

Table 1: Comparing the performances by using answer rewards from LLM-as-a-Judge 9 with purely
RL on GRPO 77 and advanced GRPO [8), and by using similarity and answer rewards from RIL
on GAIL 441 under numerous evaluation benchmarks: AI2D [2), ChartQA [9]. Math [3]
MMB/MMB CN
20 MM- Vet 4 MM- Vet-v2 (67]. MMMU [3]. MMMU-Pro 21,, MMStar [2
BLINK (6, SEED 23 SEED2+ 24,, and RealWorIdQA (RWQA). Note that, for RIL, we set large
VLMs as the largest version of same small VLMs, such as Owen2.5-VL-77 < -wen2.5-VLL-22B
and Intern VL3-8B <- Intern VL3-78B.
VLMs
AI2D
ChamQJA Mara Vista MMB
MMB
XM-Ver MM-Verr22 MMMU
MMMU-Pro MMStar BLINK
SEED
SEED2+ KWQA
Owen22.5-VL.. 5-VL-7
83.9
87.3
67.8
83.5
83.4
71.8
63.7
55.0
38.3
63.9
56.4
77.0
70.4
68.5
(GRPO)
84.1
88.8
83.8
83.9
72.8
63.9
56.2
40.3
65.2
58.9
77.4
70.6
69.4
(D:GRO)
84.5
90.3
84.4
73.5
64.2
57.2
41.8
66.3
60.77
78.0
70.3
RIL GDPO GAIL)
95.4
87.2
77.3
66.1
61.8
48.2
71.1
68.5
30.7
81.6
84.0
61.2
79.1
78.1
61.3
$7.6
51.2
31.6
55.9
47.6
74.0
67.6
65.4
w. RL (GRPP)
K1.9
87.8
61.8
80.3
79.7
62.55
55.4
$2.0
33.7
$7.7
$2.6
75.7
68.6
66.4
RI. (D=GRPP)
82.4
89.7
62.5
81.8
81.2
63.88
55.9
$2.5
34.3
59.1
53.7
76.5
69.2
67.0
GRPO GAIL.
95.4
84.2
53.7
36.8
61.2
55.5
70.3
Intem
82.2
62.7
11.3
68.2
55.5
77.1
69.7
w. (GRPO)
85.9
89.6
72.3
84.6
84.2
78.7
66.4
63.8
42.5
70.0
57.1
77.9
70.6
71.3
w. RI. (L.(D))RP))
86.3
91.2
72.9
85.2
85.3
79.9
66.5
64.9
42.8
70.7
57.5
7X 1
71.1
72.0
RII. (Di GRPO GAIL)
87.4
95.5
74,1
88.7
89.3
78.4
66.5
66.8
44,8
75.4
60.1
80.6
73.0
73.3
Intem VL3-2B
78.7
60.2
57.0
78.4
53.9
48.6
24.9
60.7
47.0
75.0
64.6
64.3
(GRPO)
86.9
80.0
25.7
47.3
75.8
61.9
w. RL.(D:GRPO)
74,8
90.3
38.5
83.2
81.3
62.1
53.8
50.5
26.4
62.4
47.9
76.5
65.
65.2
w. RJL (DEG GAIL)
81.1
93.6
63.0
85.6
133.2
61.5
$4.2
$2.9
27.5
63.2
48.3
78.2
66.7
66.8
Intem VL.3-11
69.4
75.3
45.8
72.5
67.9
58.7
47.5
43.4
17.5
51.5
42.9
71.2
58.2
58.2
RL (GRPP)
69.6
83.0
16.7
73.8
69.4
57.7
43.0
42.8
17.2
52.5
42.7
71.5
58.5
58.5
(DDGRPO)
69.9
86.7
47.6
74.9
70.5
47.0
42.8
17.3
$3.4
42.9
71.9
59.4
59.4
RIL (UEG GAIL)
72.0
93.6
75.7
56.9
47.4
43.1
18.3
57.4
44.1
74.2
61.4
student VLMs are correct, and (4) utilizing the generated text responses from teacher VLMs when updates student VLMs, serving as a direct guidance to reach the correct text responses. These approach not only stabilizes RIL training but also provides mechanism for student VLMs to potentially exceed the performance of teacher VLMs. Moreover, RIL do not require an explicit think-answer process, thereby maintaining the same inference, and it is independent of any particular image embedding methods or language tokenizers used in the student or the teacher VLMs.
GRPO 43
3 Unified Reinforcement and Imitation Learning for VLMs
3.1 Core Model Components for RIL
Our RIL framework utilizes several key model components. For the student models-which are the VLMs we aim to enhance-we we employ recently released architectures such as Qw (3B and 7B variants) [I] and Intern (IB, 2B, and 8B variants) [18). Starting from these pre-trained checkpoints, RIL directly updates their parameters with the goal of developing efficient, high- performing VLMs capable of rivaling leading closed-source systems like GPT-4o (50), Gemini [SI], and C35 Sonnet [52]. For the teacher models, which provide the target behavior, we select powerful large teacher VLMs recognized for their strong performance on benchmarks like VLM Leaderboard (6), specifically Qwen2.5-VL-772
[ and Intern VL3 [8. The student VLMs are trained to mimic the text generation patterns and response styles of these teacher models. Central to the imitation learning aspect is a discriminator, designed to assess the similarity between student- generated responses and those from the teacher VI.Ms. To promote training stability and mitigate the balance problem common in adversarial setups [47 48 -where one component might overpower the other- the discriminator's architecture and initial parameters deliberately mirror those of the student VLMs. Lastly, to evaluate the factual correctness of the generated text, we utilize an LLM- as-a-Judge [9], specifically Qwen2.5-32B 13 following the methodology of Zheng et al. [ This model is not trained but is used solely to determine whether the predicted response accurately reflects the meaning of the ground truth.
The subsequent sections will first elaborate on the discriminator's architecture and pre-training (Sec- tion 3.2), followed by a detailed exposition of the RIL algorithm and its reward design (Section 3.3
3.2 Discriminator Architecture and Pre-training
With the student (small) VL.MS serving as the generator in our framework, the effective pre-training of the discriminator is crucial. This initial training phase equips the discriminator (Da) with the ability to reliably differentiate between text responses originating from student VLMs versus teacher
5

---

Table 2: Showing the effectiveness of employing multiple large VLMs more than a single VLM,
under the numerous evaluation benchmarks equally used in Table[[
VLMs
AIDD ChanQA MathVisia MMB MMB
MM- Vet MM--Vett Vet-VVe-22 MMMU MMMU--BS Pro MMStur BLINK
SEED SEED2+
RWQA
5-V1-78
87.3
67.8
83.4
63.7
55.0
38.3
63.9
56.4
77.0
70.4
68.5
RII. (Qwen2.5 VL-728)
95.4
87.2
77.3
66.1
61.8
48.2
68.5
80.7
74.2
w. RII. (InteraVL33-7BB)
$6.8
95.5
74.5
56.7
87.1
75.8
66.0
60.9
47.1
71.1
68.1
80.8
72.7
75.4
W. RIL (Both)
$6.1
95.6
79.7
56.3
86.5
80.4
71.1
65.7
48.5
71.1
70.0
80.5
72.8
72.8
Qwen2. 5-VL-18
81.6
84.0
61.2
70.1
78.
61.8
57.6
51.2
31,6
55.9
47.6
74.0
67.6
65.4
(Qwen2 (Qwen2.S.VI-7225
95.4
84.55
84.2
57.6
53.7
36.8
61.2
55.5
78.7
67.7
w. RIL (1nteraVL33-88))
95.4
84.7
56.3
37.7
61.1
55.3
78.8
68.2
W. RJL (Both)
83.9
95.7
71.7
85.2
84.5
74.7
62.8
60.7
43.7
65.2
60.8
78.7
71.4
73.2
IntemVL. 11.3-88
VL3-8B
65.2
86.6
71.6
83.4
82.2
7X.5
66.3
62.7
41.3
68.2
55.5
77.1
69.7
70.8
RII. (Qwen2 72.82)
87.5
95.2
74.3
88.8
89.8
77.6
65.9
67.3
45.7
75.2
59.1
80.4
72.9
72.5
RIL (1n1eraVL33-888)
K7.4
95.5
74.1
89.3
78.4
66.5
44.8
75.4
60.1
80.6
73.5
74.8
InterVL.. VL3-2B
78.7
80.2
$7.0
$1.1
78.4
62.2
53.9
48.6
24.9
61.7
47.0
75.0
61.6
64.3
w. RII. (Qwenz 72)
81.1
93.6
61.9
85.3
83.0
62.6
$2.8
$2.6
26.6
63.6
4%.8
78.00
67.1
66.3
w. RIL (IttrrVL33-5BB)
$1.1
93.6
63.0
85.6
83.2
61..5
54.2
52.9
27.5
63.2
48.3
78..
66.7
66.8
RIL (Both)
80.7
94.0
67.4
$4.7
82.2
70.7
58.4
$6.6
32,3
63.9
51.5
77.8
66.8
69.3
75.3
67.9
4344
17.5
51.5
71.2
56.2
w. RII (Qwen2. VL--772B))))
72.2
93.6
50.9
79.9
75.8
56.1
47.3
43.6
18.0
5K.0
43.6
74.6
60.7
61.0
w. RIL (InteraVL33-88B)
72.0
93.6
51.6
79.6
75.7
56.9
47.4
43.1
18.3
57.4
44.1
74.2
61.4
61.4
RIL (Both)
73.0
94.1
55.5
79.1
75.9
62.9
50.7
49.7
19.0
60.5
46.9
74.7
61.9
63.7
(large) VLMs. Without adequate pre-training, D& would yield unreliable scores, undermining its utility and potentially degrading overall model performance. To prepare for pre-training, we first collect a dataset of responses. For each given question q, N text responses are generated from both the student VLMs, denoted as (o > collection, the discriminator De, parameterized by Φ. trained to maximize the following objective:
and the teacher VLMs, denoted as {o J, Using this
max L())
1 N
Σ
log Do(1, of log(1 Dolg. }}
According to this objective, D& learns output score approaching 'one value' for responses from student VLMs and 'zero value" for those from teacher VLMs. For practical implementation, given that the discriminator's backbone is language-based (mirroring the student VLMs), the generated responses o are formatted using a specific prompt (detailed in Fig. 3 to ensure proper textual understanding. To elicit the scalar discrimination score, the standard Tanguage head of the VLM (typically mapping to vocabulary logits, (Ie is replaced with a linear discriminator head (Rdx1). The input this head is the representation of the final sequence token from the last layer of the backbone model, from which the score is directly computed with the sigmoid function.
3.3 Mimicking Large Teacher VLMs with Similarity and Answer Rewards
The RIL. process commences after an initial supervised finetuning (SFT) phase for the student (small) VLMs Ty using a comprehensive visual instruction tuning dataset (see Appendix C}. This SFT stage acts as a crucial warm-up, acclimating To to the target data distribution. Once the student VLMs and the pre-trained discriminator Do (from Section 3.2) are prepared, the RIL loop begins. For each question g in a training batch, we generate G text responses from the current student VLMs, ( Ti=1, and retrieve G responses corresponding the question q from the teacher (large) VLMM, (1G
(o
Ji=1- We note that pre-generating and caching the responses from teacher VLMs (e.g., during the discriminator's 's pre-training) can significantly reduce RIL training time. In the end, this yields a combined set of 2G responses, for |2 for each question. Within each RIL iteration, the discriminator D& using Eq. (T) on these 2G responses. This step ensures that De maintains its ability to distinguish between student and teacher outputs the student VLMs evolve. Next, the student VLMs no are updated using an objective derived from Dr.GRPO (8) an advanced variant of GRPO that provides unbiased advantage estimates (A.) The objective function is:
first update
2G
max L())= 8
Σ min (r((0) A., clip (r:(0), - e, 1 A.
-DKK. (No meee)
2G
(2)
where r0( = To(o; g)
A. = R(g 0:) 2G
1
Σ R(g. 0j).
2G
nows (o: ())
1=1
Here, (0) is the probability ratio between the current policy my and the policy before the update Tolds Jeet is typically the initial SFT model, is clipping hyperparameter, and B controls the KL-

---

Table 3: Comparing RIL-applied VLMs with standard or smaller model size open-source VLMs.
VEMS
AT2D ChariQ Vist
MMR
MMB
MM-Vss
MMMU MMMii-PPo MMSta BLINKK
SEED
SFRD2+ RWQA
LLVA--OnVVviin-T
28
81
80.
63.2
80.8
57.5
4%.8
24.1
61.9
53.0
76.7
65.4
69.9
2.
58.3
81.7
81.2
54.2
29.3
61.1
50.9
64.2
MiniCPM-V2555
78.4
54.3
74,2
51.8
MiniCPM-V26-88
81.1
60.6
60.0
49.8
27.2
57.5
55 2
74.00
65.7
65.0
MinICPM-a2.648
70
M6.1
M6.9
73.3
67.2
50.9
63.1
$3.9
75.5
67.9
68.0
Ova2.88 [1]
86.6
71.8
66.1
57.4
64.6
54,3
72.2
70.4
72.5
Ovas2-16B 21
86.3
73.7
68.4
60.7
67.2
59.0
72.1
74.1
Qwen2-VL-7B
77.5
62.0
60.7
843
84.8
82.6
62.8
56.0
34.8
76.8
69.7
70.1
Qwen2.5.VL.22
81.9
87.3
67.8
$3.55
R3.4
71.8
55,0
38.3
63.9
56.4
72.0
70.4
68.5
InteraVL33-88 [1]
85.2
$6.5
71.5
83.4
82.2
78.5
62.7
41.3
68.2
55.5
77.1
69.7
70.8
Qwea255VLLRRII-22 RBB-B (Both)
86.1
95.6
79.7
86.3
86.5
80.4
65.7
48.5
71.1
70.0
80.5
72.8
72.8
InteraVL33RRB R---RRLL-88 (Both)
95.3
77.8
Phi.3.5-Visiee-48 (72
77.8
81.8
43.9
47.5
693.
Phh-4-Malinnddal-3.66 2+
83.0
81.4
65.8
$6.7
51.9
56.0
38.5
58.9
42.
73.2
68.5
64.1
Ovis2-4B 70
85.7
69.6
65.5
49.0
61..9
53.0
76.2
69.3
71.1
InteraVL2-48 [9]
78.9
81.5
58.6
78.6
73.9
51.0
34.3
32.7
53.9
46.
73.2
63.9
60.7
InternV12..-4BBB [9]
814
84.0
79.3
60.6
52.3
58.7
50.8
66.
Qwen2.5-VL-3B
81.6
84.0
61.2
79.1
78.1
61.8
51.2
31.6
55.9
47.6
74.0
67.6
65.4
Wwen25--L-RRIL-33 (Both)
83.9
95.7
71.7
85.2
84.5
74.7
60.7
43.7
65.2
60.8
78.7
71.4
73.2
InteraVL2-288 (69)
74.1
76.2
46.3
73.2
20.9
39.5
34.3
18.2
49.8
43.6
70.9
60.0
57.3
Qwen2 VL-2B 10
60.2
73.5
43.0
74.9
73.5
49.5
41,1
21.2
47.5
45.2
72.4
61.2
62.6
Aquila
75.0
76.5
59.0
Ovis2-2B 21
82.7
64.1
58.3
45.6
56.7
47.9
74.4
67.4
66.0
mmern/112.3-20
17
94.9
79.2
$1.3
74.7
71,9
611.8
43.6
23.7
$4.3
44.0
73.2
60:00
60.1
InteraVL33 28
[1]
78.7
80.2
57.0
$1.1
78.4
62.2
48.6
24.9
60.7
47.0
75.0
64.6
64.3
InteraViL3 RIL (Both)
$0.7
94,0
67.4
$4.7
82.2
70.7
56.6
32.3
63,9
$1.5
77.8
66.8
69.3
CoVVV--OeVisiinO.... 28
57.1
61.4
32.2
31.4
Ovic2-1B [7]
76.4
59.4
50.0
36.1
52.1
44.0
71.7
61.4
63.9
InternVL2-18 69)
64.1
72.9
57.7
65.4
60.7
32.7
36.7
14.8
45.6
38.6
65.2
54.3
50.3
AMernN1122.5.1p [7]
69.3
75.9
43.2
70.7
66.3
48.8
40.9
1888
51.3
42.0
70.4
59.0
57.5
InteraV13318
69.1
75.3
45.8
72.6
67.9
58.7
43.4
17.5
51.5
42.9
71.2
58.2
58.2
InteraVL3 RIL (Beth)
73.0
94.1
55.5
75.9
49.7
60.5
divergence penalty. The total reward R(g, 0z) for each response or is a crucial composite signal, comprising two binary components:
1. Similarity Reward:
This indicates if the response o, resembles those from teacher VLMs, denoted by 1 (g o.) < 0.5). A discriminator score below 0.5 suggests higher similarity to teacher responses, consistent with Do being trained to output 'zero value" for teacher samples [I..
2. Answer Reward: It assesses the factual correctness of o against ground truth answer determined by LLM-as-a- aas-a- a, o;). This evaluation uses the prompt detailed in Fig. 4
These two rewards are summed form the overall reward R(q, o:). The inclusion of the answer reward is essential, as the discriminator alone primarily captures stylistic similarity and does not guarantee factual accuracy, the absence of which could lead significant performance issues. This dual-reward strategy ensures that mimicking large teacher VLMs involves not just replicating surface- level text patterns but also encompasses the deeper 'verbalization effect' [45]. which characterizes how proficient VLMs articulate correct and contextually appropriate answers. The comprehensive RIL procedure is detailed in Algorithm 2
4 Experiments
4.1 Dissecting RII.
To understand the distinet contributions of its components, we first analyze RIL by comparing against reinforcement learning (RI. baselines that utilize only answer rewards. Specifically, we
evaluate RL agents trained solely with GRPO [43] or its advanced variant, Dr. GRPO (8) without the discriminator or the associated similarity rewards. As shown in Table I RIL, which integrates reinforcement and imitation learning elements based on GRPO and GAIL [44), demonstrably outperforms these RL-only approaches, confirming the efficacy of our unified training algorithm.
key finding is the substantial benefit of employing multiple large teacher VLMs. As detailed in Table 2 using responses from a combination of large models (specifically Awen2.5-VL-722 (1 and Intern VL.3788 (18) yields significantly better vision-language performance than relying on any single teacher. This advantage arises from the richer diversity of textual responses, which strengthens the discriminator's training. Furthermore, incorporating this diverse set of high-quality responses from teacher VLMs directly into the GRPO optimization step provides student VLMs with clearer and
7

---

Table 4: Comparing RII-applied VLMs with large size open-source and closed-source VLMs.
VLMS
A1213
ChariQA Vist MMH MM-Vet MM-Vet-v2 MMMUI MMMU-PP MMSar
INK
SPHD SRED22 RWQA
NVLM-73B [54]
85.2
86.0
66.6
58.9
59.7
63.7
48.0
75.5
68.4
69.9
LLaVA-OV-72B
31,0
77.5
Molme 72B (33
$3.4
87.3
58.6
$1.1
54.1
63.3
49.7
74.5
67.6
73.7
VV-22-V [6]
$8.1
58.3
70.5
$6.5
74.0
68.7
64..5
46.2
68.3
60.5
77.9
72.3
71.8
Quen2.5-VI-7228
88.7
89.5
74.2
58.6
76.9
76.2
68.2
46.2
70.8
64.4
79.5
73.0
75.7
1xcmV1.2-768 [0]
57.6
88.4
65.5
86.3
65.7
685
62.7
40.0
67.4
56.8
77.6
69:7
IncemV1...-7885
89.1
88.1
72.3
N8.3
72.3
65.5
70.1
48.6
69.3
63.8
77.0
71.3
78.7
1sccmVL3-788 18
89.7
89.7
79.0
89.0
81.3
70.0
72.2
47.3
72.5
66.3
78.7
71.9
78.0
Stmmet3.5-Somstt
70.1
Claude-37-Someet [9
$2.5
66.8
70.0
71.0
65.1
56.6
74,3
67.6
55.4
Gemini- Pro [1
79.1
87.2
63.9
73.9
64.0
66.9
62.2
46.9
59.1
59.1
76.0
70.8
67.5
(emini-20--Hash ST
70.4
73.6
54.4
69.4
GPT-4o (24.08.
85.7
63.8
69.1
$1.9
77.1
72.0
GPT-4.1 (25.04,14)
85.9
70.4
78.8
74.9
69.88
68.5
78.0
73.1
78.7
Qwea2.5.VL RIL-RI-BB (Both)
86.1
95.6
79.7
$6.3
80.4
71.1
65.7
48.5
71.1
70.0
80.5
72.8
72.8
IncemVV1.3- (Both)
87.3
95.3
77.8
88.1
80.1
47.8
M0.5
73.8
Similarity Reward
Answer Reward
Average P Performanc (%)
0.75
16.5
75.0
0.60
0.53
73.5
0.45
0.50
12.0
0.30
70.5
0.40
09.0
0.15
0
200
400
600
800
1000
200
400
500
500
1000
200
400
600
BC0
1200
Ieration
lteration
Ileration
Figure 5: Illustrating training dynamics of small VLMs during RIL, where (Left) showing the evolution of similarity rewards over training iterations and (Mid) accuracy rewards obtained using LLM-as-a-Judge a-duge (9) ensuring that generated responses are both contextually appropriate and factually correct. (Right) displaying the overall average performance of evaluation benchmarks used T
more varied exemplars of correct answer generation. To be simplified in the manner of an algorithm, given a response 'TI' from one teacher VLM and 'T2' from another teacher VLM, and a response 'S' from the student VLM, then the discriminator is trained to output zero when 'S' is provided and one when either 'T1' or 'T2' provided. Additionally, the rewards and their advantages for 'S', 'T1', and 'T2' are computed based on similarity and answer quality in order to train the student VLM by using RIL's objective loss.
Beyond these component analyses, our RIL-trained student VLMs exhibit highly competitive per- formance against a range recent open-source and closed-source models. Comparisons across various model sizes (detailed in Table for smaller models and Table 4 for larger ones) and diverse vision-language benchmarks (illustrated in Figures T and 2 show that RIL not only substantially narrows the performance gap to trate-of-te-ee VLMs but, In several instances, surpasses them.
for
Finally, the training dynamics of RIL, depicted in Figure 5 offer further insight. Over the course of training, RIL consistently increases both the similarity rewards (from the discriminator) and the answer rewards (from I.LM-as-a-Judee (9) It indicates that the student VLMs are progressively generating responses that are both more stylistically aligned with teacher models and factually correct, with these improving reward signals directly translating into significant overall performance gains.
For a comprehensive account of the specific training procedures, hyperparameter settings, evaluation setups, and dataset details that ensure the reproducibility of these, please refer to Appendix Using 256 NVIDIA A100 GPUs, pre-training the discriminator on 1.2M samples (40K [number of samples] 16 [generated responses] 2 [for both teacher and student]) takes approximately to 3 days. The SFT step on the 4M-sample SFT dataset takes around to 5 days. Conducting the RIL loop for the sampled 40K data requires additional 3 to days using 8 NVIDIA A100 GPUs.
4.2 Impacts of Hyperparameters, Techniques, and Distillation
We systematically investigate the sources of RIL's effectiveness by analyzing key training hyperpa- rameters, component design choices, and its interplay with other learning techniques. Note that, for the optimization objective of student VL.Ms in Eq. ( the clipping hyperparameter is consistently set to 0.2. Our first analyses focus on core optimization settings. We examine the number of update iterations, μ4 for the discriminator and student VLMs within each RIL cycle (related to Algorithm 2)
8

---

Table 5: Analyzing the impacts of training hyperparameters, techniques for stabilization, and the effect of distillation. Note that, these tables (a)-(c) set a backbone model as Qwen2.5-VI-7B ① and apply RIL to this model in order TO validate it by controlling multiple factors.
(a) Discriminator small VLMs Iteration (JL)
(b) KL-divergence Penalty Cocflicient (3)
(c) Importance VLM Modules
AID Maniiies
MMd
SIM-Ver
Vet MMMI
BLINK
AZDD ManViiu
MM MM--Ve
MMMI
BLINK
Medhicc
MathNatt
MMBB
Mhl-Yet
MMMU
96.1
19.7
$6.3
90.4
65.7
70L0
0.00 77.1
50.0
73.2
60.2
42.9
45.0
Full Triting
75.7
36.3
38.4
65.7
$6.3
Mi.
65.3
69.2
0mi
16.1
M6.4
66.3
75.4
$5.3
15.1
61.3
35.5
76.6
$5.5
70.5
65.0
66.4
0.05
25.7
76.5
63.5
68.3
73.3
61:3
64.3
84.5
1880
762
77.5
780
192
63.4
62.5
618
68.0
0.10
853
84.5
76.6
65.6
62.5
68.6
w. Noyen
wo PFN Doses
Guoe
75.4
79.7
17.3
86.3
16.1
98.4
75.0
75.6
65.7
64.3
948
75.
61.0
66.
684
61.
was Lang-Hlead
24.9
55.2
73.99
(d) Importance of discriminator Do
(e) Effect of Answer Rewards (AR) and SPT
(f) Effect of Distillation
VLM:
Mattioon
MMM
MMM-N
MMMM
Vila
AR
Nariicu
V156
RL.
se48
Qve.25.VV-78
A
X
00.5
$4.3
15.5
25.1
73.5
61.2
57.2
Oend V1.
66.2
603
821
RL5
516
7077"
66.3
W0.
687
25.
In
16
As shown in aa),, a single update iteration (1 = 1) for both is sufficient for strong performance; more iterations, especially for student VL. updates, can risk overfitting. Subsequently, we explore the impact of the Ki--divereence penalty coefficient 3 (Eq. in Table (t). These results confirm that while constraining policy updates is crucial for stability, an overly restrictive penalty (large 3) can hinder performance.
Next, we assess decisions related to model component updates. Table 5(c) details the importance of updating different parameter groups within the student VLMs, revealing that training the self- attention layers, word embeddings, and the language head yields more significant gains than updating networks (FFN) or layer normalization parameters. We also investigate the necessity of continuously updating the discriminator during RII training (Table Interestingly, Table shows that using fixed (not updated: A), pre-trained discriminator can outperform RI--only baselines (denoted by X), but overall RIL. efficacy still critically depends on leveraging continuously trained discriminator (denoted by V/ during training RIL.
In addition, we evaluate RIL's interaction with evaluation methodologies of the predicted answer and its synergy with other training paradigms. Ta) underscores the importance of LLM-as-a- Judge [9] for answer rewards; replacing it with conventional answer parsing (denoted A) significantly degrades performance. particularly on general visual questions (e.g.. "How to cook this dish?" from an image) where fixed parsing rules are inadequate. This highlights the need for flexible, LLM-bbsed evaluation. The same table also affirms that while initial SFT is beneficial for capturing the training data distribution, RII. subsequently delivers substantially larger improvements.
On Table we examine RIL's effectiveness on student VLMs that are already trained on distillation methods like MiniLLM (75) DistilLLM 776,, VL- (77), and VLsI [45] (on same backbones like Qwen2-Vi VL [66 and training datasets As shown in Table SiD, RIL proves particularly potent for these models compared to non-distillation counterparts, likely because prior feature alignment through distillation primes them for RIL's alignment mechanisms.
Lastly, we demonstrate that the binary similarity reward introduced in See. 3.3 enables more efficient and stable training of RII. As shown in Fig. 6, the binary reward consistently outperforms continuous and multi-level discretizations, suggesting that binary feedback provides clearer and more reliable learning signals. This improvement arises because models often struggle to interpret subtle differences in continuous rewards (e.g., why a score of 0.21 should be preferred over 0.20).
4.3 Discussion and Limitation
RIL leverages the diversity of responses generated by large VLMs, which tend to produce more varied and higher-quality answers beyond the ground truth answers used in SFT training datasets. As illustrated in Fig. / increasing the number of teacher-generated responses consistently improves the student model's performance, demonstrating that richer supervision from diverse teacher outputs enhances the student VLM's overall generalization ability. In summary, RIL exploits these diverse, high-quality teacher responses to guide smaller VLMs toward learning more flexible responses beyond fixed ground truth answers.
Furthermore, RII. offers distinct advantages over traditional knowledge distillation techniques (78] Unlike conventional distillation, which typically requires student VLMs to replicate the high-

---

0wen2.5-41-78
75
Score (%6
50
Mativiiks
MIMS
MANNU
Matovitta
MAMHU
D-1 (Continanue)
0.911,22.... 05,1
0,055
1 (Pimnry Oura!
Figure Comparison of RIL performance using different discretization levels of the similarity reward on four benchmarks: Math [3, MMB (20), MM-Vet Vet 4. and MMMU [5) with Qwen2.5. VL-7B ① and Intern VL3-8B [8]
ParTViia
15.
Figure 7:
Performance of RIL with varying numbers of generated responses from a teacher VLM across four benchmarks and their average. They are evaluated on Qwen2.5-VL-7B (..
dimensional logit features of teacher models, RIL. operates exclusively on generated text responses in natural language. This fundamental difference liberates RII. from restrictive feature-level constraints, such as the need for identical image embedding strategies perfectly aligned language tokenizers (including vocabulary, index order, and sequence length) between student and teacher VLMs. Com- often rely on KL divergence- static, non-trainable metric devoid of contextual understanding to measure feature similarity, but RIL employs a trainable discriminator which assesses natural language responses directly, learning to discern nuanced similar- ities and stylistic differences between student and teacher outputs. The adaptability of the trainable discriminator in capturing contextual relationships makes RIL a particularly flexible and powerful approach for lightweight VLM and enhancement.
mon distillation methods 175 45]
While RIL demonstrates these significant strengths, its current implementation has been focused primarily on the post-instruction tuning alignment phase. promising avenue for future research, therefore, involves extending the utility of our discriminator to the initial visual instruction tuning stage itself. We hypothesize that the discriminator's ability to learn from natural language signals and overcome conventional architectural constraints could also offer significant benefits in this earlier phase of VLM training, potentially leading to more efficient instruction following from the outset.
Conclusion
In this work, we have introduced Unified Reinforcement and Imitation Learning (RIL), novel frame- work enabling smaller VLMs to emulate the sophisticated text-generation capabilities of substantially larger counterparts, often matching even exceeding their performance. RIL directly addresses the pressing need for high--erformin By leveraging a dual reward combiinin similarity scores from discriminator with answer accuracy assessments from an LLM-as-a-Judge-Da JuL effectively guides student VLMs towards superior alignment and overall efficacy. Key findings from our research include the significant performance boost achieved by using multiple large teacher VLMs. which provides richer di- versity of training signals, and the notable success of RIL in enhancing distillation-ban Comprehensive experiments across wide array of vision- vision-language benchmarks confirm that RIL substantially narrows, and in several cases surpasses, the performance gap to state-of-the-an open- and closed-source VLMs.
lightweight VLMs deployable in resource-constree
settings.
VLMs.
10

---

References
[1] S. Bai, K. Chen, X. Liu, Wang. W. Ge, S. Song, K. Dang. P. Wang. S. Wang, J. Tang, et al., "Qwen2. 5-yl technical report," arXiv preprint arXiv:2502.13393. 2025.
[2]
Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi, "A diagram is worth a dozen images," in Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 235-251, Springer, 2016.
[3] P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and Gao, "Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts," arXiv preprint N..iv:2330.0225.5. 2023.
[4] W. Yu, Z. Yang, I.. J. Wang, K. Lin, Z. Liu, X. Wang, and L.. Wang, "Mm-vet: Evaluating large multimodal models for integrated capabilities," arXiv preprint 3rXiv:2308.02499 2023.
[5] X. Yue, Y.Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al., "Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi." arXiv preprint arXiv:2311.16500 2023.
[6] X. Fu, Y. Hu, B. Li, Y. Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W.-C. Ma, and R. Krishna, "Blink: Multimodal large language models can see but not perceive," arXiv preprint arXiv:2404. 12390, 2024.
[7] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q Zhu, S. Ma, P. Wang, X. Bi, et al., "Deepseek-rl: Incentivizing reasoning capability in I1ms via reinforcement learning," arXiv preprint arXiv:2501. 12948, 2025.
[8] Z. Liu, C. Chen, W. Li. P. Qi, T. Pang. C. Du, W. S. Lee, and M. Lin, "Understanding rl il-zero-like training: A critical perspective," arXiv preprint arXiv:2503.20783, 2025.
[9] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang. Z. Lin, Z. Li, D. Li, E. Xing, et al., "Judging lim-as-a-judge with mt-bench and chatbot arena," Advances Neural 2023.
Information Processing Systems, vol. 36, pp.
[10] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., "Llama: Open and efficient foundation language models," arXiv preprint arXiv:2302.13971 2023.
[11] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al., "The llama 3 herd of models," arXiv preprint arXiv:2407.2 22007..27702 17783, 2024.
[12] A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang. J. Wang, J. Yang, J. Tu, J. Zhang. J. Ma, J. Yang, J. Xu, J. Zhou, Ni, Zhang, P. Wang, R. Peng. R. Men, R. Gao, R. Lin, S. Wang, S. Bai. S. Tan, T. Zhu, T. Li, T. Liu,
J. Bai, He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue,
W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, X. Liu, Y. Fan, Y. Yao, Y. Zhang,
Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, Z. Guo, and Z. Fan, "Qwen2 technical report," 2024.
[13] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, Wei, et al., "Qwen2. 5 technical report," arXiv preprint arXiv:2412.15115, 2024.
[14] J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, "Finetuned language models are zero-shot learners." in International Conference on Learning Representations, 2022.
[15] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. De- hghani. S. Brahma, et al., "Scaling instruction finnttundd language models," arXiv preprint arXiv:2210.1 1416, 2022.
[16] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin, "Qwen2-vl: Enhancing lasion-language model's perception of the world at any resolution," 2024.
11

---

[17] Z. Chen, W. Wang, Y. Cao, Y. Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, et al., "Expanding performance boundaries open-source multimodal models with model, data, and test-time scaling," arXiv preprint (tx.:x241..007777 2024.
[18] J. Zhu, W. Wang. Z. Chen, Z. Liu, S. Ye, L. Gu, Y. Duan, H. Tian, W. Su, J. Shao, et al., "Internv13: Exploring advanced training and test-time recipes for open-source multimodal models," arXiv preprint Siti::2504.04799 2025.
[19] A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque, "Chartqa: A benchmark for question answering about charts with visual and logical reasoning," arXiv preprint arXiv:2203. 10244, 2022.
[20] Y. Liu, H. Duan, Y. Zhang. B. Li, Zhang. W. Zhao, Y. Yuan, Wang. C. He, Z. Liu, et al., "Mmbench: Is your multi-modal model an all-around player?," arXiv preprint arXiv:2307.0622 2023.
[21] X. Yue, T. Zheng, Y. Ni, Y. Wang. K. Zhang, S. Tong, Y. Sun, B. Yu, G. Zhang. H. Sun, et al., "Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark," arXiv preprint arXiv:2409.022 24211.2211,,... 2024.
[22] L. Chen, J. Li, X. Dong, P. Zhang, Y. 7ang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin, ef al., "Are arXiv:2403. 20330, 2024.
the right way for evaluating large vision-language models?," arXiv preprint
[23] B. Li, R. Wang. G. Wang, Y. Ge. Y. Ge, and Y. Shan, "Seed-bench: Benchmarking multimodal Ilms with generative comprehension." arXiv preprint arXiv:2307.16125. 2023.
[24] B. Li, Y. Ge, Chen, Y. Ge, R. Zhang, and Y. Shan, "Seed-bench-22pluu:: Benchmarking multimodal large language models with text-rich visual comprehension," arXiv preprint WrXiv2244.116999 2024.
[25] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei, "Scaling laws for neural language models," arXiv preprini arXiv:2001.08300 2020.
[26] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, er al., "Palm: Scaling language modeling with pathways," Journal of Machine Learning Research, vol. 24, no. 240, pp. 1-113, 2023.
[27] Y. Li, Y. Zhang, C. Wang, Z. Zhong, Y. Chen, R. Chu, S. Liu, and J. Jia, "Mini-gemini: Mining the potential of multi-modality vision language models," " arXiv preprint 2024.
[28] B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y. Li, Z. Liu, and C. Li, "Llava-onevision: Easy visual task transfer," arXiv preprint 1xXiv:2408.03326 2024.
[29] X. Yue, T. Zheng, G. Zhang, and W. Chen, "Mammoth2: Scaling instructions from the web," 2024.
[30] S. Gu, J. Zhang, S. Zhou, K. Yu. Z. Xing L. Wang, Z. Cao. J. Jia, Zhang, Y. Wang, et al., "Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data," arXiv preprint arXiv:2410. 18558, 2024.
[31] B.-K. Lee, S. Chung, C. W. Kim, B. Park, and Y. M. Ro, "Trol: Traversal of layers for large language and vision models," arXiv preprint ar MrXXv:2406.12246 2024.
[32] B.-K. Lee, S. Chung, C. W. Kim, B. Park, and Y. M. Ro, "Phantom of latent for large language and vision models," arXiv preprint arXiv: 240911133, 2024.
[33] B.-K. Lee, B. Park, C. W. Kim, and Y. M. Ro, "Collavo: Crayon large language and vision model," arXiv preprint arXiv:2402. 1248, 2024.
[34] B.-K. Lee, Park, C. W. Kim, and M. Ro, "Moai: Mixture of all intelligence for large language and vision models," arXiv preprint ar arXiv:2403.07508 2024.
12

---

[35] B.-K. Lee, C. W. Kim, B. Park, and Y. M. Ro, "Meteor: Mamba-based traversal of rationale for large language and vision models," arXiv preprint arXiv:2405.15574, 2024.
[36] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, Fan, G. Liu, L. Liu, X. Liu, et al., "Dapo: An open-source Ilm reinforcement learning system at scale," arXiv preprint arXiv:2503. 14476 2025.
[37] W. Huang, B. Jia, Z. Zhai, S. Cao, Z. Ye, F. Zhao, Z. Xu, Y. Hu, and S. Lin, "Vision-rl: Incentivizing reasoning capability in multimodal large language models," arXiv preprint urXiv22503066453 2025.
[38] Z. Lu, Y. Chai, Y. Guo, X. Yin, L. Liu, H. Wang, G. Xiong, and H. Li, "Ui-rl: Enhancing action prediction of gui agents by reinforcement learning." arXiv preprint arXiv:2503.21620, 2025.
[39] E. Yu, K. Lin, L. Zhao, J. Yin, Y. Wei, Y. Peng, H. Wei, J. Sun, C. Han, Z. Ge, et al., "Perception-rl: Pioneering perception policy with reinforcement learning," arXiv preprint arXiv:2504 24i0:25004.0954, 2025.
[40] X. Liu, J. Ni, Z. Wu, C. Du, L.. Dou, H. Wang, T. Pang, and M. Q- Shieh, "Noisyrollout: Reinforcing visual reasoning with data augmentation," arXiv preprint arXiv:2504. 2025.
[41] H. Shen, P. Liu. J. Li, C. Fang. Y. Ma, J. Liao, Q. Shen, Z. Zhang, K. Zhao, Q. Zhang, et al., "VIm-rl: A stable and generalizable rl-style large vision- laiin-la model." " arXiv preprint arXiv:2504.0766
2025.
[42] L. Chen, Li, H. Zhao, Y. Song, and Vinci, "RI-v: Reinforcing super generalization ability in vision-language models with less than $3." 2025, Accessed: 2025-02-02.
[43] Z. Shao, P. Wang. Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, al., "Deepseekmath: Pushing the limits of mathematical reasoning in open language models," arXiv preprint arXiv:2402. 03300, 2024.
[44] J. Ho and S. Ermon, "Generative adversarial imitation learning," " Advances neural informa- tion processing systems, vol. 29, 2016.
[45] B.-K. Lee, R. Hachiuma, Y.-C. F. Wang, Y. M. Ro, and Y.-H. Wu, "Visi: Verbalized layers-to- interactions from large to small vision language models," arXiv preprint arXiv:2412.01822. 2024.
[46] Y. Li, F. Wei, C. Zhang, and H. Zhang, "Eagle: Speculative sampling requires rethinking feature uncertainty," arXiv preprint arXiv:2401.15077 2024.
[47] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, "Improved techniques for training gans," Advances in neural information processing systems, vol. 29, 2016.
[48] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, "Gans trained by two scale update rule converge to local nash equilibrium," Advances in neural information processing systems, vol. 30, 2017.
[49] Z. Liu, Z. Sun, Y. Zang, X. Dong, Y. Cao, H. Duan, D. Lin, and J. Wang, "Visual-rft: Visual reinforcement fine mmnettnnng," arXiv preprint arXiv:2503.01785, 2025.
[50] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al., "Gpt-40 system card," arXiv preprint arXiv:2410.21270 2024.
[51] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al., "Gemini: a family of highly capable multimodal models," arXiv preprint arXiv:2312.1 / 2023.
[52] Anthropic, "The claude model family: Opus, sonnet, haiku." https: http://wtww.aaa.i....... com 2024.
13

---

[53] M. Deitke, C. Clark, S. Lee, R. Tripathi, Y. Yang, J. S. Park, M. Salehi, N. Muennighoff, K. Lo, L. Soldaini, et al., "Molmo and pixmo: Open weights and open data for state-of-the-an multimodal models," arXiv preprint arXiv:2409. /7146, 2024.
[54] W. Dai, N. Lee, B. Wang, Z. Yang, Z. Liu, J. Barker, T. Rintamaki, M. Shoeybi, B. Catanzaro, and W. Ping, "Nylm: Open frontier-class multimodal Imss," " arXiv preprint, 2024.
[55] Z. Zong, B. Ma, D. Shen, G. Song, H. Shao, D. Jiang, H. Li, and Y. Liu, "Mova: Adapting mixture of vision experts multimodal context," arXiv preprint arXiv:2404. 13046, 2024.
[56] M. Shi, F. Liu, S. Wang. S. Liao, S. Radhakrishnan, D.-A. Huang, H. Yin, K. Sapra, Y. Yacoob, H. Shi, et al., "Fagle: Exploring the design space for multimodal Ilms with mixture of encoders," arXiv preprint arXiv:2408. 15998, 2024.
[57] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, "Learning transferable visual models from natural language supervision." " in Proceedings of the 38th International Conference on Machine Learning (M. Meila and T. Zhang, eds.), vol. 139 of Proceedings of Machine Learning Research, pp. 8748-8763, PMLR, 18-24 Jul 2021.
[58] S. Woo, S. Debnath, Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, "Convnext v2: Co- designing and scaling convnets with masked autoencoders," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16133-16142, 2023.
[59] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, el al. "Dinov2: Learning robust visual features without supervision," " arXiv preprint arXiv:230..07193, 2023.
[60] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al., "Segment anything," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4028 2023.
[61] Y. Peng, G. Zhang, M. Zhang, Z. You, J. Liu, Q Zhu, K. Yang, X. Xu, X. Geng, and X. Yang, "Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl," arXiv preprint arXiv:2503.07533 2025.
[62] Y. Deng, H. Bansal, F. Yin, N. Peng, W. Wang, and K.-W. Chang, "Openvlthinker: An early exploration complex vision-language reasoning via iterative self-improvement" arXiv preprint arXiv: 17352, 2025.
[63] Y. Yang, X. He, H. Pan, X. Jiang, Y. Deng. X. Yang, H. Lu, D. Yin, F. Rao, M. Zhu, et al., "R1- onevision: Advancing generalized multimodal reasoning through cross-modas formalization," arXiv preprint arXiv:2503. 10615, 2025.
[64] H. Zhou, X. Li, R. Wang, M. Cheng, T. Zhou, and C.-J. Hsieh, "R1-zero's" aha moment" in visual reasoning on a 2b non-sft model," arXiv preprint arXiv:2503.05132. 2025.
[65] F. Meng. L. Du, Z. Liu, Z. Zhou, Q. Lu, D. Fu, T. Han, B. Shi, W. Wang, J. He, et al., "Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning," " arXiv preprint arXiv:2503.07365, 2025.
[66] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne, "Imitation learning: A survey of learning methods," ACM Computing Surveys (CSUR), vol. 50, no. 2. pp. 1-35, 2017.
[67] W. Yu, Z. Yang. L. Ren, L. Li, Wang, K. Lin, C.-C. Lin, Z. Liu, L. Wang, and X. Wang, "Mm-vet v2: A challenging benchmark to evaluate large multimodal models for integrated capabilities," " arXiv preprint arXiv:2408. 00765, 2024.
[68] O. Contributors, "Opencompass: A universal evaluation platform for foundation models." 2023.
https://github. com/ open www.pass/oppeccooppp..oooooooopp
[69] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, 7. Muyan, Q. Zhang, X. Zhu, L. Lu, el al., "Internvl: Scaling up vision foundation models and aligning for generic linguisic tasks." arXiv preprint arXiv: 14238, 2023.
14

---

[70] Y. Yao, T. Yu, A. Zhang, C. Wang. J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He. et al., "Miniepm-v: gpt-4v level milm on your phone," arXiv preprint arXiv:2408.00 01800, 2024.
[71] S. Lu, Y. Li, Q-G. Chen, Z. Xu, W. Luo, K. Zhang, and H.-J Ye, "Ovis: Structural embedding alignment for multimodal large language model," arXiv:2405. 20797, 2024.
[72] M. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, Bakhtiari, H. Behl, et al., "Phi-3 technical report: highly capable language model locally your phone." arXiv preprint vxXh::2404.14219 2024.
[73] A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V. Chaudhary, C. Chen, et al., "Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras, arXiv preprint arXiv:2503.01743, 2025.
[74] OpenAL, "Gpt-4v(ision) system card," 2023. Apt-4v-systemm card
https://openai. htt.p///aaaa...oe//reerraaaaaaaa
Last accessed on 2024-02-13.
[75] Y. Gu, L.. Dong, F. Wei, and M. Huang, "Minillm: Knowledge distillation of large language models," in The Twelfth International Conference on Learning Representations, 2024.
[76] J. Ko, S. Kim, T. Chen, and S.-Y. Yun, "Distillm: Towards streamlined distillation for large language models," arXiv preprint arXiv:2402.03888, 2024.
[77] Y. Cai, J. Zhang. H. He, X. He, A. Tong. Z. Gan, Wang, and X. Bai, "Llava-kd: A framework of distilling multimodal large language models," arXiv preprint arXiv:2410.16233 2024.
[78] G. Hinton, O. Vinyals, and Dean, "Distilling the knowledge neural network," arXiv preprint arXiv: 02031, 2015.
[79] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, I.. Zheng, C. Yu, Gonzalez, Zhang, and Stoica, "Efficient memory management for large language model serving with pagedattention." in Proceedings of the 29th Symposium on Operating Systems Principles, pp. 61 1-66, 2023.
[80] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, "Zero: Memory optimizations toward training trillion parameter models," in SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-16, IEEE, 2020.
[81] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization," " in International Confer- ence on Learning Representations, 2019.
[82] Z. Liu, L. Zhu, B. Shi, Z. Zhang, Y. Lou, S. Yang, H. Xi, S. Cao, Y. Gu, D. D. et al., "Nvila: Efficient frontier visual language models," arXiv preprint arXiv:2412.04468, 2024.
[83] S. Singg, A. Yadav, J. Jain, H. Shi, J. Johnson, and K. Desai, "Benchmarking object detectors with coco: A new path forward," in European Conference on Computer Vision, pp. 279-295, Springer, 2024.
[84] G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie, "The inaturalist species classification and detection dataset," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018.
[85] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, "Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering," Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[86] Z. Li, X. Wang, E. Stengel- A. Kortylewski, W. Ma, B. Van Durme, and A. L. Yuille, "Super-clevr. A virtual benchmark to diagnose domain robustness in visual reasoning," in Pro- ceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 14963- 14973, 2023.
[87] R. Zhang, Wei, Jiang, Y. Zhang, Z. Guo, C. Tong, Liu, A. Zhou, B. Wei, Zhang et al., "Mavis: Mathematical visual instruction tuning," arXiv e-prints, pp. arXiv-2407, 2024.
15

---

[88] P. Lu, R. Gong. S. Jiang, L. Qiu, S. Huang, X. Liang, and S.-C. Zhu, "Inter-gpss Interpretable geometry problem solving with formal language and symbolic reasoning," " in The 59th Anmual Meeting of the Association for Computational Linguisties (ACI.), 2021.
[89] P. Lu, S. Mishra, T. Xia, L. Qiu, K.WW Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan, "Learn to explain: Multimodal reasoning via thought chains for science question answering," Advances in Neural Information Processing Systems, vol. 35, pp. 2507-2521, 2022.
[90] Y. Zhang, R. Zhang, J. Gu, Y. Zhou, N. Lipka, D. Yang, and T. Sun, "Llavar: Enhanced visual instruction tuning for text-rich image understanding." arXiv preprint arXiv:2306. 17107, 2023,
[91] F. Liu, G. Emerson, and N. Collier, "Visual spatial reasoning," Transactions of the Association for Computational Linguisties, vol. 11, pp. 635 651, 2023.
[92] M. Acharya, K. Kafle, and C. Kanan, "Tallyqa: Answering complex counting questions," in Proceedings of the AAAI conference on artificial intelligence, vol. 33, pp. 8076-8084 2019.
[93] P. Lu, L. Qiu, K.-W. Chang, Y. N. Wu, S.-C. Zhu, T. Rajpurohit, P. Clark, and A. Kalyan, "Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning," arXiv preprint arXiv:2209. 146/0, 2022.
[94] V. Hosu, H. Lin, T. Sziranyi, and D. Saupe, "Koniq- 10k: An ecologically valid database for deep learning of blind image quality assessment," IEEE Transactions on Image Processing. vol. 29, pp. 4041-4056 2020.
[95] W. Wang, Z. Chen, W. Wang, Y. Cao, Y. Liu, Z. Gao, J. Zhu, X. Zhu, L. Lu, Y. Qiao, et al., "Enhancing the reasoning ability of multimodal large language models via mixed preference optimization," arXiv preprint arXiv:2411.100 10442, 2024.
[96] T. Yu, H. Zhang, Y. Yao, Y. Dang, D. Chen, X. Lu, G. Cui, T. He, Z. Liu, T.-S. Chua, et al., "Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness." arXiv preprint arXiv:240S.11 17220, 2024.
[97] A. D. Lindström and S. S. Abraham, "Clevr-math: A dataset for compositional language, visual and mathematical reasoning." arXiv preprint vxXiv:2208005356 2022.
[98] 7. Huang, K. Chen, He, X. Bai, D. Karatzas, S. Lu, and C. Jawahar, "Icdar2019 competition on scanned receipt ocr and information extraction," in 20/9 International Conference on Document Analysis and Recognition (ICDAR), pp. 1516-1520, IEEE, 2019.
[99] M. Mathew, D. Karatzas, and C. Jawahar, "Docvqa: A dataset for vqa on document images," in winter conference applications of computer vision, pp. 2200- Proceedings of the IEEE/CVF winter 2209, 2021.
[100] S. E. Kahou, V. Michalski, A. Atkinson, Á. Kádár, A. Trischler, and Y. Bengio, "Figureqa: An annotated figure dataset for visual reasoning," arXiv preprint arXiv:/710.07300 2017.
[101] D. A. Hudson and C. D. Manning, "Gqa: new dataset for real-world visual reasoning and compositional question answering," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp.
2019.
[102] M. Mathew, V. Bagal, R. Tito, D. Karatzas, E. Valveny, and C. Jawahar, "Infographicvqa." in Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 1697-1706, 2022.
[103] Q Chen, L. Qin, J. Zhang, Z. Chen, X. Xu, and W. Che, "M A novel benchmark for multi-domain multi-step multi-modal chain-of-thougeee of-thhught," in Proc. of ACL, 2024.
[104] S. Chang, D. Palzer, Li, E. Lusser-Lussier, and N. Xiao, "Mapqa: dataset for question answering on choropleth maps," arXiv preprint arXiv:2211.0885 2022.
[105] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi, "Ok-vqa: A visual question answering benchmark requiring external knowledge," in Proceedings of the IEFF/cvf conference on computer vision and pattern recognition. pp. 3195-3204, 2019.
16

---

[106] A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach, "Towards vqu models that can read," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
[107] Y. Lu, D. Jiang, W. Chen, W. Y. Wang, Y. Choi, and B. Y. Lin. "Wildvision: Evaluating vision- language models in the wild with human preferences," arXiv preprini arXiv:2406.1 1069, 2024.
[108] K. Kafle. B. Price, S. Cohen, and C. Kanan, "Dvqa: Understanding data visualizations via question answering," " in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5648-5656, 2018.
[109] J. Cao and Xiao, "An augmented benchmark dataset for geometric question answering through dual parallel text encoding," in Proceedings of the 29th international conference on computational linguistics, pp. 151 1551-0, 2022.
[110] M. Seo, H. Hajishirzi, A. Farhadi, O. Etzioni, and C. Malcolm, "Solving geometry prob- lems: Combining text and diagram interpretation," in Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 1466-1476, 2015.
[111] P. Lu, L. Qiu, Chen, T. Xia, Y. Zhao, W. Zhang, Z. Yu, X. Liang, and S.-C. Zhu, "Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning," arXiv preprint arXiv:21 13.114 2021.
[112] J. Chen, T. J. J. Qin, P. Lu, L. Lin, C. Chen, and X. Liang, "Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression," arXiv preprint arXiv:2212.02744. 2022.
[113] M. Kazemi, H. Alvari, A. Anand, J. Wu, X. Chen, and R. Soricut, "Geomverse: A systematic evaluation of large models for geometric reasoning," " arXiv preprint arXiv:2312.1224: 2023.
[114] J. Gao, R. Pi, J. Zhang, J. Ye, W. Zhong, Y. Wang, L. Hong, J. Han, H. Xu, Z. Li, et al., "G-llava: Solving geometric problem with multi-modal large language model," arXiv preprint arXiv:2312. 1370, 2023.
[115] W. Shi, Z. Hu, Y. Bin, Liu, Yang, S.-K. Ng. Bing, and R. K.-W. Lee, "Math-llava Bootstrapping mathematical reasoning for multimodal large language models," arXiv preprint arXiv:2406. 17944 2024.
[116] e. a. Yunxin Li, "Cognitive visual- Sanguaga mapper: Advancing multimodal comprehension with enhanced visual knowledge alignment," Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
[117] Y. Chen, H. Hu, Y. Luan, Sun, Changpinyo, A. Ritter, and M.-W. Chang, "Can pre-trainne pre-trained vision and language models answer visual information seekinn questions?," arXiv preprint arXiv:2302.1 11711,11222 2023.
[118] X. Huang, Y.-J Huang, Y. Zhang, W. Tian, R. Feng, Y. Zhang, Y. Xie, Y. Li, and L. Zhang, "Open-set set image tagging with multi-grained text supervision," arXiv preprint arXiv:2310.1522 2023.
[119] B.-K. Lee, Kim, and Y. M. Ro, "Masking adversarial damage: Finding adversarial saliency for robust and sparse network," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15126-15136, 2022.
[120] J. Kim, B.-K. Lee, and Y. M. Ro, "Distilling robust and non-robust features in adversarial examples by information bottleneck," Advances in Neural Information Processing Systems, vol. 34, pp. 17148-17159, 2021.
[121] J. Kim, B.-K. Lee, and Y. M. Ro, "Demystifying causal features on adversarial examples and causal inoculation for robust network by adversarial instrumental variable regression," in Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12302-12312, 2023.
17

---

[122] B.-K. Lee, Y. Yu, and Y. M. Ro, "Towards adversarial robustness of bayesian neural network through hierarchical variational inference," 2021.
[123] B.-K. Lee, R. Hachiuma, Y. M. Ro, Y.-C. F. Wang, and Y.-H. Wu, "Genrecal: Generation after recalibration from large to small vision-language models," arXiv preprint arXiv: 15681, 2025.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Abstract and Introduction section, and Figure 1-2
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Discussion and Limitation section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
There are no theoretical formulations that need to be proved.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Experiment section and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [NA]
Justification: Dataset is all open-source, but the code and the model checkpoints will be open once it accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/ppp wwwppbblccgguiicuceccccccoe.oopp wet.ggddddee/eeebbiniinn
ssiiisolccyyyyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experiment section and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Experiment section and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read all materials regarding the NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix D
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Method, Experiment section, Appendix C
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Experiment Section and Appendix C
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
25

---

A Algorithm of RL only with Answer Rewards from LLM-as-a-Judge
Algorithm 1 RL purely with GRPO or Dr.GRPO based accuracy rewards from LLM LLM-as-a-Judee
Require: Pre-trained VLMs Tea
1: Set reference model Tee f- Tmm
2: Set the training model πe <- Te
3: for sample batch B in Dataset do
4: Copy and freeze model Toaa < 30
5: Sample G' outputs {o:);=] Tew (-1q) for each question B
6: Compute Rewards and Advantages for all G outputs by LLM-as-a-Judge
7: for VLMs Updating iteration = 1, 1,2,
do
8: Update To by using the objective of GRPO [43 or Dr. GRPO
9: end for
10: end for
B Algorithm of RIL for VLMs
Algorithm 2 RIL for VLMs
Require: Pre-trained discriminator D& and Pre-trained student VLMs To..
Require: Saved collection for the generated text responses {o()} } from teacher VLMs
Set reference model Te f- Te...
2: Set the training model गश f- Te...
3: for sample batch B in Dataset do
4: Copy and freeze model Taas f- T0
5: Sample G outputs {o (91 ~ Te. (144 for each question E
6: Extract outputs {o, 1
(=1 in the saved collection for each question g B 7: for Discriminator Updating iteration = 1,2, ... , do
8: Update D& by using Equation
9: end for
Compute Rewards and Advantages for all 2G outputs by Discriminator and LLM-as-a-Judge
11: for Student VLMs Updating iteration = 1,2, do
Update Te by using Equation 2
13: end for
end for
C Implementation Detail
Details of Training and Evaluation. We train and evaluate RIL, mainly NVIDIA A100 80GB GPUs. Because it is practically critical point to get fast text generation during training, we utilize vLLM 79 built PagedAttention. For the pre-training step of discriminator, to GPUs for fast text generation, and we generate N=16 text responses for each question at student and teacher VLMs, respectively. Once it finished, we DeepSpeed engine with ZeRO-3 88 for 8 GPUs, and we use Adam W optimizer [8T] and apply a linearly decayed learning rate from le-5 le-6 pre-training discriminator and SFT of student VLMs. In subsequent step, when training both discriminator and student VLMs, one of GPUs is assigned to conduct online generation of student VLMs by vLLM 79). In addition, another GPU is assigned conduct LLM-as-a- Judge [9] by vLLM (79. The other 6 GPUs are assigned to use DeepSpeed engine with ZeRO-3 [0). Mimicking step requires static learning rate le-6 and μ=l iteration to train both student VLMs and the discriminator. Note that, when we generate text responses, we generate G=4 responses for each question, by setting temperature to 1.0, top-p to 0.95, top-k to 50, and repetition penalty to 1.05, in order to get diverse text responses. For stable training, we handle large batch sizes by using gradient accumulation with 6 steps. At every step, we use batches per one GPU, leading to total 144 batches. For evaluation,
assign vLLM (79
distributed data parallel to load student VLMs for all 8 GPUs and
26

---

score the correctness or generation quality of their text responses by using their default generation hyperparameter.
RIL requires increased computational costs during training due teacher, student, discriminator, and LLM-as-aa au-auueg [9]. However, we would like to emphasize that discriminator and student costs can be at least mitigated. While pretraining the discriminator is a necessary step and cannot be avoided from computational standpoint, we can reduce training costs of discriminator and student during the RIL loop (Algorithm 2) by managing model weights efficiently between CPU and GPU. Specifically, we offload the weights of the discriminator and student the CPU when they not in When needed, load the appropriate weights from the CPU to the GPU using DeepSpeed API [80]. This allows us to load or unload model weights without uploading both architecture to GPU, thereby minimizing computational overhead. This approach is feasible because we use the same model architecture for both the discriminator and student, enabling seamless weight replacement and resource-efficien training.
Computational Cost. Compared to GRPO 43
Visual Instruction Tuning Dataset. We assemble a 4M-sample visual instruction tuning dataset that encompasses diverse array of vision- lisiun-lagguage tasks, including general visual question answer- ing, dense image captioning, chart, diagram, and document understanding, common-sense knowledge, scientific and mathematical problem-solving. and multidimensional reasoning. For SFT, utilize the entire 4.M-sample dataset, and then we curate a 40K-sample dataset for RIL of VLMs based on sig--bbbablity sampling (82) and overlong filtering [36]. Our dataset integrates both real-world
and synthetic sources: COCO-ReM [83], iNaturalis20018 [84], VQA-v2 (85). Super-CLEVR [86).
MAVIS [87], Geometry3K (88) SQA [89]. A12D 22. SA-1B (60), LAR (00). VSR
99] Tal-
lyQA [92), TabMWP 93), KonlQ [94] InternVL [95]-filtered synthetic knowledge dataset covering
politics, math. physics, chemistry, RLAI-F [96), CLEVR-Math (97]. SROIE (98), ChartQA [I9].
DocVQA 199]. FigureQA [100]. GQA [101]. Info VQA (102). M3CoT [00]] MapQA [104).
OK-VQA TOS), TextVQA [066. WildVision [177]. DVQA [08], GeoQA+ 109, GeOS [TTO]
IconQA I. UniGEO 112,, Geom Verse
[13), 70K [14), Math [115), multimodal
wikipedia knowledge [16], InfoSeek [177, and RAM++ [I8 filtered synthetic data of Infinity-
MM 30 covering coarse and fine- tne-grained perception, relation, attribute, and logic reasoning.
D Broader Impacts
Our reinforcement and imitation learning (RIL) approach democratises access to advanced vi- sion-language capabilities by enabling lightweight, efficient models low-resource
edge devices and in lowering both deployment costs and environmental impact. RIL fosters seamless interoperability across diverse model architectures and tokenization schemes. The use of multiple large teacher VLMs to generate varied training samples further accelerates perfor- mance gains, underscoring the value of collaborative model development. We believe RII. will inspire future research into efficient multimodal learning techniques and support the widespread, sustainable adoption of lightweight [119] and robust 112 122] AI solutions across broad range of applications even with different tokenizers [123]. We hope that by bridging efficiency and performance, RIL become landscape of building sophisticated multimodal AI technologies and accelerating innovation across industries and academia.
27


Reviews:
[{"review_id": "04ZiugFyeq", "reviewer": "Reviewer_Hg1D", "summary": "- This paper introduces RIL, a training framework that enables smaller vision-language models to mimic the text generation capabilities of significantly larger teacher models by combining reinforcement learning with adversarial imitation learning.\n- RIL employs a dual reward system consisting of similarity rewards from an LLM-based discriminator that distinguishes between student and teacher outputs, and accuracy rewards from LLM-as-a-Judge evaluation.\n- Extensive experiments across a wide array of vision-language benchmarks using multiple model architectures (Qwen2.5-VL and InternVL3 variants from 1B to 8B parameters) demonstrate consistent performance improvements, with RIL-trained models achieving competitive results with state-of-the-art open and closed-source VLMs.", "questions": "Please refer to the weaknesses section for details regarding the discriminator design and presentation issues.", "limitations": "Technical limitations are discussed, but potential negative societal impact is not discussed (but this doesn't appear to be critical from my understanding).", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The framework demonstrates impressive performance gains, with 7B parameter models achieving competitive or superior results compared to models 10x larger and proprietary systems across well-established vision-language benchmarks. Visualizations in Figure 1 is effective as well.\n- The integration of GRPO with GAIL provides a principled approach to combining reinforcement learning with imitation learning for comprehensive training optimization.\n- Evaluation across two distinct model families (InternVL3 and Qwen2.5-VL) with multiple parameter scales (1B to 8B) demonstrates the method's broad applicability and generalizability across different vision-language architectures.\n- Comprehensive ablation studies systematically investigate major design choices.\n\n## Weaknesses\n- The framework lacks clarity in several key aspects. The initialization and training dynamics of LLM-as-a-Judge are not well explained - it's unclear whether it's trained simultaneously with SFT or how its learning evolves during RIL. Additionally, training duration and stopping conditions are not specified, and the paper doesn't adequately explain why RIL produces considerably different improvements across benchmarks.\n- Converting continuous discriminator scores to binary values may discard informative signals, e.g., to which extent the responses from the student model is indiscernible. Concrete justification beyond \"drawing inspiration from prior works\" could help strengthen the motivation behind this design.\n- There seems to be presentation issues with experimental results appearing before the proposed framework is explained (e.g., Table 1 on page 5), and excessive tables with extensive numerical results throughout the main paper. The core contribution could be communicated more effectively by moving comprehensive numerical results to the appendix while retaining only the most critical comparisons in the main text.", "was_revised": true, "final_justification": "I have carefully reviewed the other reviewers' comments and the authors' rebuttal.\n\nAdditional clarifications were helpful in resolving my concerns (e.g., on training details and performance analysis), and I believe most of the weaknesses I identified have been adequately addressed through extensive experiments, including the ones provided in response to the other reviewers' comments. I will maintain my original rating of 5.", "rebuttal": "**Q1. The framework lacks clarity in several key aspects. The initialization and training dynamics of LLM-as-a-Judge are not well explained - it's unclear whether it's trained simultaneously with SFT or how its learning evolves during RIL.**\n\nWe would like to clarify that, as mentioned in line 164, Qwen2.5-32B is used as LLM-as-a-Judge. It is not trained during SFT or RIL steps. This model serves solely to determine whether the predicted response accurately reflects the meaning of the ground truth. We will revise the manuscript to clarify this point and avoid reader confusion.\n\n---\n\n**Q2. Additionally, training duration and stopping conditions are not specified.**\n\nUsing 256 NVIDIA A100 GPUs, pre-training the discriminator on 1.2M samples (40K [number of samples] $\\times$ 16 [generated responses] $\\times$ 2 [for both teacher and student]) takes approximately 1 to 3 days. The SFT step on the 4M-sample SFT dataset takes around 3 to 5 days. Conducting the RIL loop for the sampled 40K data requires an additional 3 to 5 days using 8 NVIDIA A100 GPUs. We do not apply specific stopping conditions; instead, we train one epoch but conduct four repetitive sampling per question-answer pair.\n\n---\n\n**Q3. The paper doesn't adequately explain why RIL produces considerably different improvements across benchmarks.**\n\nWe believe the performance differences of RIL across benchmarks stem from the nature of the tasks themselves. Some benchmarks require only short or factual answers, for example, \"what color of t-shirt is a person wearing in the image?\", (e.g, SEED2+ and RWQA), while others demand longer reasoning (e.g., MathVista, MMMU), which covers image description, advanced chart-document reasoning, knowledge understanding, and multi-dimensional or step-by-step solutions. Since reinforcement learning introduces a generalization effect, especially when mimicking the text style of large VLMs, the smaller models naturally learn to produce more coherent, logical, and well-structured responses. This is particularly beneficial in tasks that require reasoning or explanation. Therefore, RIL tends to yield greater improvements on benchmarks that benefit from such reasoning ability.\n\n| Models                   | SEED2+ | RWQA | MathVista | MMMU |\n|--------------------------|:------:|:----:|:---------:|:----:|\n| Qwen2.5-VL-7B                   |  70.4  | 68.5 |    67.8   | 55.0 |\n| Qwen2.5-VL-RIL-7B (Both) |  72.8  | 72.8 |    79.7   | 65.7 |\n| $+\\Delta$ (\\%)                     |   2.4  |  4.3 |    11.9    | 10.7 |\n\n---\n\n**Q4. Converting continuous discriminator scores to binary values may discard informative signals, e.g., to which extent the responses from the student model is indiscernible.**\n\nWe would like to clarify that using binary rewards enables efficient and stable training by controlling the reward steps applied. As shown in the table, a simpler reward design appears to positively impact training stability and efficiency.\n\n| Models        | Reward Step (Similarity Reward)                   | MathVista | MMB  | MM-Vet | MMMU | Avg  |\n|---------------|---------------------------------------------------|:-----------:|:------:|:--------:|:------:|:------:|\n| Qwen2.5-VL-7B | 0~1 (Continuous)                                  | 74.0      | 80.6 | 74.7   | 60.0 | 72.3 |\n| Qwen2.5-VL-7B | 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1 | 75.3      | 81.9 | 76.0   | 61.3 | 73.6 |\n| Qwen2.5-VL-7B | 0, 0.5, 1                                         | 78.8      | 85.4 | 79.5   | 64.8 | 77.1 |\n| Qwen2.5-VL-7B | 0,1 (Binary, **Ours**)                            | 79.7      | 86.3 | 80.4   | 65.7 | 78.0 |\n| InternVL3-8B  | 0~1 (Continuous)                                  | 74.8      | 85.1 | 77.1   | 65.6 | 75.7 |\n| InternVL3-8B  | 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1 | 76.0      | 86.3 | 78.3   | 66.8 | 76.9 |\n| InternVL3-8B  | 0, 0.5, 1                                         | 77.1      | 87.4 | 79.4   | 67.9 | 78.0 |\n| InternVL3-8B  | 0,1 (Binary, **Ours**)                            | 77.8      | 88.1 | 80.1   | 68.6 | 78.7 |\n\nThere are two possible reasons for this: (a) it is difficult for models to understand why a score of 0.21 is better than 0.2, and (b) discriminator scores might be overconfident or underconfident—a well-known phenomenon in deep learning. This has led to many studies highlighting the importance of calibrated confidence. However, utilizing continuous rewards is not the focus of our work, and therefore, we do not incorporate any calibration techniques.\n\n---\n\n**Q5. Concrete justification beyond \"drawing inspiration from prior works\" could help strengthen the motivation behind this design.**\n\nPrior works [1-2] have argued that natural language response-based distillation is more effective than high-dimensional feature distillation, emphasizing the importance of utilizing the language head—referred to as the verbalization effect. Motivated by this insight, we began by leveraging the similarity in natural language responses between teacher and student models through reinforcement learning and imitation learning. We will incorporate this explanation into the manuscript to better clarify the motivation behind our design.\n\n[1] Lee, Byung-Kwan, et al. \"VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[2] Li, Yuhui, et al. \"Eagle: Speculative sampling requires rethinking feature uncertainty.\" arXiv preprint arXiv:2401.15077 (2024).\n\n---\n\n**Q6. There seems to be presentation issues with experimental results appearing before the proposed framework is explained (e.g., Table 1 on page 5), and excessive tables with extensive numerical results throughout the main paper. The core contribution could be communicated more effectively by moving comprehensive numerical results to the appendix while retaining only the most critical comparisons in the main text.**\n\nWe will revise the structure of the paper to ensure that the proposed framework and its components are clearly introduced before presenting the corresponding experimental results, including Table 1. Additionally, we will relocate the more exhaustive numerical results to the appendix and retain only the most essential comparisons to highlight our core contributions.\n\n---\n\nThank you for your encouraging review. We hope our responses have further clarified the points you raised."}, {"review_id": "81USF5ZJH5", "reviewer": "Reviewer_SQq6", "summary": "This paper presents Unified Reinforcement and Imitation Learning, a novel training framework designed to enhance small VLMs by enabling them to imitate the performance and generation style of significantly larger models. RIL integrates reinforcement learning with adversarial imitation learning, allowing student models to optimize for both stylistic similarity (to large teacher models) and factual correctness (via LLM-as-a-Judge evaluations). Extensive experiments on a variety of benchmarks demonstrate that student VLMs trained with RIL can not only match but sometimes outperform state-of-the-art open- and closed-source models, despite their relatively smaller sizes.", "questions": "1. How sensitive is RIL to the quality or diversity of teacher outputs? Would performance degrade if teacher models produced inconsistent or conflicting answers?\n\n2. Are there any observed failure modes (e.g., mode collapse, overfitting to teacher phrasing) during RIL training? If so, how are they mitigated?\n\n3. How is the answer parsed from the responses generated by RIL-finetuned models, given that no specific response format is enforced? For instance, ChartQA requires short phrase answers, while several other benchmarks demand a discrete choice selection.\n\n4. What prompt format is used for the teacher models? For questions with deterministic answers, the similarity and answer rewards may become highly correlated, potentially limiting the extent of knowledge transfer from teacher to student.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. Innovative Integration: The paper successfully combines reinforcement and imitation learning in a unified framework, addressing limitations in prior work that rely solely on one learning paradigm.\n\n2. Strong Empirical Results: The method demonstrates state-of-the-art or competitive performance across numerous benchmarks and significantly narrows the gap between small and large VLMs.\n\n3. Robustness Across Models: RIL is shown to work with different student and teacher architectures and is particularly effective when multiple teachers are used.\n\n**Weaknesses**\n\n1. Limited Discriminator Design: As shown in Figure 3, the discriminator does not receive any information about the question. Despite fine-tuning, it relies solely on limited cues from the response, and no examples of its decision outputs are provided.\n\n2. Increased Complexity and Cost: Unlike the original GRPO method that employs rule-based rewards, the proposed approach relies on multiple models to compute rewards. Some components, such as the discriminator, require pretraining or continual fine-tuning, increasing both complexity and computational cost.\n\n3. Insufficient Discussion of LLM Bias and Reward Hacking: The core idea of using multiple LLMs as evaluators introduces risks of bias, (e.g., https://arxiv.org/html/2410.21819v1 ). Moreover, compared to rule-based rewards, LLM-based evaluation is more susceptible to reward hacking, a concern not adequately addressed in the paper.", "was_revised": true, "final_justification": "Most of my issues have been addressed after the authors included additional sensitive experiments. As a result, I raise my score to borderline accept. That said, the paper still requires major revisions, particularly in the areas of computational demands and case study. I recommend that the authors revise these aspects if the paper is accepted. However, rejecting the paper and reevaluating it through a new submission would also be acceptable.", "rebuttal": "**Q1. The discriminator relies solely on limited cues from the response, and no examples of its decision outputs are provided.**\n\nWhile the discriminator only receives the response as input, this design is intentional for the sake of training efficiency. We have observed that incorporating additional input cues (such as the question including image) does not consistently improve performance and instead introduces more randomness. This suggests that the key contribution lies not in carefully tuning the input format, but in leveraging the discriminator itself as a training signal. Therefore, to reduce computational cost, we deliberately shorten the discriminator's input. For greater clarity, we will definitely include qualitative examples (image, question, predicted response, decision) of the discriminator's outputs in the revised version.\n\n| Models        | Discriminator Input                  | MathVista | MMB  | MM-Vet | MMMU | Avg  |\n|---------------|--------------------------------------|:-----------:|:------:|:--------:|:------:|:------:|\n| Qwen2.5-VL-7B | Question + Response                  | 79.0      | 85.5 | 80.2   | 66.5 | 77.8 |\n| Qwen2.5-VL-7B | Response                             | 79.7      | 86.3 | 80.4   | 65.7 | 78.0 |\n| InternVL3-8B  | Question + Response                  | 78.5      | 88.0 | 81.0   | 67.7 | 78.8 |\n| InternVL3-8B  | Response                             | 77.8      | 88.1 | 80.1   | 68.6 | 78.7 |\n\n---\n\n**Q2. Increased Complexity and Cost**\n\nCompared to the original rule-based GRPO, we agree that RIL introduces increased computational costs during training. However, we would like to emphasize that these costs can be mitigated through efficient technical implementation strategies. While pretraining the discriminator is a necessary step and cannot be avoided from a computational standpoint, we can reduce training costs during the RIL loop by managing model weights efficiently. Specifically, we offload the weights of the small VLMs and the discriminator to the CPU when they are not in use. When needed, we load the appropriate weights from the CPU to the GPU using DeepSpeed API. This allows us to dynamically swap model weights without uploading both architecture to GPU, thereby minimizing computational overhead. This approach is feasible because we use the same model architecture for both the small VLMs and the discriminator, enabling seamless weight replacement and resource-efficient training.\n\n---\n\n**Q3. Insufficient Discussion of LLM Bias and Reward Hacking**\n\nTo mitigate reward hacking and bias when using LLM-as-a-Judge, we employ a structured, template‑based format: we wrap the user query, the ground truth, and the model’s output in question, ground truth, and generated text tags, respectively. It is then prompted to explain its reasoning and generate its judgment within answer tags. This structured approach has been widely adopted in prior work [1–5]. Additionally, we cut off the answer reward signal of LLM-as-a-Judge to avoid overestimation or reward hacking, which leads to more stable training. The table below shows how performance varies with different granularities of the answer reward, where a dash '–' denotes mode collapse—training failed because small VLMs could not reliably distinguish, for instance, an answer reward of '0.2' from '0.21'.\n\n\n| Models        | Reward Step (Answer Reward)                       | MathVista | MMB  | MM-Vet | MMMU | Avg  |\n|---------------|---------------------------------------------------|:-----------:|:------:|:--------:|:------:|:------:|\n| Qwen2.5-VL-7B | 0~1 (Continuous)                                  | -         | -    | -      | -    | -    |\n| Qwen2.5-VL-7B | 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1 | 60.2      | 78.2 | 64.2   | 50.9 | 63.4 |\n| Qwen2.5-VL-7B | 0, 0.5, 1                                         | 74.9      | 84.3 | 76.4   | 62.0 | 74.4 |\n| Qwen2.5-VL-7B | 0,1 (Binary)                                      | 79.7      | 86.3 | 80.4   | 65.7 | 78.0 |\n| InternVL3-8B  | 0~1 (Continuous)                                  | -         | -    | -      | -    | -    |\n| InternVL3-8B  | 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1 | 64.8      | 76.7 | 71.4   | 55.9 | 67.2 |\n| InternVL3-8B  | 0, 0.5, 1                                         | 73.8      | 84.6 | 77.4   | 64.7 | 75.1 |\n| InternVL3-8B  | 0,1 (Binary)                                      | 77.8      | 88.1 | 80.1   | 68.6 | 78.7 |\n\n\n[1] Saha, Swarnadeep, et al. \"Learning to plan and reason for evaluation with thinking-llm-as-a-judge.\" arXiv preprint arXiv:2501.18099 (2025).\n\n[2] Yu, Jiachen, et al. \"Improve llm-as-a-judge ability as a general ability.\" arXiv preprint arXiv:2502.11689 (2025).\n\n[3] Chen, Nuo, et al. \"Judgelrm: Large reasoning models as a judge.\" arXiv preprint arXiv:2504.00050 (2025).\n\n[4] Whitehouse, Chenxi, et al. \"J1: Incentivizing thinking in llm-as-a-judge via reinforcement learning.\" arXiv preprint arXiv:2505.10320 (2025).\n\n[5] Huang, Hui, et al. \"Think-j: Learning to think for generative llm-as-a-judge.\" arXiv preprint arXiv:2505.14268 (2025).\n\n---\n\n**Q4. How sensitive is RIL to the quality or diversity of teacher outputs?**\n\nBoth the diversity and the quality of teacher outputs are critical to RIL’s success. As the ablation study below shows, RIL’s performance degrades when teacher responses lack diversity or accuracy. (We assume diversity is satisfied by generating a larger number of responses, since we set the inference hyperparameters—temperature, top‑p, and top‑k.)\n\nThe first table reports performance as we vary the number of pre‑generated teacher responses used to train the discriminator and drive the RIL loop:\n\n| Number of Generated Responses | MathVista | MMB  | MM-Vet | MMMU | Avg  |\n|-------------------------------|:-----------:|:------:|:--------:|:------:|:------:|\n| 0 (Dr.GRPO)                   | 69.5      | 84.3 | 73.5   | 57.2 | 71.1 |\n| 2                             | 73.0      | 85.5 | 75.0   | 60.5 | 73.5 |\n| 4                             | 76.5      | 86.0 | 78.0   | 64.3 | 76.2 |\n| 8                             | 77.0      | 86.2 | 79.0   | 68.6 | 77.7 |\n| 16                            | 79.7      | 86.3 | 80.4   | 65.7 | 78.0 |\n| 32                            | 80.0      | 86.5 | 80.3   | 65.2 | 78.0 |\n\nNote that, this result is based on Qwen2.5-VL-RIL-7B (Both).\n\nWe also evaluated the effect of introducing incorrect answers by randomly shuffling a percentage of generated responses relative to their questions (a dash “–” indicates mode collapse):\n\n| Shuffle Percent  | MathVista | MMB  | MM-Vet | MMMU | Avg  |\n|------------------|:-----------:|:------:|:--------:|:------:|:------:|\n| 0\\%              | 79.7      | 86.3 | 80.4   | 65.7 | 78.0 |\n| 10\\%             | 75.0      | 82.0 | 77.0   | 59.6 | 73.4 |\n| 30\\%             | 62.0      | 68.0 | 60.0   | 30.0 | 55.0 |\n| 50\\%             | -         | -    | -      | -    | -    |\n| 100\\%            | -         | -    | -      | -    | -    |\n\nNote that, this result is based on Qwen2.5-VL-RIL-7B (Both).\n\nIn summary, RIL benefits substantially from both diverse and accurate teacher outputs. Performance improves steadily as the number of high‑quality, varied responses increases. Conversely, introducing inconsistent or conflicting answers causes a marked degradation in performance.\n\n---\n\n**Q5. Are there any observed failure modes during RIL training? If so, how are they mitigated?**\n\nBased on our answers to Q3 and Q4 above, we have addressed these failure modes. Briefly, to prevent them: (1) we use a structured prompt with tags and explicit reasoning for LLM-as-a-Judge, and (2) we convert the continuous reward signal into a binary one to stabilize training. Our response to Reviewer Hg1D’s Q4 provides additional details on training stabilization in the discriminator and the similarity reward signal. Furthermore, to avoid failure modes, we have already conducted the hyperparameter tuning (e.g., the number of updating parameter of discriminator and small VLMs in one training iteration, and coefficient of KL divergence), as shown in Table 5(a) and (b).\n\n---\n\n**Q6. How is the answer parsed from the responses generated by RIL-finetuned models?**\n\nAll evaluation benchmarks for VLMs already provide guidelines on how to parse responses and evaluate accuracy or other metrics. We follow each benchmark's parsing rules, in which we typically use pre-defined input prompts such as: \"Answer the question using a single word or phrase\", \"Please answer yes or no\", and \"Answer with the option’s letter from the given choices directly\".\n\n---\n\n**Q7. What prompt format is used for the teacher models? For questions with deterministic answers, the similarity and answer rewards may become highly correlated, potentially limiting the extent of knowledge transfer from teacher to student.**\n\nAs you mentioned, deterministic answers can clearly limit the extent of knowledge transfer from teacher to student. To address this, we sampled 40K examples from the 4M SFT dataset based on answer length, specifically to avoid deterministic or overly simple questions and instead focus on open-ended or advanced ones. We assumed that if a ground-truth answer required more than 256 tokens, it was more likely to represent an advanced problem. Additionally, we filtered for diverse domains—including image description, advanced chart-document reasoning, knowledge understanding, and multi-step or multi-dimensional solutions—to ensure a broad range of complex tasks. In this way, we made a deliberate effort to avoid deterministic questions and promote richer knowledge transfer. We will definitely add this explanation about how we constructed RIL dataset to the manuscript.\n\n---\n\nWe sincerely hope that our rebuttals have resolved your concerns and will lead to a reconsideration of the scores."}, {"review_id": "tr4aqfVn31", "reviewer": "Reviewer_oXVe", "summary": "This paper introduces a new approach for combining reinforcement learning and imitation learning to improve the reasoning capabilities of Vision-Language Models (VLMs). The core idea is to use a larger teacher VLM whose responses guide the post-training of a smaller VLM through imitation. This motivation is supported by improved performance observed under this training paradigm. The authors present their findings with clear experiments and well-designed ablations that effectively support the proposed method.", "questions": "* Based on the experiments, it appears that applying SFT before RIL improves post-training outcomes. In Table 1, when comparing against the GRPO baseline, could the authors clarify whether the GRPO baseline also included SFT as a prior step?\n* I understand the single-teacher LLM setup, where the discriminator model provides a binary signal indicating whether a response comes from the teacher. Could the authors clarify how this approach extends to the multiple-teacher scenario?\n* Finally, could the authors provide a comparison of the speed to achieve good performance versus vanilla GRPO? It would be interesting to see whether the proposed RIL approach leads to faster convergence.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n* The paper is clearly written and easy to follow, with strong experimental support and well-designed ablations for the proposed method.\n* The idea of combining reinforcement learning and imitation learning is well established in traditional RL literature, and adapting this approach to VLMs is both novel and interesting.\n\n**Weaknesses**\n\n* One limitation is that the paper focuses solely on VLMs; it would have been valuable to see the approach extended to LLMs as well.\n* Given the large number of tasks in Table 1, it would be helpful if the authors included an average metric to make it easier to assess the overall improvement at a glance.", "was_revised": true, "final_justification": "The paper is technically solid and proposes a novel way to combine imitation learning (mimicing a teacher) and RL to improve VLMs. Given the experiments and the analyses performed I would stick with my original rating of 'Accept'", "rebuttal": "**Q1. One limitation is that the paper focuses solely on VLMs; it would have been valuable to see the approach extended to LLMs as well.**\n\nWe have also applied RIL to LLMs to evaluate its generalization capabilities. Based on these results, we confirm that RIL is effective for LLMs as well.\n\n| Student LLMs (Teacher LLMs)                        | MMLU | GPQA | MATH |\n|------------------------------|:------:|:------:|:------:|\n| Qwen2.5-7B                   | 74.2 | 36.4 | 49.8 |\n| Qwen2.5-7B-RIL (Qwen2-72B)   | 79.6 | 37.0 | 50.4 |\n| Qwen2.5-7B-RIL (Qwen2.5-72B) | 81.0 | 41.2 | 56.7 |\n| Qwen2-72B                    | 84.2 | 37.4 | 50.9 |\n| Qwen2.5-72B                  | 86.1 | 45.9 | 62.1 |\n\nNote that, rows without parentheses indicate baseline models without any distillation.\n\n---\n\n**Q2. Given the large number of tasks in Table 1, it would be helpful if the authors included an average metric to make it easier to assess the overall improvement at a glance.**\n\nWe will definitely include an average metric to facilitate easier assessment of overall improvements at a glance. As shown in the table below, it is clear that the proposed approach, \"w. RIL (Dr.GRPO + GAIL)\", achieves the best performance across all VLM families—even when using only a single teacher VLM.\n\n| VLMs                    | Avg   |\n|-------------------------|:-------:|\n| Qwen2.5-VL-7B           | 69.4  |\n| w. RL (GRPO)            | 70.3  |\n| w. RL (Dr.GRPO)         | 71.1  |\n| w. RIL (Dr.GRPO + GAIL) | 75.1  |\n| Qwen2.5-VL-3B           | 64.1  |\n| w. RL (GRPO)            | 65.5  |\n| w. RL (Dr.GRPO)         | 66.4  |\n| w. RIL (Dr.GRPO + GAIL) | 68.7  |\n| InternVL3-8B            | 71.4  |\n| w. RL (GRPO)            | 72.5  |\n| w. RL (Dr.GRPO)         | 73.1  |\n| w. RIL (Dr.GRPO + GAIL) | 75.3  |\n| InternVL3-2B            | 62.6  |\n| w. RL (GRPO)            | 63.6  |\n| w. RL (Dr.GRPO)         | 64.5  |\n| w. RIL (Dr.GRPO + GAIL) | 66.1  |\n| InternVL3-1B            | 55.7  |\n| w. RL (GRPO)            | 56.5  |\n| w. RL (Dr.GRPO)         | 57.2  |\n| w. RIL (Dr.GRPO + GAIL) | 59.8  |\n\n---\n\n**Q3. Based on the experiments, it appears that applying SFT before RIL improves post-training outcomes. In Table 1, when comparing against the GRPO baseline, could the authors clarify whether the GRPO baseline also included SFT as a prior step?**\n\nYes. All experiments reported in the manuscript, including the GRPO baseline, were conducted after completing the SFT training phase.\n\n---\n\n**Q4. I understand the single-teacher LLM setup, where the discriminator model provides a binary signal indicating whether a response comes from the teacher. Could the authors clarify how this approach extends to the multiple-teacher scenario?**\n\nGiven a response T1 from one teacher VLM and T2 from another teacher VLM, and a response S from the student VLM, then the discriminator is trained to output zero when S is provided and one when either T1 or T2 is provided. Additionally, the rewards and their advantages for S, T1, and T2 are computed based on similarity and answer quality. The student VLM is updated using RIL’s objective loss. We will clarify the multi-teacher training setup in the revised manuscript.\n\n---\n\n**Q5. Finally, could the authors provide a comparison of the speed to achieve good performance versus vanilla GRPO? It would be interesting to see whether the proposed RIL approach leads to faster convergence.**\n\nIn terms of convergence speed, GRPO actually converges faster than the RIL framework. This is because GRPO does not rely on any external models, such as a discriminator or a teacher model, but instead improves upon itself. This self-reliance imposes a natural limitation on performance improvement, making the convergence appear relatively quick. In contrast, RIL leverages external models to guide the learning process, allowing it to gradually improve and ultimately surpass the performance of GRPO. While this results in slower convergence compared to GRPO, the overall performance gain is significantly higher, as demonstrated across multiple experiments. (\\% denotes training percentage, and the results describe average metrics for 14 evaluation benchmarks based on Table 1 in the manuscript.)\n\n| Methods | 0\\%  | 10\\%| 20\\%| 30\\%| 40\\%| 50\\%| 60\\%| 70\\%| 80\\%| 90\\%| 100\\%|\n|---------|:------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:------:|\n| GRPO    | 69.7 | 69.9| 70.0| 70.1| 70.2| 70.3| 70.3| 70.3| 70.3| 70.3| 70.3 |\n| Dr.GRPO | 69.7 | 70.0| 70.3| 70.7| 70.8| 71.0| 71.1| 71.1| 71.1| 71.1| 71.1 |\n| RIL (Single Teacher)     | 69.7 | 69.9| 70.3| 71.2| 71.5| 72.6| 74.0| 74.7| 75.0| 75.1| 75.1 |\n\nNote that, this result is based on Qwen2.5-VL-7B.\n\n---\n\nThank you for your encouraging review. We hope our responses have further clarified the points you raised."}, {"review_id": "KXT5ja9Fs9", "reviewer": "Reviewer_2zPb", "summary": "This paper presents a novel training framework, Unified Reinforcement and Imitation Learning (RIL), designed for efficient and lightweight Vision-Language Models (VLMs). By integrating reinforcement learning (RL) and adversarial imitation learning (GAIL), RIL enables smaller \"student\" VLMs to mimic sophisticated text-generation patterns of larger \"teacher\" models and simultaneously improve through reinforcement signals. A key innovation is the use of a large language model (LLM)-based discriminator to distinguish between teacher and student outputs, combined with an LLM-as-a-Judge system to verify factual correctness, ensuring both stylistic and factual alignment. Experiments demonstrate that RIL significantly improves student model performance, achieving results competitive with or superior to leading open- and closed-source models.", "questions": "1. What mechanisms or hyperparameter tuning strategies are critical to maintaining training stability, particularly in adversarial imitation settings?\n2. Could RIL be effectively combined with other recent model alignment methods (e.g., instruction tuning or retrieval augmentation)?\n3. How sensitive is the approach to the selection and quality of large teacher VLMs? Would performance degrade substantially if lower-quality teachers were used?\n4. In the manuscript, \"Eq. (x)\" and \"Eq. x\" are used interchangeably. Please standardize their usage throughout the manuscript.\n5. If my concerns are addressed, I will increase my score after the rebuttal.", "limitations": "See Questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The unified use of reinforcement learning and adversarial imitation learning is novel and effective.\n2. The incorporation of an LLM-based discriminator and a judge-model (LLM-as-a-Judge) enhances training robustness.\n3. Extensive evaluation across diverse vision-language benchmarks (AI2D, MathVista, MM-Vet, MMMU, BLINK, etc.) confirms the superiority of RIL-trained models compared to state-of-the-art baselines.\n4. Well-written with clear explanations of methodology, extensive experiments, ablation studies, and detailed training procedures.\n\nWeaknesses:\n1. The requirement of multiple large VLMs and LLMs for both discriminator and judge-model training might limit accessibility due to significant computational resource demands.\n2. This manuscript is well-written and presents a strong empirical contribution to efficient vision-language modeling. However, the degree of innovation might appear incremental given the existing frameworks. Please further emphasize the innovation and the problems addressed by this paper.", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The requirement of multiple large VLMs and LLMs for both discriminator and judge-model training might limit accessibility due to significant computational resource demands.**\n\nCompared to the original rule-based GRPO, we agree that RIL introduces increased computational costs during training. However, we would like to emphasize that these costs can be mitigated through efficient technical implementation strategies. While pretraining the discriminator is a necessary step and cannot be avoided from a computational standpoint, we can reduce training costs during the RIL loop by managing model weights efficiently. Specifically, we offload the weights of the small VLMs and the discriminator to the CPU when they are not in use. When needed, we load the appropriate weights from the CPU to the GPU using DeepSpeed API. This allows us to dynamically swap model weights without uploading both architecture to GPU, thereby minimizing computational overhead. This approach is feasible because we use the same model architecture for both the small VLMs and the discriminator, enabling seamless weight replacement and resource-efficient training.\n\n---\n\n**Q2. This manuscript is well-written and presents a strong empirical contribution to efficient vision-language modeling. However, the degree of innovation might appear incremental given the existing frameworks. Please further emphasize the innovation and the problems addressed by this paper.**\n\nWe would like to emphasize two key points regarding the innovation of our work:\n\n- While individual components such as Dr.GRPO and LLM-as-a-Judge have been previously explored, the overall framework we propose is novel in that it enables the distillation of knowledge from multiple large VLMs into a single, smaller VLM. This unified design allows us to leverage the strengths of various large models in a systematic and scalable manner.\n\n- Our approach is orthogonal to advancements in VLM architectures and agnostic to specific VLM series, thanks to the generalized nature of our framework. This makes it broadly applicable and future-proof as new VLMs emerge.\n\nSpecifically, RIL provides a generalization effect similar to RL and mimics the benefits of high-quality SFT, guiding smaller VLMs to learn more flexible responses beyond fixed ground-truth answers. This is because RIL basically leverages the diversity of responses generated by large VLMs, which are more likely to produce high-quality answers compared to smaller VLMs, and these generated responses are directly used to simultaneously train both the discriminator and the smaller VLMs, as described in lines 6–13 of Algorithm 2. \n\nThanks to such sophisticated framework design, the experimental results showed the significant performance improvement on multiple benchmarks with multiple VLM families. Therefore, once the new VLM is developed in the community, this approach can exploit such model to develop lighter and more accurate VLMs.\n\nWe will add this explanation to the manuscript to better clarify this important point.\n\n---\n\n**Q3. What mechanisms or hyperparameter tuning strategies are critical to maintaining training stability, particularly in adversarial imitation settings?**\n\nWe have addressed the training stability from multiple perspectives:\n\n- Structured Prompt for LLM-as-a-Judge: Please refer to our response of Reviewer SQq6's Q3.\n- Binary Answer Reward: Please refer to our response of Reviewer SQq6's Q3.\n- Binary Similarity Reward: Please refer to our response of Reviewer Hg1D’s Q4.\n- Quality and Diversity of Generated Responses from Teacher Models: Please refer to our response of Reviewer SQq6’s Q4.\n- Hyperparameter: To prevent unstable training, we conducted several ablation studies on the number of discriminator and small VLM training iterations, as well as on the KL divergence hyperparameter, as shown in Table 5(a) and (b).\n\n---\n\n**Q4. Could RIL be effectively combined with other recent model alignment methods (e.g., instruction tuning or retrieval augmentation)?**\n\nWe have already discussed RIL's limitations in the discussion section (lines 285–291) on page 9, noting that the current implementation of RIL primarily focuses on the post-instruction tuning alignment phase. Extending RIL to earlier stages, such as the visual instruction tuning phase, would be a promising direction for future research.\n\n---\n\n**Q5. How sensitive is the approach to the selection and quality of large teacher VLMs? Would performance degrade substantially if lower-quality teachers were used?**\n\nBuilding on our answer to Reviewer SQq6’s Q4, we have conducted additional experiments based on your comment regarding the use of lower-quality teachers.\n\nBoth the diversity and the quality of teacher outputs are critical to RIL’s success. As the ablation study below shows, RIL’s performance degrades when teacher responses lack diversity or accuracy. (We assume diversity is satisfied by generating a larger number of responses, since we set the inference hyperparameters—temperature, top‑p, and top‑k.)\n\nThe first table reports performance as we vary the number of pre‑generated teacher responses used to train the discriminator and drive the RIL loop:\n\n| Number of Generated Responses | MathVista | MMB  | MM-Vet | MMMU | Avg  |\n|-------------------------------|:-----------:|:------:|:--------:|:------:|:------:|\n| 0 (Dr.GRPO)                   | 69.5      | 84.3 | 73.5   | 57.2 | 71.1 |\n| 2                             | 73.0      | 85.5 | 75.0   | 60.5 | 73.5 |\n| 4                             | 76.5      | 86.0 | 78.0   | 64.3 | 76.2 |\n| 8                             | 77.0      | 86.2 | 79.0   | 68.6 | 77.7 |\n| 16                            | 79.7      | 86.3 | 80.4   | 65.7 | 78.0 |\n| 32                            | 80.0      | 86.5 | 80.3   | 65.2 | 78.0 |\n\nNote that, this result is based on Qwen2.5-VL-RIL-7B (Both).\n\nWe also evaluated the effect of introducing incorrect answers by randomly shuffling a percentage of generated responses relative to their questions (a dash “–” indicates mode collapse):\n\n| Shuffle Percent  | MathVista | MMB  | MM-Vet | MMMU | Avg  |\n|------------------|:-----------:|:------:|:--------:|:------:|:------:|\n| 0\\%              | 79.7      | 86.3 | 80.4   | 65.7 | 78.0 |\n| 10\\%             | 75.0      | 82.0 | 77.0   | 59.6 | 73.4 |\n| 30\\%             | 62.0      | 68.0 | 60.0   | 30.0 | 55.0 |\n| 50\\%             | -         | -    | -      | -    | -    |\n| 100\\%            | -         | -    | -      | -    | -    |\n\nNote that, this result is based on Qwen2.5-VL-RIL-7B (Both).\n\nAdditionally, we evaluated the impact of using downgraded versions of \"Qwen2-VL-72B\" and \"InternVL2.5-78B\" as lower-quality teachers on RIL’s performance:\n\n| Models                                               | MathVista | MMB  | MM-Vet | MMMU | Avg  |\n|------------------------------------------------------|:-----------:|:------:|:--------:|:------:|:------:|\n| Qwen2.5-VL-RIL-7B (Qwen2-VL-72B and InternVL2.5-78B) | 70.5      | 84.2 | 76.1   | 62.6 | 73.3 |\n| Qwen2.5-VL-RIL-7B (Qwen2.5-VL-72B and InternVL3-78B) | 79.7      | 86.3 | 80.4   | 65.7 | 78.0 |\n| InternVL3-RIL-8B (Qwen2-VL-72B and InternVL2.5-78B)  | 72.0      | 86.0 | 77.4   | 67.2 | 75.7 |\n| InternVL3-RIL-8B (Qwen2.5-VL-72B and InternVL3-78B)  | 77.8      | 88.1 | 80.1   | 68.6 | 78.7 |\n\n\n**Recap**: Based on these results and analyses, we argue that our approach is orthogonal to advancements in VLM families and agnostic to VLM series, due to the generalized nature of our framework. Therefore, as new VLMs are developed by the community, our approach can seamlessly incorporate them to train smaller and more accurate VLMs.\n\nIn summary,\n\n- RIL benefits substantially from both diverse and accurate teacher outputs.\n- Performance improves steadily as the number of high-quality, varied responses increases.\n- Conversely, introducing inconsistent or conflicting answers causes significant performance degradation.\n- The use of lower-quality teachers leads to noticeably reduced performance.\n\n---\n\n**Q6. In the manuscript, \"Eq. (x)\" and \"Eq. x\" are used interchangeably. Please standardize their usage throughout the manuscript.**\n\nWe have identified that \"Eq. x\" appears on lines 205 and 250. We will revise these instances to \"Eq. (x)\" to ensure consistency throughout the manuscript.\n\n---\n\nThank you for your encouraging review. We hope our responses have further clarified the points you raised."}]

Author rebuttal:

Meta-review: This paper introduces a new approach to training small VLMs via mimicking the behaviour of larger ones based on a newly proposed Reinforcement Learning with Imitation Learning framework. All reviewers agree on the novelty of the proposed, and that the paper presents some strong results on an impactful research problem. Almost all weaknesses were addressed during rebuttal and authors also promised to address issues raised by SQq6 regarding the computational demands with a detailed analysis of time and memory efficiency, Hence, clear accept.
Decision: Accept (poster)