Unified Reinforcement and Imitation Learning for Vision-Language Models
Authors: name: Byung-Kwan Lee, name: Ryo Hachiuma, name: Yong Man Ro, name: Yu-Chiang Frank Wang, name: Yueh-Hua Wu
Organization: NVIDIA

Vision-Language Models (VLMs) have achieved remarkable progress, yet their
large scale often renders them impractical for resource-constrained
environments. This paper introduces Unified Reinforcement and Imitation
Learning (RIL), a novel and efficient training algorithm designed to create
powerful, lightweight VLMs. RIL distinctively combines the strengths of
reinforcement learning with adversarial imitation learning. This enables
smaller student VLMs not only to mimic the sophisticated text generation of
large teacher models but also to systematically improve their generative
capabilities through reinforcement signals. Key to our imitation framework is
an LLM-based discriminator that adeptly distinguishes between student and
teacher outputs, complemented by guidance from multiple large teacher VLMs to
ensure diverse learning. This unified learning strategy, leveraging both
reinforcement and imitation, empowers student models to achieve significant
performance gains, making them competitive with leading closed-source VLMs.
Extensive experiments on diverse vision-language benchmarks demonstrate that
RIL significantly narrows the performance gap with state-of-the-art open- and
closed-source VLMs and, in several instances, surpasses them.