When Words Outperform Vision: VLMs Can Self-Improve Via Text-Only
  Training For Human-Centered Decision Making
Authors: name: Zhe Hu, name: Jing Li, name: Yu Yin

Embodied decision-making is fundamental for AI agents operating in real-world
environments. While Visual Language Models (VLMs) have advanced this
capability, they still struggle with complex decisions, particularly in
human-centered situations that require deep reasoning about human needs and
values. In this study, we systematically evaluate open-sourced VLMs on
multimodal human-centered decision-making tasks. We find that LLMs receiving
only textual descriptions unexpectedly outperform their VLM counterparts of
similar scale that process actual images, suggesting that visual alignment may
hinder VLM abilities. To address this challenge, we propose a novel text-only
training approach with synthesized textual data. This method strengthens VLMs'
language components and transfers the learned abilities to multimodal
inference, eliminating the need for expensive image-text paired data.
Furthermore, we show that VLMs can achieve substantial performance gains
through self-improvement, using training data generated by their LLM
counterparts rather than relying on larger teacher models like GPT-4. Our
findings establish a more efficient and scalable approach to enhancing VLMs'
human-centered decision-making capabilities, opening new avenues for optimizing
VLMs through self-improvement mechanisms.