LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact
  Language Model
Authors: name: Musashi Hinck; name: Matthew L. Olson; name: David Cobbley; name: Shao-Yen Tseng; name: Vasudev Lal
Published: 2024-03-29 21:32:50

Abstract: We train a suite of multimodal foundation models (MMFM) using the popular
LLaVA framework with the recently released Gemma family of large language
models (LLMs). Of particular interest is the 2B parameter Gemma model, which
provides opportunities to construct capable small-scale MMFMs. In line with
findings from other papers in this space, we test the effect of ablating three
design features: pretraining the connector, utilizing a more powerful image
backbone, and increasing the size of the language backbone. The resulting
models, which we call LLaVA-Gemma, exhibit moderate performance on an array of
evaluations, but fail to improve past the current comparably sized SOTA models.
Closer analysis of performance shows mixed effects; skipping pretraining tends
to reduce performance, larger vision models sometimes improve performance, and
increasing language model size has inconsistent effects. We publicly release
training recipes, code and weights for our models for the LLaVA-Gemma models.