GLOV: Guided Large Language Models as Implicit Optimizers for Vision
  Language Models
Authors: name: M. Jehanzeb Mirza, name: Mengjie Zhao, name: Zhuoyuan Mao, name: Sivan Doveh, name: Wei Lin, name: Paul Gavrikov, name: Michael Dorkenwald, name: Shiqi Yang, name: Saurav Jha, name: Hiromi Wakaki, name: Yuki Mitsufuji, name: Horst Possegger, name: Rogerio Feris, name: Leonid Karlinsky, name: James Glass

In this work, we propose a novel method (GLOV) enabling Large Language Models
(LLMs) to act as implicit Optimizers for Vision-Langugage Models (VLMs) to
enhance downstream vision tasks. Our GLOV meta-prompts an LLM with the
downstream task description, querying it for suitable VLM prompts (e.g., for
zero-shot classification with CLIP). These prompts are ranked according to a
purity measure obtained through a fitness function. In each respective
optimization step, the ranked prompts are fed as in-context examples (with
their accuracies) to equip the LLM with the knowledge of the type of text
prompts preferred by the downstream VLM. Furthermore, we also explicitly steer
the LLM generation process in each optimization step by specifically adding an
offset difference vector of the embeddings from the positive and negative
solutions found by the LLM, in previous optimization steps, to the intermediate
layer of the network for the next generation step. This offset vector steers
the LLM generation toward the type of language preferred by the downstream VLM,
resulting in enhanced performance on the downstream vision tasks. We
comprehensively evaluate our GLOV on 16 diverse datasets using two families of
VLMs, i.e., dual-encoder (e.g., CLIP) and encoder-decoder (e.g., LLaVa) models
-- showing that the discovered solutions can enhance the recognition
performance by up to 15.0% and 57.5% (3.8% and 21.6% on average) for these
models.