Towards Language Models That Can See: Computer Vision Through the LENS
  of Natural Language
Authors: name: William Berrios; name: Gautam Mittal; name: Tristan Thrush; name: Douwe Kiela; name: Amanpreet Singh
Published: 2023-06-28 17:57:10

Abstract: We propose LENS, a modular approach for tackling computer vision problems by
leveraging the power of large language models (LLMs). Our system uses a
language model to reason over outputs from a set of independent and highly
descriptive vision modules that provide exhaustive information about an image.
We evaluate the approach on pure computer vision settings such as zero- and
few-shot object recognition, as well as on vision and language problems. LENS
can be applied to any off-the-shelf LLM and we find that the LLMs with LENS
perform highly competitively with much bigger and much more sophisticated
systems, without any multimodal training whatsoever. We open-source our code at
https://github.com/ContextualAI/lens and provide an interactive demo.