Generalist Multimodal Models
Yong Jae Lee
University of Wisconsin-Madison

Image of LLaVA generated by GLIGEN"a cute lava llama with glasses" + box promptComputer Vision in the Deep Learning Era(2012 - Present)

Image classification

Semantic segmentation

Object detection

Surface normal prediction
Pose recognition

3D prediction… and many moreExplosion in ...
Students!
Startups!
Funding!
Hiring!

However, Prevailing Paradigm Thus Far:“Specialist” models: single-model, single-task

Object Detection Only

Pose Recognition Only

Object Detector
1. Finetune and expand vocabulary
to indoor settings
2. Detect: printer.
3. There is no printer.

User: Can I print my documents here?

Final output to the user
OCR Engine
1. Result: BUSINESS CENTER <coords>
2. Answer: Probably?

🥴

Hmm.. I am not sure. Maybe no, maybe yes.

Specialist models are insufficientfor open-ended generic tasks
5

Rise of “Generalist” Foundation Models (2020s)• Single-model,many tasks• Large Language
Models (e.g., GPT4)• Vision Transformers• Image-Text Models(e.g., CLIP)

Image credit: https://blogs.nvidia.com/blog/what-are-foundation-models/Today’s talk:Large Generalist Multimodal Models• Generalist vision-language models that understand visual data andcommunicate in natural language• Controllable (“aligned”) models that produce desirable outputs forwide-concept knowledge• Challenge: How to effectively train such models with minimalsupervision?
• Solution: Adapt pre-trained foundation models, and design semiautomatic methods for data collection

Humans See and Reason about the Visual World;
Express and Interact with Natural Language

How to Build Generalist Multimodal Models?

• Prior methods (before early 2023) generally lacked instruction following capabilities[J. Wang et al. 2022. GIT: A Generative Image-to-text Transformer for Vision and Language][J. Li et al. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models][J.-B. Alayrac. 2022. Flamingo: a visual language model for few-shot learning]…

How to Build Generalist Multimodal Models?
• GPT-4V: Strong language and visual reasoning , but closed-source[GPT-4 Technical Report, OpenAI. March 2023.]

How to Build Generalist Multimodal Models?
Instruction
What is unusual about this image?
ImageOutput
The unusual aspect of this image is …
Language Decoder
Cross-modal Connector
Visual Encoder

1. Data to tune the model for instruction-following capabilities
2. Strong pretrained vision and language models
3. Connecting vision and language

How to Build Generalist Multimodal Models?
Instruction
What is unusual about this image?
Image

OutputThe unusual aspect of this image is …
Language Decoder
Cross-modal Connector
Visual Encoder

1. Data to tune the model for instruction-following capabilities
2. Strong pretrained vision and language models
3. Connecting vision and language

Lessons from Training Large Language Models• Step 1: Representation learning- Self-supervised pre-training

Lessons from Training Large Language Models• Step 1: Representation learning- Self-supervised pre-training• Step 2: Instruction Tuning- Supervised fine-tuning / RLHF

Lessons from Instruction Tuning in LLMs
Instruction

OutputExplain human's behavior.
Behavior: cry.

There could be many reasons why a person might cry.
They could be feeling sad, scared, angry, orfrustrated…

Recommend a movie for me.

Certainly! Without specific preferences in mind, I'll
recommend a popular and highly-rated movie: "TheShawshank Redemption." It's a classic drama film …

…

…

Lessons from Instruction Tuning in LLMs
• How to collect instruction tuning data?
• Human: high-quality, hand-written by humans à high cost
• Machine: strong LLM-based teacher like ChatGPT à affordable cost
Manual seed instruction-output pairs
Explain human's behavior.

<reference answer>

Recommend a movie for me.

<reference answer>

Please generate new instruction-output pairs that meet the following requirements: …
<new instruction 1><new output 1>…

LLaMa à Alpaca, Vicuna, etc.
[Yizhong Wang. 2022. Self-instruct: Aligning language model with self generated instructions.]

Instruction Tuning in Large Multimodal ModelsInstruction
What is unusual about this image?
ImageOutput
The unusual aspect of this image is …
Language Decoder
Cross-modal Connector
Visual Encoder• How to obtain Image-Instruction-Output triplet training data?
• Can we use a large language model for this?

Text-only GPT Assisted Visual Instruction Data CreationImage• How do we get a text-only LLM to “see” animage?
à Let it read context information written in text
Context (caption)

Context (layout)

A group of people standing
outside of a black vehicle
with various luggage.

MS-COCO
Each image associated with:• 5 captions• Object categories / bounding boxes
person: [0.68, 0.24, 0.77, 0.69], person: [0.63, 0.22, 0.68, 0.51],
person: [0.44, 0.23, 0.48, 0.34], backpack: [0.38, 0.69, 0.48, 0.91],….

[T.-Y. Lin et al. 2014. Microsoft coco: Common objects in context.]

Text-only GPT Assisted Visual Instruction Data Creation
Context

Instruction

OutputContext (caption)

A group of people standing
outside of a black vehicle
with various luggage.

What are the challenges
these people might befacing?

Context (layout)

person: [0.68, 0.24, 0.77,0.69, backpack: [0.38, 0.69,0.48, 0.91] …

They may be having difficulty fitting
all luggage into the back of the SUV.
There are many bags, suitcases
already in the back, while more…

Manual seed example(s) of context-instruction-output triplets.

<new context (caption) 1><new context (layout) 1>
Please generate new Context-Instruction-Output triplets that meet the following requirements: …
Text-only ChatGPT
Visual Instruction-following Data:
Triplet (image, instruction, output)<new instruction 1><new output 1>

What are the two people
doing?

The two people are talking in frontof a whiteboard about math …

Text-only GPT Assisted Visual Instruction Data Creation

LLaVA-Instruct-158K
Conversation: 58K
Detailed description: 23K
Complex reasoning: 77K

LLaVA: Large Language-and-Vision AssistantVision Encoder: CLIP-ViT-L/14
Projection:
Linear layer (MLP in LLaVA-1.5)
Language Model: Vicuna, LLaMA-2-Chat, MPT-Chat, etc.
[H. Liu*, C. Li*, Q. Wu, Y. J. Lee. NeurIPS 2023. Visual Instruction Tuning. https://llava-vl.github.io]

Stage 1: Pre-training for Feature Alignment

Data: Creative Captions (CC3M) data subset of 595K image-text pairs[H. Liu*, C. Li*, Q. Wu, Y. J. Lee. NeurIPS 2023. Visual Instruction Tuning. https://llava-vl.github.io]

Stage 2: End-to-end Visual Instruction Tuning

Data: LLaVA-Instruct-158K for open-ended user-oriented visual instruction following tasks[H. Liu*, C. Li*, Q. Wu, Y. J. Lee. NeurIPS 2023. Visual Instruction Tuning. https://llava-vl.github.io]

Strong Visual Reasoning
Capability
Example: Extreme Ironing

Strong Visual Reasoning
Capability
Example: Parodied Mona Lisa

Extensions: LLaVA-1.5• Stronger performance on
visual understanding
benchmarks• Better OCR, Yes/No
answering, etc., due toscaling up data, model,
image resolution“Improved Baselines with Visual Instruction Tuning” Haotian Liu, Chunyuan Li, Yuheng Li, Yong Jae Lee. CVPR 2024 highlightExtensions: LLaVA-NeXT (1.6)• Significantly outperforms LLaVA-1.5-13B• Matched Gemini Pro (in Jan 2024) on selected benchmarks
Model

MMMU (val)

MMMU (test)

MathVista

MMBench-EN

MMBench-CN

MM-VetGPT-4V

56.8

55.7

49.9

75.8

73.9

67.6

Gemini Ultra

59.4-53---

Gemini Pro

47.9

-

45.2

73.6

74.3

64.3

LLaVA-1.5-13B

36.4

33.6

27.6

67.8

63.3

36.3

LLaVA-1.6-34B

51.1

45.3

46.5

79.3

79.0

57.4

“LLaVA-NeXT: Improved reasoning, OCR, and world knowledge” Haotian Liu et al. January 2024 (blog)

Combinatorial Task Generalization
Seen Training Data
Multilingual Text-only
Conversation

Generalized Capabilities
English-Only
Visual Conversations

Multilingual
Visual Conversations

Emergent Multilingual
Capability
Example: French Quarter

Translation: What is the name ofthis area? Please describe briefly.

Combinatorial Task Generalization
Seen Training Data

Generalized Capabilities

Multilingual Text-only
Conversation

English-Only
Visual Conversations

Multilingual
Visual Conversations

VQA/OCR data

Visual Conversations

Improved Visual Groundedness / OCR
in Visual Conversations

Longer Writing
Text-only Conversations

Shorter (casual)
Visual Conversations

Improved Writing inVisual Conversations

Do not need to create all combinations of data in training; let LMMs generalize!

Community Efforts on LMMs

LLaVA

2023

Community Efforts on LMMs

LLaVA

2023

LLaVA that Understands Visual Prompts?
• Existing Large Multimodal Models (LMMs) do a good job in whole
image understanding• Can we make them also understand regions / visual prompts (e.g.,arrows, scribbles)?

ViP-LLaVA: Understanding Freeform Visual Prompts
Input: Visual Prompt

Input: Text Prompt

Large
Multimodal
Model

Output:Text Generation: The person markedwith the red arrow isholding a green flag.
This flag is used for ...

: What is the person marked
with the red arrow holding?

“Making Large Multimodal Models Understand Arbitrary Visual Prompts” Mu Cai, et al. CVPR 2024

Prior/Concurrent Work• Text to denote box coordinates e.g. Shikra, MiniGPT-v2, Ferret• Region of Interest feature e.g. GPT4ROI• Learned vocabularies or positional embedding e.g. GPT4ROI, Kosmos-2
Input: Image
Output:Text Generation: This person isholding a snowboard.
Snowboards are used
for …

Large
Can’t understand freeform visual prompts such asMultimodalscribblesorarrows!
Input: Text Prompt
Model: What is the person
holding?
(A) within [100, 200, 150, 310]

(B)(C) [pos_10, pos_20, pos_15, pos_31]

ViP-LLaVA: Understanding Freeform Visual Prompts+Separate Images=Overlay• Super simple!
• A portion of CLIP pretraining data has visual prompt annotations[Shtedritski et al. "What does CLIP know about a red circle? Visual prompt engineering for VLMs." ICCV 2023]

ViP-LLaVA: Architecture“Making Large Multimodal Models Understand Arbitrary Visual Prompts” Mu Cai, et al. CVPR 2024

Which Visual Prompts Can ViP-LLaVA Support?

•Leverage region level visual recognition and reasoning datasets(RefCOCOg, Visual7W, PointQA-LookTwice, VCR) - 520k images

Quantitative Evaluation
Table 1: Object recognition
in Visual7W

Table 2: Object counting
in PointQA-LookTwice

Table 3: Visual Reasoning
in VCR

Table 4: Region Captioning
in RefCOCOg• SoTA results on recognition, counting, captioning, andcommonsense reasoning tasks“Making Large Multimodal Models Understand Arbitrary Visual Prompts” Mu Cai, et al. CVPR 2024

Visual Prompt Understanding Benchmark●Q: Which one has the lowest price, the fruit within thered rectangle, within the blue rectangle, or within thegreen rectangle? Respond with the name of the fruit.
GT: Orange.
Required capabilities: Recognition, OCR, Math

303 questions:
Recognition, OCR(Optical CharacterRecognition),
Knowledge, Math,
Object relationshipreasoning, Language
generation

Yo’LLaVA: Your Personalized LMM
Can you see if <thao> is in this photo?
User<thao> is in the photo, and she is standing
next to another person. They both appear tobe having a good time, possibly at a party orevent. <thao> is smiling and making a peace
sign with her hand, which suggests a positiveand fun atmosphere.
I’m Yo’LLaVA
Your Personalized
LLaVA!
“Yo’LLaVA: Your Personalized Language & Vision Assistant” Thao Nguyen et al. arXiv 2024

Matryosha Multimodal Models• Adaptively control visual token length (i.e. information granularity) in LLaVA“Matryoshka Multimodal Models” Mu Cai et al. arXiv 2024

Inspired by:
Matryosha Representation
Learning

Looking Forward: Is Visual Understanding Solved?
Not quite …

Check out: “CounterCurate”,
J. Zhang*, M. Cai*, et al. ACLFindings 2024.

Looking Forward: Limitations of Current Models• Capabilities• Hallucinations• Video understanding• Smaller performant models• Multimodal agents•…

• Understanding• Origination of emergent behaviors like OCR• How does the performance of LLMs affect the capability of the LMMs
• Impact of instruction tuning on knowledge•…

Looking Forward: Multimodal AI Agents• AI Agents that can self reflect, use tools, plan, and collaborate with other agents
Image Source: The Rise and Potential of Large Language Model Based Agents: A Survey, Xi et al. 2023

Visual Prompting for Robot Learning

“PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs” Google DeepMind. ICML 2024

“PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs” Google DeepMind. ICML 2024

Conclusions• From specialist to generalist multi-modal models• Controllable (“aligned”) image understanding for open-world concepts• Build upon pre-trained foundation models, design semi-automatic datacollection methods• Code, models, online demo available:
https://llava-vl.github.io/, https://vip-llava.github.io/Thank you• Haotian Liu, Yuheng Li, Utkarsh Ojha, Mu Cai, Thao Nguyen, Chunyuan
Li, Jianwei Yang, Jianfeng Gao , Siva Mustikovela, Greg Meyer, Yuning
Chai, Dennis Park

Utkarsh Ojha

Mu CaiThao Nguyen