Towards Data-Centric Multimodal ML:
Vision-Language Dataset Distillation

Xindi Wu

Department of Computer Science
Princeton University
https://xindiwu.github.io/
xindiw@princeton.edu

Overview
1

Motivation

2

Previous Work

3

Method4

Experiments

5Conclusion

| Motivation1

| Motivation
What is Data-centric Why is dataset
Multimodal ML?
distillation important?

1

What is Multimodal?
A dictionary definition:
Multimodal: with multiple modalities

A research-oriented definition:
Multimodal is the science of heterogeneous and interconnected data

Liang, Zadeh, and Morency. Tutorial on Multimodal Machine Learning. CVPR 2022, NAACL 2022

2

Data is the cornerstone of innovation in multimodal ML

Flickr30K (2014)
COCO (2014)

Visual Genome (2016)

LAION 400M/5B (2021)

Conceptual Captions (2018)

DataComp (2023)

3

Data-Centric Multimodal ML

Flickr30K (2014)
COCO (2014)

Visual Genome (2016)

LAION 400M/5B (2021)

Conceptual Captions (2018)

DataComp (2023)

4

Data-Centric Multimodal ML

Data = Information + Noise

5
John Wright and Yi Ma. High-dimensional data analysis with low-dimensional models: Principles, computation, and applications. Cambridge University Press, 2022.

Data-Centric Multimodal ML

Data = Information + Noise

How can we identify the most critical information from datasets?

5

Data-Centric Multimodal ML

Data = Information + Noise

How can we identify the most critical information from datasets?

Dataset distillation is one promising solution!

5

What is Dataset Distillation?
CIFAR10 examples
N images
plane
car
bird
cat
deer
dog
frog
horse
ship
truck

CIFAR10(5000 images/class)6

What is Dataset Distillation?
CIFAR10 examples
N images
plane
car
bird
cat
deer

N >> M

dog
frog
horse
ship
truck

CIFAR10(5000 images/class)6

What is Dataset Distillation?
CIFAR10 examples
N images

M images

plane
car
bird
cat
deer

N >> M

dog
frog
horse
ship
truck

CIFAR10(5000 images/class)Distilled
CIFAR10(1 images/class)6

What is Dataset Distillation?
CIFAR10 examples
N images

M images

plane
car
bird
cat
deer

N >> M

dog
frog
horse
ship
truck

CIFAR10(5000 images/class)Distilled
CIFAR10(1 images/class)7

What is Dataset Distillation?
CIFAR10 examples
dog
CIFAR10(5000 images/class)Distilled
CIFAR10(1 images/class)8

What is Dataset Distillation?
CIFAR10 examples

Distilled CIFAR10 (dog)(10 images/class)9

What is Dataset Distillation?
Goal
Distilled
DatasetDatasetDistillation

N >> M

Similar Test
Performance

Original
DatasetN imagesTrain

M images
Train

Figure credit Yu, Ruonan, Songhua Liu, and Xinchao Wang. "Dataset distillation: A comprehensive review." arXiv preprint arXiv:2301.07014 (2023).

10What is Dataset Distillation?
Goal
Distilled
DatasetDatasetDistillation

N >> M

Similar Test
Performance

Original
DatasetN imagesTrain

M images
Train

Figure credit Yu, Ruonan, Songhua Liu, and Xinchao Wang. "Dataset distillation: A comprehensive review." arXiv preprint arXiv:2301.07014 (2023).

10What is Dataset Distillation?
Defination(Loose) Definition[1]:Approaches that aim to synthesize tiny and high-fidelity data summaries which
distill the most important knowledge from a given target dataset.

[1] Sachdeva, Noveen, and Julian McAuley. "Data distillation: A survey." TMLR, 2023.

11

What is Dataset Distillation?
Defination(Loose) Definition[1]:Approaches that aim to synthesize tiny and high-fidelity data summaries which
distill the most important knowledge from a given target dataset.
Such distilled summaries are optimized to serve as effective drop-in replacementsof the original dataset for efficient and accurate data-usage applications.

[1] Sachdeva, Noveen, and Julian McAuley. "Data distillation: A survey." TMLR, 2023.

12

What is Dataset Distillation?
Application

NeuralArchitecture
SearchFederated
Learning

Continual
Learning

Differential
Privacy13

| Previous Work

14

Previous Work
Image Classification

plane
car
bird
cat
deer

N >> M

dog
frog
horse
ship
truck

CIFAR10(5000 images/class)Distilled
CIFAR10(1 images/class)15

Previous WorkT﻿axonomy

Figure credit: Sachdeva, Noveen, and Julian McAuley. "Data distillation: A survey." TMLR, 2023.

16

Previous Work
Meta-Model Matching Framework

Dataset distillation can be formulated as a bi-level meta-learning problem

Large Scale Dataset
Small Synthetic Dataset

17

Previous Work
Meta-Model Matching Framework

Dataset distillation can be formulated as a bi-level meta-learning problem

Large Scale Dataset
Small Synthetic Dataset
Model
Generalization Loss18

Previous Work
Meta-Model Matching Framework

Dataset distillation can be formulated as a bi-level meta-learning problem

Inner level

19

Previous Work
Meta-Model Matching Framework

Dataset distillation can be formulated as a bi-level meta-learning problem

Inner level
Outer level

19

Previous Work
Meta-Model Matching Framework

Dataset distillation can be formulated as a bi-level meta-learning problem

Inner level
Outer level

20

Previous Work
Trajectory Matching Framework

Cazenavette, George, et al. "Dataset distillation by matching training trajectories." CVPR. 2022.

21

Comparison

22Comparison

23Comparison
What’s unique about Vision-Language Dataset?
image

text
A dog wearing a green sweater and fannypack walks on a snow-covered field.
A small dog wearing a green sweaterand a backpack walks through snow.
A dog wearing a green sweater and abackpack walking on snow.
A dog wearing a green sweater andbackpack running in the snow.
A small dog wearing a sweater walking inthe snow.
24Comparison
What’s unique about Vision-Language Dataset?
image

text
A dog wearing a green sweater and fannypack walks on a snow-covered field.
A small dog wearing a green sweaterand a backpack walks through snow.
A dog wearing a green sweater and abackpack walking on snow.
A dog wearing a green sweater andbackpack running in the snow.
A small dog wearing a sweater walking inthe snow.
24Comparison
What’s unique about Vision-Language Dataset?
image

text
A dog wearing a green sweater and fannypack wa﻿lks on a snow-covered field.
A small dog wearing a green sweaterand a backpack walks through snow.
A dog wearing a green sweater and abackpack walking on snow.
A dog wearing a green sweater andbackpack running in the snow.
A small dog wearing a sweater walking inthe snow.
24Comparison
What’s unique about Vision-Language Dataset?
image

text
A dog wearing a green sweater and fannypack walks on a snow-covered field.
A small dog wearing a green sweaterand a backpack walks through snow.
A dog wearing a green sweater and abackpack walking on snow.
A dog wearing a green sweater andbackpack running in the snow.
A small dog wearing a sweater walking inthe snow.
24Comparison
What’s unique about Vision-Language Dataset?
image

text
A dog wearing a green sweater and fannypack walks on a snow-covered field.
A small dog wearing a green sweaterand a backpack walks through snow.
A dog wearing a green sweater and abackpack walking on snow.
A dog wearing a green sweater andbackpack running in the snow.
A small dog wearing a sweater walking inthe snow.
24Comparison
What’s unique about Vision-Language Dataset?
image

text
A dog wearing a green sweater and fannypack walks on a snow-covered field.
A small dog wearing a green sweaterand a backpack walks through snow.
A dog wearing a green sweater and abackpack walking on snow.
A dog wearing a green sweater andbackpack running in the sno﻿w.
A small dog wearing a sweater walking inthe snow.
24Comparison
image

text
A dog wearing a green sweater and fannypack walks on a snow-covered field.

A small dog wearing a green sweater
andthea backpack
What are the common approaches to selectmost walks through snow.

critical information in the vision-language
dataset?
A dog
wearing a green sweater and abackpack walking on snow.

A dog wearing a green sweater andbackpack running in the snow.
A small dog wearing a sweater walking inthe snow.
25

Comparison
image

text
A dog wearing a green sweater and fannypack walks on a snow-covered field.

A small dog wearing a green sweater
andthea backpack
What are the common approaches to selectmost walks through snow.

critical information in the vision-language
dataset?
A dog
wearing a green sweater and abackpack walking on snow.

A dog wearing a green sweater andbackpack running in the snow.
Baseline: Coreset Selection
A small dog wearing a sweater walking inthe snow.
25

Baseline
Coreset Selection26Baseline
Coreset Selection

Herding

Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 1121–1128, 2009.

27

Baseline
Coreset Selection

Herding
Dataset center
Coreset center

Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 1121–1128, 2009.

27

Baseline
Coreset Selection

Herding
Dataset center
Coreset center

Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 1121–1128, 2009.

27

Baseline
Coreset Selection

Herding

K-center
Dataset center
Coreset center

Until select K
samples

Sener, Ozan, and Silvio Savarese. "Active learning for convolutional neural networks: A core-set approach." arXiv preprint arXiv:1708.00489 (2017).
Reza Zanjirani Farahani and Masoud Hekmatfar. Facility location: concepts, models, algorithms and case studies. Springer Science & Business Media, 2009.

28

Baseline
Coreset Selection

Herding

Forgetting

K-center
Dataset center
Coreset center

Until select K
samples

Forgetting event:
When the model correctly predicts
the example in one epoch but fails
in the next.

Track these forgetting events during
training and identify the ones withthe least forgetting events.

Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example
forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159, 2018.

29

Our Task
Large Vision-Language Dataset

N pairs of image + text30

Our Task
Large Vision-Language Dataset

N pairs of image + text

30K pairs

Flickr30K30

Our Task
Large Vision-Language Dataset

N pairs of image + text

30K pairs

Flickr30K
N >> M
Small Synthetic Dataset

M pairs of image + text30

Our Task
Large Vision-Language Dataset

N pairs of image + text

30K pairs

Flickr30K
N >> M
Small Synthetic Dataset

100 pairs

M pairs of image + text30

Challenges
Vision-Language Dataset Distillation

Complex Cross-Modal
Relationships

Discrete text
optimization issue

Heavy computational cost31

| Method32

Method
Pipeline Overview

33

Method
Problem Formulation

34

Method
Problem Formulation

Large Scale Dataset
Small Synthetic Dataset

35

Method

How do we handle text optimization?

36

Method
How do we handling text optimization?

We initialize the continuous sentence
embeddings using a pretrained BERT modeland update the distilled text in the continuous
embedding space.

37

Method
Problem Formulation

How do we encode the image and text features?

38

Method
Problem Formulation

39

Method
Problem Formulation

40

Method
Problem Formulation

41

Method
Problem Formulation

42

Method
Bi-Trajectory Guided Vision-Language Co-Distillation

Concretely, our approach consists of two stages:
Stage 1 Expert training
Training multiple models for epochs on the full dataset
Obtaining the expert training trajectories, with each trajectory

For our multimodal setting, the models are trained with contrastive loss.
43

Method
Bi-Trajectory Guided Vision-Language Co-Distillation

Concretely, our approach consists of two stages:
Stage 1 Expert training

44

Method
Bi-Trajectory Guided Vision-Language Co-Distillation

Concretely, our approach consists of two stages:
Stage 2 Distillation

Training a set of student models on the current distilled dataset
using the same contrastive loss.
Updatebased on the bi-trajectory matching loss of the student models’parameter trajectories and the expert trajectories.
45

Method
Bi-Trajectory Guided Vision-Language Co-Distillation

Bi-trajectory matching loss:

46

Method
Bi-Trajectory Guided Vision-Language Co-Distillation

Bi-trajectory matching loss:

47

Method
Bi-Trajectory Guided Vision-Language Co-Distillation

Bi-trajectory matching loss:

48

Method
Bi-Trajectory Guided Vision-Language Co-Distillation

Bi-trajectory matching loss:

49

Method
Bi-Trajectory Guided Vision-Language Co-Distillation

Concretely, our approach consists of two stages:
Stage 2 Distillation

50

Method
Low-Rank Adaptation Matching

Vision Transformers (ViTs)

1. High dimensionality of the embeddings
2. Large number of parameters

51

Method
Low-Rank Adaptation Matching

Consider a linear layer withm input units, n output units
The weight matrix for this layer has dimensions m x n

Y=WX

Output (y)Pretrained
Weights (w)Input (x)52Method
Low-Rank Adaptation Matching

Consider a linear layer withm input units, n output units
The weight matrix for this layer has dimensions m x n

Y=WX
m = 800n = 3200
W: 800 x 3200 = 2,560,000 weights.

Output (y)Pretrained
Weights (w)Input (x)53Method
Low-Rank Adaptation Matching
Low-Rank Adaptation

Low-Rank Adaptation (LoRA)
We keep W fixed and introduce two matrices,
A and B

Output (y)Output (y)Pretrained
Weights (w)Y = W X + A*B X
Input (x)Input (x)54

Method
Low-Rank Adaptation Matching
Low-Rank Adaptation

Low-Rank Adaptation (LoRA)
We keep W fixed and introduce two matrices,
A and B

Output (y)Output (y)Pretrained
Weights (w)Y = W X + A*B X
Input (x)Input (x)Matrix A has a shape of 800 x r, and matrix B has a shape of r x 3200.
m = 800, n = 3200, r = 1(800 x 1) + (1 x 3200) = 4000

55

Method
Low-Rank Adaptation Matching
Low-Rank Adaptation

Low-Rank Adaptation (LoRA) Matching

Output (y)LoRA matching optimizes the trajectories of low
rank adapters instead of the full parameters.

Input (x)56

Method
Low-Rank Adaptation Matching
Low-Rank Adaptation

Low-Rank Adaptation (LoRA) Matching

Output (y)LoRA matching optimizes the trajectories of low
rank adapters instead of the full parameters.

VIT_base
86 million

Input (x)57

Method
Low-Rank Adaptation Matching
Low-Rank Adaptation

Low-Rank Adaptation (LoRA) Matching

Output (y)LoRA matching optimizes the trajectories of low
rank adapters instead of the full parameters.

VIT_base

LoRA_VIT_base

86 million

18 million
r=4

Input (x)58

Method
Low-Rank Adaptation Matching
Low-Rank Adaptation

Low-Rank Adaptation (LoRA) Matching

Output (y)LoRA matching
optimizes the
trajectories
of low parameters.
Efficientmodel adaptation
withminimal extra
rankadapters
insteadofthefullparameters.
Focus on optimizing a smaller parameter set.
Efficiently optimize trajectory loss during distillation.
VIT_base

LoRA_VIT_base

86 million

18 million
r=4

Input (x)59Challenges <> Solutions
Vision-Language Dataset Distillation

Complex Cross-Modal
Relationships

Discrete text
optimization issue

Heavy computational cost60

Challenges <> Solutions
Vision-Language Dataset Distillation

Complex Cross-Modal
Relationships

Discrete text
optimization issue

Heavy computational cost

Bi-trajectory Matchingwith Contrastive Loss

60

Challenges <> Solutions
Vision-Language Dataset Distillation

Complex Cross-Modal
Relationships

Discrete text
optimization issue

Bi-trajectory Matchingwith Contrastive Loss

Continuous Sentence
Embeddings

Heavy computational cost60

Challenges <> Solutions
Vision-Language Dataset Distillation

Complex Cross-Modal
Relationships

Discrete text
optimization issue

Heavy computational cost

Bi-trajectory Matchingwith Contrastive Loss

Continuous Sentence
Embeddings

Finetuning with LowRank Adaptation60

| Experiments

61

Experiments
Evaluation Setting

DatasetFlickr30K

A small dog wearing asweater walking in the snow.

....

30K

COCO

a grey dog seated ona chair of a vehicle

....

328K
62

Experiments
Evaluation Setting

DatasetFlickr30K

Task
COCO

Image-text retrieval
image-to-text retrieval (TR) &text-to-image retrieval (IR)A small dog wearing asweater walking in the snow.

....

30K

a grey dog seated ona chair of a vehicle

....

328K

R@K (with K = 1, 5, 10)
compute the fraction of times thecorrect result is found among the
top K items.
63

Experiments
Evaluation Setting

DatasetFlickr30K

Task
COCO

Image-text retrieval
image-to-text retrieval (TR) &text-to-image retrieval (IR)A small dog wearing asweater walking in the snow.

....

30K

a grey dog seated ona chair of a vehicle

....

328K

R@K (with K = 1, 5, 10)

Model
Image encoder:
NFNet
ViT
Language encoder:
BERT

compute the fraction of times thecorrect result is found among the
top K items.
64Experiments
Quantitative Results | Baseline ComparisonsRandom selection of training examples (R) | Herding (H) K-center (K) | Forgetting (F)65Experiments
Quantitative Results | Baseline ComparisonsRandom selection of training examples (R) | Herding (H) K-center (K) | Forgetting (F)65Experiments
Quantitative Results | Baseline ComparisonsRandom selection of training examples (R) | Herding (H) K-center (K) | Forgetting (F)65Experiments
Qualitative Results | Distilled Examples

a man in a black wet suitis surfing a huge wave inthe beautiful blue watera man surfs over a hugewave in the ocean

before the distillation

after 2000 distillation steps

Note that the texts visualized here are nearest sentence decodingsin the training set corresponding to the distilled text embeddings.

66

Experiments
Qualitative Results | Before and After Distillation

67

Experiments
Qualitative Results | Before and After Distillation

68Experiments
Ablation Studies

With and without LoRA on ViT(Here we only reports R@1, more details are in the paper.)69

Experiments
Ablation Studies

With and without LoRA on ViT(Here we only reports R@1, more details are in the paper.)69

Experiments
Ablation Studies

With and without LoRA on ViT

Different vis./lan. encoders(Here we only reports R@1, more details are in the paper.)70

Experiments
Ablation Studies

With and without LoRA on ViT

Different vis./lan. encoders(Here we only reports R@1, more details are in the paper.)70

Experiments
Ablation Studies

With and without LoRA on ViT
Single-modality vs. multi-modality
Different vis./lan. encoders

Takeaway:
Distillation would be impossible if
we solely optimize one modality

T: text-only, I: image-only(Here we only reports R@1, more details are in the paper.)71

Experiments
Ablation Studies

With and without LoRA on ViT
Single-modality vs. multi-modality

Different vis./lan. encoders

Takeaway:
Distillation would be impossible if
we solely optimize one modality

T: text-only, I: image-only(Here we only reports R@1, more details are in the paper.)71

Experiments
Ablation Studies

With and without LoRA on ViT
Single-modality vs. multi-modality

Different vis./lan. encoders

Image-Text Pair Initialization

Takeaway:
Takeaway:
Distillation would be impossible if
Initializing texts from scratch
we solely optimize one modality
Initializing images from scratch

T: text-only, I: image-only(Here we only reports R@1, more details are in the paper.)72

Experiments
Ablation Studies

With and without LoRA on ViT
Single-modality vs. multi-modality

Different vis./lan. encoders

Image-Text Pair Initialization

Takeaway:
Takeaway:
Distillation would be impossible if
Initializing texts from scratch
we solely optimize one modality
Initializing images from scratch

T: text-only, I: image-only(Here we only reports R@1, more details are in the paper.)72

Experiments
Ablation Studies

With and without LoRA on ViT
Single-modality vs. multi-modality

Different vis./lan. encoders

Image-Text Pair Initialization

Takeaway:
Takeaway:
Distillation would be impossible if
Initializing texts from scratch
we solely optimize one modality
Initializing images from scratch
image component plays a morecritical role in the distilled dataset.

T: text-only, I: image-only(Here we only reports R@1, more details are in the paper.)72

| Conclusion

73SummaryIn this work, we propose the first vision-language dataset distillation
method.
Our experiments show that co-distilling different modalities via bitrajectory matching holds promise.
We hope that the insights we gathered can serve as roadmap for futurestudies exploring more complex settings.

Byron Zhang

Zhiwei Deng

Olga Russakovsky

Website

arXiv

Code
74

Looking Forward

Item 3
33.3%

item 1
33.3%

Scale-driven capabilities

Optimal data
Data-centric
Multimodal ML

Item 2
33.3%

Fundamental
scaling and limits

75

Looking Forward
Optimal data

Item 3
33.3%

item 1
33.3% What subsets of the data are most

Scale-driven capabilities

Data-centric
Multimodal ML

Item 2
33.3%

Fundamental
scaling and limits

important for the capabilities offoundation multimodal models?
What kind of synthetic multimodal data
we can build?

76

Looking Forward
Optimal data

Item 3
33.3%

item 1
33.3% What subsets of the data are most

Scale-driven capabilities

Data-centric
Multimodal ML

Fundamental
scaling and limits

Item 2
33.3%

important for the capabilities offoundation multimodal models?
What kind of synthetic multimodal data
we can build?

Given a fixed compute budget, is it betterto increase the multimodal model size or
the multimodal dataset size?

77

Looking Forward
Scale-driven capabilities

Optimal data

Item 3a more rigorous understanding of33.3%

what increasing the scale does tothe multimodal training procedureand how these desirable
emergent capabilities come about

item 1
33.3% What subsets of the data are most

Data-centric
Multimodal ML

Fundamental
scaling and limits

Item 2
33.3%

important for the capabilities offoundation multimodal models?
What kind of synthetic multimodal data
we can build?

Given a fixed compute budget, is it betterto increase the multimodal model size or
the multimodal dataset size?

78

Looking Forward
Scale-driven capabilities

Optimal data

Item 3a more rigorous understanding of33.3%

what increasing the scale does tothe multimodal training procedureand how these desirable
emergent capabilities come about

item 1
33.3% What subsets of the data are most

Data-centric
Multimodal ML

Fundamental
scaling and limits

Item 2
33.3%

important for the capabilities offoundation multimodal models?
What kind of synthetic multimodal data
we can build?

Given a fixed compute budget, is it betterto increase the multimodal model size or
the multimodal dataset size?

79

Item 3
33.3%

item 1
33.3%

Data-centric
Multimodal ML

Summary

Item 2
33.3%

In this work, we propose the first vision-language dataset distillation
method.
Our experiments show that co-distilling different modalities via bitrajectory matching holds promise.
We hope that the insights we gathered can serve as roadmap for futurestudies exploring more complex settings.

Byron Zhang

Zhiwei Deng

Olga Russakovsky

Website

arXiv

Code
80