Towards “Data-ef icient”
Machine Learning Systems
Noveen Sachdeva@noveens97

UC San DiegofFinal Defense | UC San Diego | May ‘24

A Few Examples of Successful ML Systems
Generative Media

Recommender Systems

2

Self-Driving Cars

Typical ML Training Recipe
Excluding Many Secret Sauces

Generative Media
Training Data

Recommender Systems
Machine Learning Model3

Self-Driving Cars
Deploy! 💰Typical Recipes for Success
Machine Learning Model

Image credit: https://medium.com/@harishdatalab/unveiling-the-power-of-large-language-models-llms-e235c4eba8a94

Typical Recipes for SuccessTraining Data5

Question: Is more data really needed for training better models?
Routinely over-heard at big-tech:6

This Dissertation

ffi

Data Eciency

This Dissertation
Why Data Eciency?

Save money to train

Save time to train

Less CO2 emissions due to training

ffi

More accurate models

This Dissertation
How to be Data-E

cient?

Data Sampling

Original Dataset

Data Summary

Data Synthesis

ffi

Original Dataset

Data Summary

This Dissertation
Outline

Chapter - I: Data SamplingPart-I: Recommender
Systems
Original Dataset

Part-II: LLM
Pre-Training
Data Summary

Chapter - II: Data Synthesis
Part-I: Recommender
Systems
Original Dataset

Part-II:Auto-Regressive Data
Data Summary

On Sampling Collaborative Filtering Datasets
Noveen Sachdeva 1

Carole-Jean Wu 2

University of California, San Diego 1

11Julian McAuley 1
Meta AI 2

ScopeObjective

Recommender Systems

Infer the Ranking of N-different Recommendation Models

11Recommendation
Algorithms

1111Users

11R1

1111R2Dataset
DR31R4Items

R5…

Movies, Ads, Songs …

Ranking
User / Item
Popularity

R4R2R1R5( Head )…

( Tail / Niche )12Objective
Naive vs. Data-Ef cient

Recommendation
Algorithms
R1

R2
R4Dataset
DR3

Naive:

1. Train all candidate algorithms on the entire dataset
2. Evaluate all algorithms
3. Measure the ranking of all algorithms

R5…

Data-Ef cient:

R4
R2
R1
R5…

13fi

EFFICIENT 🥳

1. Train all candidate algorithms on a smaller sampleof the dataset
2. Evaluate all algorithms
3. Measure the ranking of all algorithms

Rankingfi

EXPENSIVE 💰SVP-CF

Down-sampling Recommendation DataPremise: Easy parts of a dataset are most likely easy for all recommendation algorithms.
Hence, removing such easy segments of data is unlikely to affect the relative ordering of algorithms.

Rec. Dataset
DSVP- CF

Train an inexpensive
proxy modelδ : (u, i) → ℝ

user1, item1, rel1
user1, item2, rel2

Down-sampling Recommendation Data

user2, item3, rel3
user3, item1, rel4…

Mark the “hardness”
of each interaction in D∇(u, i, r) =∑epochRobust framework:∇(u, i, r) =∑epoch(δ(u, i) − r)

2Explicitfeedback− AUC (δ(u, i), r) Implicit/Seq.
feedback• Uses a proxy model to tag the overall hardness of eachuser-item interaction• Can ef ciently handle various recommendation

Sample the hardest p% of:scenarios, e.g., explicit, implicit, sequential, etc.

(1) Interactions, i.e., highest ∇(u, i, r)(2) Users, i.e., highest ∇(u)(3) Items, i.e., highest ∇(i).

Rec. Dataset
Dp• Can sample across a variety of data axes: interactions,users, items, or even combinations of themuser1, item1, rel1
user1, item2, rel2
user2, item3, rel3
user3, item1, rel4
…

fi15.

.

Rec. Dataset
DSVP- CF- Prop

Train an inexpensive
proxy modelδ : (u, i) → ℝ

user1, item1, rel1
user1, item2, rel2

Propensity Correctionuser2, item3, rel3
user3, item1, rel4…

Mark the “hardness”
of each interaction in D(δ(u, i) − r)∇(u, i, r) =
∑pu,i
epoch

2

∇(u, i, r) =Due to the large catalog of items, account for potentially
missing data, especially for long-tail items∑epoch−AUC (δ(u, i), r)pu,iExplicit
feedback
Implicit/Seq.
feedback

Propensity Modelpu,i = P(interaction obser ved | relevant)• Re-weigh the hardness scores using the probability of auser-item interaction going missing (propensity)

Sample the hardest p% of:• Implicitly handles the long-tail and data sparsity issues(1) Interactions, i.e., highest ∇(u, i, r)(2) Users, i.e., highest ∇(u)(3) Items, i.e., highest ∇(i).

Rec. Dataset
Dpin user-item interaction data

user1, item1, rel1
user1, item2, rel2
user2, item3, rel3
user3, item1, rel4
…
16.

.

🧞 DATA-GENIE

Which sampler is best for my dataset?

Premise: Can we build an oracle-model which given (1) a dataset, (2) list of sampling strategies,and (3) a sampling budget, can automatically predict which sampling scheme would be the best?

Rec. Dataset
D🧞DATA-GENIE

SampleSampled Datasets

p%According to diff. sampling strategies

Ds1,p

Ds2,pDs3,pWhich sampler is best for my dataset?

Ds4,p• Dynamically predict the performance of aDs5,p

For all samplerssampling strategy for any given dataset

D• Circumvents the time-consuming process of
training and benchmarking various
recommendation algorithms

Ds,pDATA-GENIE🧞• A trained DATA-GENIE model can transfer toany dataset, and can predict the utility of
any sampling strategy

Agreement between:ℛ̂ s,p1. Ranking of models
trained on D
2. Ranking of models
trained on Ds,pRanking of different sampling strategies
Sorted according to predicted ℛ̂ s,pS4, S3, S1, S5 …
18

Ground-truth

🧞DATA-GENIERanking

CF Dataset
DRecommendation
Algorithms

Training Objective
Sample p%
Sampler “s”

R1R2R4• DATA-GENIE-regression:arg min̂ℛ−ℛs,ps,p)∑ (, s, p∑∑, p ℛsi,p > ℛsj,pℛs,pRanking

R5Train

CF Dataset
Ds,p• DATA-GENIE-ranking:arg minKendall’s
Tau

R3…
2Train

R4
R2
R1R5

R2R4
R1
R3

🧞 DATA-GENIE− ln σ (ℛ̂ si,p − ℛ̂ sj,p)Embed theentire dataset

D

Unsupervised
Graph Convolution
Network𝒟𝒟Ds,p19

EEs,pFusion
Multi-layer
Perceptronℛ̂ s,pExperiments
Setup• 16 different sampling strategies• 6 collaborative ltering datasets• Explicit/Implicit/Sequential feedback for each CF-dataset• 7 recommendation algorithms in our benchmarking suite• A total of 400k recommendation models trained (∼9 months of single-GPU compute time!)
Table: Samplingstrategies used in ourexperiments
fi20

Experiments
Major Results• Widely used practice of making dense data subsets (e.g., Head-user, centrality) seem to be
the worst ideas of all sampling strategies.

• SVP-CF signi cantly outperforms other samplers in retaining the ranking of different
recommendation algorithms.

• Using SVP-CF, we can ef ciently gauge theranking of different algorithms with adequate
con dence on 40-50% data sub-samples, leadingin an ∼2x time speedup.

• DATA-GENIE enjoys the same level of performance
Figure: Does DATA-GENIE improve sampling
performance with extreme sampling?
Table: Average Kendall’s Tau ofvarious sampling strategies

fififi21

with only 10% of the original data, equating to∼5.8x time speedup!

How to Train Data-E cient LLMs
Noveen Sachdeva 1

Ed H. Chi 2

James Caverlee 2

University of California, San Diego 1

22ffi

Lichan Hong 2

Benjamin Coleman 2

Wang-Cheng Kang 2
Julian McAuley 1
Google DeepMind 2

Jianmo Ni 2
Derek Z. Cheng 2

ScopeObjective

Language Modeling

Perform Accurate Language ModelingThat is, learn better next-token predictors:• δ : [token1, token2, …, tokenn] ↦; ∀ tokeni ∈Naive vs. Data-Ef cient
Naive:

Pre-Training

Train the modelon the entire dataset• Very large models• Very large datasets collected from all of the internet

Data-Ef cient:• Very expensive training procedure

Train the modelon the sampled
version of the dataset

fifi𝒯𝒯• Evaluation over hundreds of different tasks23

Ask-LLM

Sampling High-Quality LLM Pre-Training Data

Premise: Can we prompt an existing LLM to estimate the quality of a pre-training document?

Ask-LLM
Sampling High-Quality LLM Pre-Training Data

Robust framework:• Leverages the reasoning capabilities of modern LLMs
rather than common heuristics like perplexity• We prompt Flan-T5 and Gemma-7B for data quality• Explicit control over what kind of data we prefer
Why P(“yes” | prompt) is a good idea:• Real-valued “con dence” score needed to sort millionsof documents• One-shot decoding and no majority voting neededfi25

Density

Sampling Diverse LLM Pre-Training Data

Premise: Can we sample datapoints from diverse topics in the original dataset?

Density
Sampling Diverse LLM Pre-Training Data

Robust framework:• Estimate data density using hashedsentence-T5 embeddings• Up-weights the tail components and downweights the head components• No need for expensive techniques likeclustering, graph-cuts, etc. to localize a notionof coverage

27

Ask-LLM & Density
Metric: E ective Model Size

• With 100s of metrics, hard to devise a single notion of“quality.” Some metrics are hard-to-move whereas someare easy.

• We devise an “Effective Model Size” metric that is ascaling-law averaged normalized metric over all
downstream tasks:“ If our ablations (data sampling) lead to x performance, what
sized LLM should I have trained in the original setting (thefull dataset) to achieve the same x performance? ”ff28

Ask-LLM & Density
Experiments

Setup• We train T5-Large (800M parameters) for 524B tokens

Figure A: Does training on Ask-LLM sampled data converge faster?

Effective Model Sizeon the C4 dataset

Conclusions• Up to 44% speedup while training T5-Large• Training on data sampled by Ask-LLM (Gemma) isequivalent to training a 2x sized model on the entire
dataset• Density sampling recovers full-data performance( at-line) but Ask-LLM consistently exceeds it

Size of Sampled Dataset
Figure B: Size of sampled data vs. nal model quality

fifl29

This Dissertation
Outline

Chapter - I: Data SamplingPart-I: Recommender
SystemsOriginal Dataset

Part-II: LLM
Pre-Training
Data Summary

Chapter - II: Data Synthesis
Part-I: Recommender
Systems
Original Dataset

Part-II:
Auto-Regressive Data
Data Summary

Data Distillation: Automated Data Optimization
Inner Loop

Outer Loop
Data summary
optimized as free-parameters

Optimal model parameters trained onthe data summary

Empirical risk on the original dataset

Empirical risk on the data summary31

Data Distillation: Automated Data Optimization
Back-propogate directly
to the data summary
Train
Meta-Loss<< 50K distilled images

Learning algorithm

Inner Loop

EvaluateLoss

Outer Loop

50K real images
32

Data Distillation: Automated Data Optimization
TL;DR Directly optimize the data summary (stored as free parameters) via meta-learning

E.g. for CIFAR-10:

Completelyfake!

No Heuristics

Data summary is optimized
for training models

Data is optimized for aspeci c model

Computationally Expensive

DataDistillation

Most notably, this framework also requires:• The distilled data to be “optimizable”, e.g., pixel values in an image• Performing data distillation for discrete data settings like user-item interactions, text,graphs, etc. becomes highly non-trivialfi

33

In nite Recommendation Networks:
A Data-Centric Approach
Noveen Sachdeva 1

Mehak Dhaliwal 1

Carole-Jean Wu 2

University of California, San Diego 1fi

34

Meta AI 2

Julian McAuley 1

ScopeObjective
Perform Accurate Recommendation
That is, learn better relevance predictors:• δ : (user, item) ↦ ℝ; ∀ user ∈Implicit-feedback Recommender Systems1

1

Naive vs. Data-Ef cient1
1, item ∈ ℐ1

1Users11Naive:

11

Train therecommendation modelon the entire dataset

11

1

Items

Movies, Ads, Songs …

Data-Ef cient:

fifi

𝒰

Train therecommendation modelon the distilled versionof the dataset

35

∞-AEA Better Model for Recommendation

Premise: Does stretching the bottleneck layer of an autoencoder till ∞ help in better recommendation?

∞-AE

Primer: Neural Tangent Kernel• In nite-width Correspondence: Performing Kernelized Ridge

Regression with the Neural Tangent Kernel (NTK) emulates thetraining of an in nite-width NN for an in nite number of SGD steps.

df:ℝ↦ ℝ, its
Foragivenneuralnetworkarchitecture•θdd
corresponding NTK : ℝ × ℝ ↦ ℝ is given by:∂fθ(x) ∂fθ(x′)(x, x′) =,θ∼W [⟨ ∂θ∂θ ⟩]• Learning follows a double-descent phenomenon• Finite-width counterparts empirically outperform NTK for standard

￼

𝔼

37
fi

fi

fifi

￼

𝕂

𝕂

image classi cation tasks

Credit: https://openai.com/blog/deep-double-descent/∞-AE
Methodology• Xu is the bag-of-items representation for user u i.e. all the items that u interacted
with, and we aim to reconstruct it along with missing user preferences• Due to the in nite-width correspondence, ∞-AE optimizes in closed-form:
X̂ = K ⋅ (K + λI)−1 ⋅ X s.t. Ku,v :=(Xu, Xv) ∀u, v• The optimization has only a single hyper-parameter λTraining:(U 2 ⋅ I+U 2.376)

Inference:(U ⋅ I)• Memory complexity

Training:(U ⋅ I+U 2)

Inference:(U ⋅ I)

fi𝕂𝒪

𝒪𝒪𝒪• Time complexity

38

∞-AE
Experiments

Dataset

NeuMF

GCN

MVAE

EASE

∞-AEMagazine

13.6

22.5

12.1

22.8

23.0

ML-1M

25.6

28.8

22.1

29.8

32.8

Douban

13.3

16.6

16.1

19.4

24.9

Net ix

12.0

—20.8

26.8

30.5*

Table: nDCG@10 performance (higher is better) of various recommendation algorithms.
* represents training on 5% random users.

Figure: Performance of ∞-AE with varying depth.

• ∞-AE outperforms various state-of-the-art methods, even when trained on just 5% random users• 1 layer seems to be enough for optimal recommendation performance: common folk-knowledge• Even though the model is expensive; it is simplistic, easy to implement (thanks, JAX), and the performance is great!
But how to scale it up? 🤔

fl39

Distill-CF

Data Distillation for Recommendation Data

Key Idea: Use a smooth prior matrix followed by differentiable Gumbel sampling to distill discrete data

Distill-CF
Overview & Challenges

Outer loop: optimize the data summary for a xed learning algorithm
Data Summary

Unique challenges for distilling recommendation data:

Differentiable cost-functionsD• consists of discrete (u, i, r) tuples• Semi-structuredness: some users/items are more
popular than otherssD• is typically extremely sparse

Optimal recommendation
algorithm trained on Ds

Inner loop: optimize the learning algorithm for a xed data summary

fifi41Distill-CF

Items
Fake
Users

Methodology

Sampling prior matrixsX

Multi-stepGumbel Sampling(with replacement)
000

Robust framework:sX
Uses
Gumbelsamplingon
to mitigate the•heterogeneity of the problem

01
0

0
0
1

1
00

1
10

1
00

100

Intermediate matrix
̂sTrain ∞-AE on X̂s

Back-propagate
to X s(Inner loop)(Outer loop)

Trained ∞-AE• Perform Gumbel sampling multiple times for each

Compute Loss on theOriginal Dataset

fake-user to handle dynamic user/item popularityŝ
control sparsity in X by controlling the• Automaticallyentropy in X sLoss42X

Distill-CF
Experiments

Figure A: Size of data summary vs. trained model quality (Log-scale)• Using Distill-CF, we can get 96-105% of
full-data performance on as small as
0.1% data sub-samples, leading to as
much as ∼1000x time speedup!

• Distill-CF works well even for thesecond-best “Baseline” model, eventhough the data isn’t optimized using“Baseline”

Dataset∞-AE
NeuMF GCN MVAE EASE ∞-AE
(Distill-CF)

Magazine

13.6

22.5

12.1

22.8

23.0

23.8

ML-1M

25.6

28.8

22.1

29.8

32.8

32.5

Douban

13.3

16.6

16.1

19.4

24.9

24.2

Net ix

12.0

—20.8

26.8

30.5*

30.5

Table: nDCG@10 performance of various recommendation
algorithms. * represents training on 5% random users. Distill-CF has
a user budget of just 500 (0.1% for Net ix).

fl

fl

43

Figure B: Distill-CF + Baseline
for the ML-1M dataset.

Farzi Data: Autoregressive Data Distillation
Noveen Sachdeva 1
Wang-Cheng Kang 2

Zexue He 1

Jianmo Ni 2

Benjamin Coleman 2

Derek Z. Cheng 2

University of California, San Diego 1

44Julian McAuley 1

Google DeepMind 2

ScopeObjective

1. Sequential Recommender Systems

Perform Accurate Recommendation / LM
That is, learn better next-item/token predictors:→?

→• δ : [item1, item2, …, itemn] ↦ ℐ; ∀ itemi ∈ ℐ• δ : [token1, token2, …, tokenn] ↦?

→; ∀ tokeni ∈?

Naive vs. Data-Ef cient
Naive:

2. Language ModelingTrain the modelon the entire dataset
Data-Ef cient:

fifi𝒯𝒯

Train the modelon the distilled
version of the dataset

45

Farzi

Distilling Auto-Regressive Data

Key Idea: Think of a discrete sequence-of-events as a sequence-of-distributions that can be now
distilled via data distillation

Farzi
Intuition

47

FarziCan we distill this 3d tensor?

Challenge:
The data summary is 3-dimensional ⟹computationally intractable
Idea:
Keep a factorized data summary instead!

48

Farzi
Methodology (Contd.)

Challenge:
No closed-form inner-loop solvers ⟹How to get meta-gradient?

Derivation
Hessian Vector Products

Solution:
Ef cient reverse-mode Adam derivation• Naïve auto-diff memory complexity: (T ⋅ )• Reverse-mode Adam memory complexity: ( )𝒢𝒢fi𝒪𝒪Ours
49

FarziExperiments• Using Farzi, we can get 98-120% of full-Figure A: Size of data summary vs. trained model quality (Log-scale)data performance on as small as 0.1%
data sub-samples, leading to as much as∼1000x time speedup!

• Farzi also improves the performance ofmodels on the tail-portion of users anditems — which is of very valuable
importance in practice

Figure B: Performance of models trained on Farzi Data vs. Full Data on theuser/item coldness spectrum.
50This Dissertation
Future Roadmap

New Data Modalities

Data Optimization• Language: SFT, RLHF

• Ef ciency: Scalable ways to perform data

• Audio• Video•…

• Transferability: Better ways to create universal,
drop-in replacement data summaries• Order-sensitive data optimization techniques

New Applications

Fairness & Privacy• Continual Learning• How to optimize for these constraints• Neural Architecture Search• Hyper-parameter Opt.
fidistillation for bigger models & datasets

while sampling/distillation• DP: Can we guarantee impossibility of deanonymization when learning on datasummaries?

Gratitude
The McAuley Lab
UC San Diego

Julian McAuley
UC San Diego“Best Advisor Ever.”My Wonderful Collaborators

What we covered:Thank you!
Questions?
@noveens97fi

For papers & code: noveens.com

01

What is Data-Ef ciency

02

Data Sampling for RecSys

03

Data Sampling for LLMs

04

Data Distillation for RecSys

05

Data Distillation for Autoregressive Data