Data-Centric Approaches to Recommendation
Question: Is more data what you need for better recommendation?

Noveen Sachdeva@noveens97
UC San Diego

Google Brain, July 2022

2Talk Layout• Primer, Premise & Scope• SVP-CF & Data-Genie 🧞• In nite Recommendation Networks• Dataset Distillation (Distill-CF)

fi• Future Directions

Primer
Recommender Systems

111111Users

111111

1

Items

Movies, Ads, Songs …

• Extremely sparse feedback• Inherently bi-partite• Long-tailed• Missing-not-at-randomItemPopularity4UsersItems

Premise
What is Data-Centric AI?

Model-Centric AI

Data-Centric AI

Data

ModelData

ModelFreeze

Improve

Improve

Freeze• Expensive• Under-studied• Well studied

5• Scalable

Premise
Why Data-Centric Recommender Systems?

{ u, i, r }SurfaceClick{ u, i, r }Log{ u, i, r }…
Very large!

• Unsupervised → large quantities of user-feedback• Scaling-up systems by scaling-down data
• Shift focus from data quantity → data “quality”• Dimension in performance : resources tradeoff• Savings in time, human-effort & environment

Train simpler models
on large datadegradation

E.g. Linear modeling, Matrix
Factorization, Item-item CF, etc.

6?

Train expressive models
on down-sampled data
E.g. Higher-order modeling,
User-user CF, etc.

ScopeScaling-up Systems by Scaling-down Data

Data down-sampling

Bias-sensitive

Variance-sensitive

Generate a data sample which can guarantee similar
performance of the same downstream model whentrained on the full-dataset vs. data summary

Generate a data sample which can accurately retain therelative ordering of different learning algorithms whentrained on the full-dataset vs. data summary
V(R1) = 0.2

V(R1) = 0.24
V(R2) = 0.2?

V(R2) = 0.22

V(R1) = 0.3?

V(R2) = 0.32

V(R3) = 0.26

V(R3) = 0.24

V(R3) = 0.34

V(R4) = 0.22

V(R4) = 0.26

V(R4) = 0.36

Data Summary1

Full Dataset

Data Summary2

7

ScopeScaling-up Systems by Scaling-down Data

Data down-sampling

Bias-sensitive

Variance-sensitive

Generate a data sample which can guarantee similar
performance of the same downstream model whentrained on the full-dataset vs. data summaryGenerate a data sample which can accurately retain therelative ordering of different learning algorithms whentrained on the full-dataset vs. data summary• Direct deployment of models trained on data summary• Model search e.g. NAS, hyper-parameter optimization• Faster research iterations• Of ine model-to-model comparison• Need modeling assumptions (at least for RecSys)• No modeling assumptions

fl8

ScopeScaling-up Systems by Scaling-down Data

Data down-sampling

Sampling

Synthesis

Pick the most informative subset of data-points

Generate a set of fake and informative data-points• Heuristics• Random, Head-user, Random-walks, Centrality…

• Typically, treat the to-be-synthesized data as• Coreset construction• Combinatorial optimization• In addition to being useful, the synthesized data is• Expressivity limited by the collected data

• Expressivity limited by the optimization procedureparameters, and learn them through gradient descent

fake — easy to share, release …

9

SVP-CF
Selection-via-proxy for collaborative filtering data
Premise: Easy parts of a dataset are most likely easy for all recommendation algorithms. Hence, removing
such data is unlikely to change the relative ordering of algorithms.

CF Dataset
DSVP- CF

Train an inexpensive
proxy modelδ : (u, i) → ℝ

user1, item1, rel1
user1, item2, rel2

Selection-via-proxy for collaborative ltering data

user2, item3, rel3
user3, item1, rel4…

Mark the “importance” of eachinteraction in D∇(u, i, r) =∑epochRobust framework:∇(u, i, r) =∑epoch(δ(u, i) − r)

2

Explicit
feedback− AUC (δ(u, i), r) Implicit/Seq.
feedback• Uses a proxy model to tag the importance of eachinteraction• Ef ciently handle multiple recommendation scenarios

Greedily retain either p% of :e.g. explicit, implicit, sequential, etc.

(1) Interactions with highest ∇(u, i, r)(2) Users with highest ∇(u)(3) Items with highest ∇(i).

CF Dataset
Dp• Sample across varieties of data modalities: interactions,users, items, or even combinations of themuser1, item1, rel1
user1, item2, rel2
user2, item3, rel3
user3, item1, rel4…

fifi11.

.

CF Dataset
D

SVP- CF- Prop

Train an inexpensive
proxy modelδ : (u, i) → ℝ

user1, item1, rel1
user1, item2, rel2

Handling the missing-not-at-random characteristics

user2, item3, rel3
user3, item1, rel4…

Mark the “importance” of eachinteraction in D(δ(u, i) − r)∇(u, i, r) =
∑
pu,i
epoch

2

∇(u, i, r) =

∑

epoch

−AUC (δ(u, i), r)pu,iExplicit
feedback
Implicit/Seq.
feedback

Propensity Model• Re-weigh the importance scores in SVP-CF using theprobability of a user-item interaction going missing(propensity).

pu,i = P(interaction obser ved | relevant)

Greedily retain either p% of :• Implicitly also handles the long-tail and data sparsity
issues in user-item interaction data.

(1) Interactions with highest ∇(u, i, r)(2) Users with highest ∇(u)(3) Items with highest ∇(i).

CF Dataset
Dp
user1, item1, rel1
user1, item2, rel2
user2, item3, rel3
user3, item1, rel4
…
12.

.

🧞 DATA-GENIE
Which sampler is best for me?

Premise: Can we build an oracle-model which given (1) a dataset, (2) list of sampling strategies, and (3) asampling budget, can automatically predict which sampling scheme would be the best?

CF Dataset
D🧞DATA-GENIESubsamplep%Sampled Datasets
According to diff. sampling strategies

Ds1,pDs2,pDs3,pWhich Sampler is best for me?

Ds4,p

Ds5,p

For all samplers• Dynamically predicts the performance of a sampling

Dstrategy for any given CF-dataset.

• A trained DATA-GENIE model can transfer to any dataset,Ds,pand can predict the utility of any sampling strategy.

DATA-GENIE🧞ℛ̂ s,pRanking of different sampling strategies
Sorted according to predicted ℛ̂ s,p

S4, S3, S1, S5 …
14

Ground-truth

🧞DATA-GENIE

Recommendation
Algorithms

How is it trained?
Subsample p%Sampler s• Circumvents the time-consuming processof training and benchmarking variousalgorithms.

• DATA-GENIE-regression:arg min̂ℛ−ℛs,p)∑ ( s,p2Ranking

CF Dataset
DR1

R2
R4Train

R4R2
R1
R5
Kendall’sTau

R3ℛs,pRanking

R5…

Train

CF Dataset
Ds,pR2
R4R1R3, s, p🧞 DATA-GENIE• DATA-GENIE-ranking:arg min∑∑, p ℛsi,p > ℛsj,p− ln σ (ℛ̂ si,p − ℛ̂ sj,p)Embed theentire dataset

DGraph ConvolutionNetwork𝒟𝒟Ds,p15

EEs,pFusion
Multi-layer
Perceptronℛ̂ s,pExperiments
Setup• 16 different sampling strategies• 6 collaborative ltering datasets• Explicit/Implicit/Sequential feedback for each CF-dataset• 7 recommendation algorithms in our benchmarking suite• A total of 400k recommendation models trained! (∼9 months of compute time!)
Table 1: Samplingstrategies used in ourexperiments
fi16Experiments
Major Results• Widely used practice of making dense data subsets (e.g. Head-user, centrality) seem to be
the worst ideas of all sampling strategies.

• SVP-CF signi cantly outperforms other samplers in retaining the ranking of different
recommendation algorithms.

• Using SVP-CF, we can ef ciently gauge theranking of different algorithms with adequate
con dence on 40-50% data sub-samples, leadingin an ∼2x time speedup.

• DATA-GENIE enjoys the same level of performance
Table 2: Average Kendall’s Tau ofvarious sampling strategies

Figure 3: Does DATA-GENIE improve sampling
performance with extreme sampling?

fififi17with only 10% of the original data, equating to∼5.8x time speedup!

∞-AEInfinite-width AutoEncoder for Recommendation

Premise: Does stretching the bottleneck layer of an autoencoder till ∞ help in better recommendation?

∞-AE
Primer: Neural Tangent Kernel• In nite-width Correspondence: Performing Kernelized Ridge

Regression with the Neural Tangent Kernel (NTK) emulates thetraining of an in nite-width NN for an in nite number of SGD steps.

df:ℝ↦ ℝ, itsForagivenneuralnetworkarchitecture•θdd
corresponding NTK : ℝ × ℝ ↦ ℝ is given by:∂fθ(x) ∂fθ(x′)(x, x′) =,θ∼W [⟨ ∂θ∂θ ⟩]• Learning follows a double-descent phenomenon• Finite-width counterparts empirically outperform NTK for standard

￼

𝔼

19
fi

fi

fifi

￼

𝕂𝕂

image classi cation tasks

Credit: https://openai.com/blog/deep-double-descent/∞-AE
Methodology• Xu is the bag-of-items representation for user u i.e. all the items that u interacted
with, and we aim to reconstruct it along with missing user preferences• Due to the in nite-width correspondence, ∞-AE optimizes in closed-form:
X̂ = K ⋅ (K + λI)−1 ⋅ X s.t. Ku,v :=(Xu, Xv) ∀u, v• The optimization has only a single hyper-parameter λTraining:(U 2 ⋅ I + U 2.376)

Inference:(U ⋅ I)• Memory complexity

Training:(U ⋅ I + U 2)

Inference:(U ⋅ I)

fi𝕂

𝒪

𝒪𝒪𝒪

• Time complexity

20

∞-AE
Experiments

Dataset

NeuMF

GCN

MVAE EASE∞-AEMagazine

13.6

22.5

12.1

22.8

23.0

ML-1M

25.6

28.8

22.1

29.8

32.8Douban

13.3

16.6

16.1

19.4

24.9

Net ix

12.0

—

20.8

26.8

30.5*

Table 5: nDCG@10 performance (higher is better) of various recommendation algorithms.
* represents training on 5% random users.

Figure 6: Performance of ∞-AE with varying depth.

• ∞-AE outperforms various state-of-the-art methods, even when trained on just 5% random users• 1 layer seems to be enough for optimal recommendation performance: common folk-knowledge• Even though the model is expensive; it is simplistic, easy to implement (thanks, JAX), and the performance is great!
But how to scale it up? 🤔

fl21

Distill-CF
Data Distillation for Collaborative Filtering Data
Premise: Treat the to-be-synthesized data as parameters, and learn them through a bilevel optimization.

11
1
11?

1

Users

1

1
1

1

1
1
1

1
1

Items
Movies/Ads/Songs …

Fake
Users

0.2
0.8
0.3

0.1
0.9
0.4

0.5
0.1
1

0.8
0.5
0.8

0.1
1
0.9

1
0.4
0.2

Items
Movies/Ads/Songs …

0.9
0.4
0.2

Distill-CF
Overview & Challenges

Outer loop — optimize the support set for a xed learning algorithm
Support dataset

Challenges:

Differentiable cost-functionsD• consists of discrete (u, i, r) tuples• Semi-structuredness: some users/items are more
popular than otherssD• is typically extremely sparse

Optimal recommendation
algorithm trained on Ds

Inner loop — optimize the learning algorithm for a xed support setfifi23

ItemsDistill-CF

FakeUsers

Methodology

Sampling prior matrixsX

Multi-stepGumbel Sampling(with replacement)
000

Robust framework:

010001100110100100Intermediate matrix
̂sX

Train ∞-AE on X̂ss
XUses
Gumbelsamplingonto mitigate the•heterogeneity of the problem(Inner loop)

α = (KXŝ Xŝ + λI)−1 ⋅ X ̂s

• Perform Gumbel sampling multiple times for each fakeuser to handle dynamic user/item popularity

Back-propagate on Xs
̂X
Automaticallycontrolsparsity
inby controlling the•entropy in X ss

(Outer loop)

arg min• Optimizes for data-quality rather than quantity
𝔼Xs

24Xu ⋅ log(KXu Xŝ ⋅ α)+u [(1 − Xu) ⋅ log(1 − KX X ŝ ⋅ α)]u+ λ2 ⋅ | | X ̂s | |1

Distill-CFExperiments

Figure 7: Does Distill-CF outperform other samplers? (Log-scale)• Using Distill-CF, we can get 96-105% of
full-data performance on as small as
0.1% data sub-samples, leading to as
much as ∼1000x time speedup!

• Distill-CF works well even for thesecond-best EASE model, even thoughdata isn’t optimized for it∞-AE
Dataset NeuMF GCN MVAE EASE ∞-AE
(Distill-CF)
Magazine

13.6

22.5

12.1

22.8

23.0

23.8

ML-1M

25.6

28.8

22.1

29.8

32.8

32.5Douban

13.3

16.6

16.1

19.4

24.9

24.2

Net ix

12.0

—20.8

26.8

30.5*

30.5

Table 8: nDCG@10 performance of various recommendation
algorithms. * represents training on 5% random users. Distill-CF has
a user budget of just 500 (0.1% for Net ix).

fl

fl

25

Figure 9: Distill-CF + EASE for
the ML-1M dataset.

Distill-CFExperiments (Contd.)• Distill-CF is robust to noise (even though
not optimized for it), and is able to offer
signi cant performance even at highnoise ratios and very small support
datasets!

Figure 10: Performance of different samplers when there is noise in the original data.

• Less is more: EASE is more accurate

when trained on lesser amounts of data
generated by Distill-CF, compared totraining on the full-data
Figure 11: Performance comparison of ∞-AE vs. EASE when trained on down-sampled, noisy data.

fi26

Future Directions

Extensions∞ Recommendation NetworksRanksets• Making it more scalable — sparse kernel computations• Formalize the notion of variance-sensitive sampling• More applications — search, XC, …

• 🧞 DATA-GENIE is still a two step-process. How tooptimize for a rankset?

• Extending to sequential recommendation

Fairness & Privacy

Applications• How to optimize for these while sampling/distilling• Continual Learning — catastrophic forgetting• Guaranteeing data privacy in distills, such that de-• NAS, Hyper-parameter Optimization

anonymization is impossible28

Acknowledgement
AdvisorJulian McAuley
UC San Diego

Collaborators

References[1] On Sampling Collaborative Filtering Datasets. Sachdeva, Wu, McAuley. In WSDM ’22.

fi

[2] In nite Recommendation Networks: A Data-Centric Approach. Sachdeva, Dhaliwal, Wu, McAuley. arXiv ’22.

Thank you!
Questions?
@noveens97
For papers, code, and these slides:
noveens.com