Infinite Recommendation Networks
A Data-Centric Approach

Question: Is more data what you need for better recommendation?
Noveen Sachdeva , Mehak Preet Dhaliwal , Carole-Jean Wu , Julian McAuley@noveens97
UC San Diego & Meta AI

NeurIPS, 2022

∞-AE

Infinite-width Autoencoder for Recommendation

Premise: Does stretching the hidden layers of an autoencoder till ∞ help in better recommendation?

∞-AE
Primer: Neural Tangent Kernel• In nite-width Correspondence: Performing Kernelized Ridge

Regression with the Neural Tangent Kernel (NTK) emulates thetraining of an ∞-width NN for an ∞ number of SGD steps.

dForagivenneuralnetworkarchitecture
f:ℝ↦ ℝ, its•
θcorresponding NTK, : ℝd × ℝd ↦ ℝ is given by:∂fθ(x) ∂fθ(x′)(x, x′) =,θ∼W [⟨ ∂θ∂θ ⟩]• Learning follows a double-descent phenomenon• Finite-width counterparts empirically outperform NTK for𝔼￼

fifi

￼

𝕂𝕂

standard image classi cation tasks

3Figure 1: Credit: https://openai.com/blog/deep-double-descent/∞-AEMethodology• Xu is the bag-of-items representation for user u i.e. all the items that u interacted
with, and we aim to reconstruct it along with missing user preferences• Due to the in nite-width correspondence, ∞-AE optimizes in closed-form:
X̂ = K ⋅ (K + λI)−1 ⋅ X s.t. Ku,v ≜(Xu, Xv) ∀u, v• The optimization has only a single hyper-parameter λTraining:(U 2 ⋅ I + U 2.376)

Inference:(U ⋅ I)• Memory complexity

Training:(U ⋅ I + U 2)

Inference:(U ⋅ I)

fi𝕂𝒪𝒪𝒪𝒪• Time complexity4

∞-AEExperiments

Dataset

NeuMF

GCN

MVAE EASE∞-AEMagazine13.6

22.5

12.1

22.8

23.0

ML-1M

25.6

28.8

22.1

29.8

32.8

Douban

13.3

16.6

16.1

19.4

24.9

Net ix

12.0

—

20.8

26.8

30.5*

Table 2: nDCG@10 performance (higher is better) of various recommendation algorithms.
* represents training on 5% random users.

Figure 3: Performance of ∞-AE with varying depth.

• ∞-AE outperforms various state-of-the-art methods, even when trained on just 5% random users (Net ix)• 1 layer seems to be enough for optimal recommendation performance (common folk-knowledge)• Even though the model is expensive; it is simplistic, easy to implement (thanks, JAX), and the performance is great!
But, how to scale it up? 🤔

flfl

5

Distill-CF
Data Distillation for Collaborative Filtering Data
Premise: Can we summarize the massive & sparse user-item matrix into a terse data summary?

111
11?

1Users

1

1
1

1

1
1
1

1
1

Items
Movies/Ads/Songs …

Fake
Users

0.2
0.8
0.3

0.1
0.9
0.4

0.5
0.1
10.8
0.5
0.8

0.1
10.9

10.4
0.2

Items
Movies/Ads/Songs …

0.9
0.40.2

Premise
What is Data-Centric AI?

Model-Centric AI

Data-Centric AI

Data

ModelData

Model

FreezeImprove

Improve

Freeze7Premise
Why Data-Centric Recommender Systems?

{ u, i, r }SurfaceClick{ u, i, r }Log{ u, i, r }…
Very large!

• Unsupervised → large quantities of user-feedback• Scaling-up systems by scaling-down data
• Shift focus from data quantity → data “quality”• Savings in time, human-effort & environmental
resources

Train simpler models
on large data
E.g. Linear modeling, Matrix
Factorization, Item-item CF, etc.

8?

Train expressive models
on down-sampled data
E.g. Higher-order modeling,
User-user CF, etc.

Distill-CF
Overview & Challenges

Idea: Treat the to-be-synthesized data as parameters, andlearn them through a bilevel optimization.

Outer loop — optimize the data summary for a xed learning algorithm• Challenges:• Data consists of discrete (u, i, r) tuples• Data is extremely sparse• Dynamic users/item popularity• Expensive bilevel optimization• Use ∞-AE for closed-form computation

Original Dataset

Optimal recommendation
algorithm trained on Dsof the inner loop• Optimizes for data-quality rather than quantityfi

Differentiable
cost-function

Inner loop — optimize the learning algorithm for a xed data summary

9fi

DataSummary

ItemsDistill-CF

Fake
Users

Methodology

Sampling prior matrixsX

Multi-stepGumbel Sampling(with replacement)
0000
1000
1

100

1
10

100

100

Intermediate matrix
̂sX

Train ∞-AE on X̂ss
X
Uses
Gumbelsamplingon
to mitigate the•heterogeneity of the problem(Inner loop)

α = (KXŝ Xŝ + λI)−1 ⋅ X ̂s

• Perform Gumbel sampling multiple times for each fakeuser to handle dynamic user/item popularity

Back-propagate to Xŝ
XAutomaticallycontrolsparsity
inby controlling the•entropy in X ss(Outer loop)

arg min𝔼Xs

10

Xu ⋅ log(KXu Xŝ ⋅ α)+u [(1 − Xu) ⋅ log(1 − KX X ŝ ⋅ α)]u+ λ2 ⋅ | | X ̂s | |1

Distill-CF
Experiments

Figure 4: Does Distill-CF outperform other samplers? (Log-scale)• Using Distill-CF, we can get 96-105% offull-data performance on as small as
0.1% data sub-samples, leading to as
much as ∼1000x time speedup!

• Distill-CF works well even for thesecond-best model (EASE), even though
the data isn’t optimized for it∞-AEDataset NeuMF GCN MVAE EASE ∞-AE
(Distill-CF)
Magazine

13.6

22.5

12.1

22.8

23.0

23.8

ML-1M

25.6

28.8

22.1

29.8

32.8

32.5

Douban

13.3

16.6

16.1

19.4

24.9

24.2

Net ix

12.0

—

20.8

26.8

30.5*

30.5

Table 5: nDCG@10 performance of various recommendation
algorithms. * represents training on 5% random users. Distill-CF hasa user budget of just 500 (0.1% for Net ix).

flfl

11

Figure 6: Distill-CF + EASE for
the ML-1M dataset.

Distill-CF
Experiments (Contd.)• Distill-CF is robust to noise (even though
not optimized for it), and is able to offer
signi cant performance even at highnoise ratios and very small support
datasets!

Figure 7: Performance of different samplers when there is noise in the original data.

• Less is more: EASE is more accurate

when trained on lesser amounts of data
generated by Distill-CF, compared totraining on the full-data
Figure 8: Performance comparison of ∞-AE vs. EASE when trained on down-sampled, noisy data.

fi12

Thank you!
@noveens97
For paper, code, and these slides:
noveens.com