On Sampling Collaborative
Filtering Datasets
Noveen Sachdeva ; Carole-Jean Wu ; Julian McAuleyCF Dataset
DResearch Goal

CF Dataset
DpSubsamplep%

Recommendation Algorithms
R1

Generate a sample of a collaborative ltering (CF)dataset which can accurately retain the relative
ordering of different recommendation algorithms

R2
R4• Minimize the data subset size, such that

R3
R5…

difference between the two rankings is minimal• Directly correlates with the con dence in theresults of any paper comparing different
recommendation models trained on sub-sampleddata (vast majority)

Train

Train

Ranking

Ranking

R4R2
R1
R5R2
R4R1
R3fifi…

Minimize the difference
b/w these two rankings…

SVP-CF
Selection-via-proxy for collaborative filtering data
Premise: Easy parts of a dataset are most likely easy for all recommendation algorithms. Hence, removing
such data is unlikely to change the relative ordering of algorithms.

CF Dataset
DSVP- CF

Train an inexpensive
proxy modelδ : (u, i) → ℝ

user1, item1, rel1

Selection-via-proxy for collaborative ltering data

user1, item2, rel2
user2, item3, rel3
user3, item1, rel4…

Mark the “importance” of eachinteraction in D∇(u, i, r) =Robust framework:∑epoch∇(u, i, r) =• Uses a proxy model to tag the importance of each

∑

epoch

(δ(u, i) − r)

2

Explicit
feedback− AUC (δ(u, i), r) Implicit/Seq.
feedbackinteraction• Ef ciently handle multiple recommendation

Greedily retain either p% of :• Sample across varieties of data modalities:(1) Interactions with highest ∇(u, i, r)(2) Users with highest ∇(u)(3) Items with highest ∇(i)

scenarios e.g. explicit, implicit, sequential, etc.

.

interactions, users, items, or even combinationsof them

CF Dataset
Dp
user1, item1, rel1
user1, item2, rel2
user2, item3, rel3

fifi

user3, item1, rel4…

.

.

SVP- CF- Prop
Handling the missing-not-at-random characteristics

CF Dataset
DTrain an inexpensive
proxy modeluser1, item1, rel1
user1, item2, rel2
user2, item3, rel3
user3, item1, rel4…

Mark the “importance” of eachinteraction in D(δ(u, i) − r)∇(u, i, r) =∑pu,iepoch2∇(u, i, r) =• Re-weigh the importance scores in SVP-CF using
the probability of a user-item interaction goingmissing (propensity).

δ : (u, i) → ℝ

∑

epoch

−AUC (δ(u, i), r)pu,iExplicit
feedback
Implicit/Seq.
feedback

Propensity Modelpu,i = P(interaction obser ved | relevant)

Greedily retain either p% of :• Implicitly also handles the long-tail and data
sparsity issues in user-item interaction data.

(1) Interactions with highest ∇(u, i, r)(2) Users with highest ∇(u)(3) Items with highest ∇(i).

CF Dataset
Dpuser1, item1, rel1
user1, item2, rel2
user2, item3, rel3
user3, item1, rel4…

.

.

🧞 DATA-GENIE
Which sampler is best for me?

Premise: Can we build an oracle-model which given (1) a dataset, (2) list of sampling strategies, and (3) asampling budget, can automatically predict which sampling scheme would be the best?

🧞DATA-GENIE

CF Dataset
DSubsamplep%Sampled Datasets
According to diff. sampling strategies

Ds1,pDs2,p

Ds3,pWhich Sampler is best for me?
Ds4,pDs5,pFor all samplers• Dynamically predicts the performance of asampling strategy for any given CF-dataset.

• A trained DATA-GENIE model can transfer to any
dataset, and can predict the utility of any
sampling strategy.

DDs,pDATA-GENIE🧞ℛ̂ s,pRanking of different sampling strategies
Sorted according to predicted ℛ̂ s,pS4, S3, S1, S5 …

Ground-truth

🧞DATA-GENIE

Recommendation
Algorithms

How is it trained?
Subsample p%• Circumvents the time-consuming
process of training and benchmarking
various algorithms.
• DATA-GENIE-regression:arg min̂ℛ−ℛs,p)∑ ( s,p2Ranking

CF Dataset
DSampler s

R1R2R4Train

R4R2
R1R5Kendall’s
Tau

R3ℛs,pRanking

R5…

Train

CF Dataset
Ds,pR2
R4R1
R3, s, p

🧞 DATA-GENIE• DATA-GENIE-ranking:arg min∑∑, p ℛsi,p > ℛsj,p− ln σ (ℛ̂ si,p − ℛ̂ sj,p)

D

𝒟

𝒟

Ds,pEmbed theentire dataset
Graph Convolution
NetworkEEs,pFusion
Multi-layer
Perceptronℛ̂ s,pExperiments
Setup• 16 different sampling strategies• 6 collaborative ltering datasets• Explicit/Implicit/Sequential feedback for each CF-dataset• 7 recommendation algorithms in our benchmarking suite• A total of 400k recommendation models trained! (∼9 months of compute time!)

fiTable 1: Samplingstrategies used in ourexperiments

Experiments
Major Results• Widely used practice of making dense data subsets (e.g. Head-user, centrality)
seem to be the worst ideas of all sampling strategies.

• SVP-CF signi cantly outperforms other samplers in retaining the ranking ofdifferent recommendation algorithms.

fiTable 1: Average Kendall’s Tau ofvarious sampling strategies

Figure 2: Does DATA-GENIE improve sampling
performance with extreme sampling?

Experiments
Major Results• Widely used practice of making dense data subsets (e.g. Head-user, centrality)
seem to be the worst ideas of all sampling strategies.

• SVP-CF signi cantly outperforms other samplers in retaining the ranking ofdifferent recommendation algorithms.

fiTable 1: Average Kendall’s Tau ofvarious sampling strategies

Figure 2: Does DATA-GENIE improve sampling
performance with extreme sampling?

Experiments
Major Results• Widely used practice of making dense data subsets (e.g. Head-user, centrality)
seem to be the worst ideas of all sampling strategies.

• SVP-CF signi cantly outperforms other samplers in retaining the ranking ofdifferent recommendation algorithms.

fiTable 1: Average Kendall’s Tau ofvarious sampling strategies

Figure 2: Does DATA-GENIE improve sampling
performance with extreme sampling?

Experiments
Major Results• Widely used practice of making dense data subsets (e.g. Head-user, centrality)
seem to be the worst ideas of all sampling strategies.

• SVP-CF signi cantly outperforms other samplers in retaining the ranking ofdifferent recommendation algorithms.

fiTable 1: Average Kendall’s Tau ofvarious sampling strategies

Figure 2: Does DATA-GENIE improve sampling
performance with extreme sampling?

Experiments
Major Results• Widely used practice of making dense data subsets (e.g. Head-user, centrality)
seem to be the worst ideas of all sampling strategies.

• SVP-CF signi cantly outperforms other samplers in retaining the ranking ofdifferent recommendation algorithms.

• Using SVP-CF, we can ef ciently gauge theranking of different algorithms withadequate con dence on 40-50% data subsamples, leading in an ∼2x time speedup.

• DATA-GENIE enjoys the same level offifi

fi

Table 1: Average Kendall’s Tau ofvarious sampling strategies

Figure 2: Does DATA-GENIE improve sampling
performance with extreme sampling?

performance with only 10% of the original
data, equating to ∼5.8x time speedup!

Environmental Consequences
Given an average weekly RecSys development cycle consisting of:• Training / testing various recommendation algorithms• On a medium-sized industrial dataset• Over a modest GPU setup
Table 1: CO2 emissions comparison

We compare the downstream CO2 emissions of a brute-force search vs. DATA-GENIE

Future Directions• Relative ordering of recommendation algorithms is just a start — encourage the community to think more about general coresets inthe context of recommendation.

• Analyzing the fairness aspects of training recommendation algorithms on data subsets.

fi• Transfer to other domains — classi cation, clustering, graphs, etc.

Thanks!
@noveens97
For paper, code, and these slides: