Infinite Recommendation Networks: A Data-Centric Approach
Authors: Noveen Sachdeva
Mehak Preet Dhaliwal
Carole-Jean Wu
Julian McAuley
Venue: NeurIPS 2022 Accept (2022)
Keywords: Dataset Distillation
NTK
Data-centric AI
Recommender Systems

Abstract: We leverage the Neural Tangent Kernel and its equivalence to training infinitely-wide neural networks to devise $\infty$-AE: an autoencoder with infinitely-wide bottleneck layers. The outcome is a highly expressive yet simplistic recommendation model with a single hyper-parameter and a closed-form solution. Leveraging $\infty$-AE's simplicity, we also develop Distill-CF for synthesizing tiny, high-fidelity data summaries which distill the most important knowledge from the extremely large and sparse user-item interaction matrix for efficient and accurate subsequent data-usage like model training, inference, architecture search, etc. This takes a data-centric approach to recommendation, where we aim to improve the quality of logged user-feedback data for subsequent modeling, independent of the learning algorithm. We particularly utilize the concept of differentiable Gumbel-sampling to handle the inherent data heterogeneity, sparsity, and semi-structuredness, while being scalable to datasets with hundreds of millions of user-item interactions. Both of our proposed approaches significantly outperform their respective state-of-the-art and when used together, we observe $96-105$% of $\infty$-AE's performance on the full dataset with as little as $0.1$% of the original dataset size, leading us to explore the counter-intuitive question: Is more data what you need for better recommendation?

Infinite Recommendation Networks: A Data-Centric Approach
Noveen Sachdeva
Mehak Preet Dhaliwal
Carole-Jean Wu
Julian McAuley
University of Californa,, San Diego
Meta AIF edu
{nosachde, carolejeanvummeta.coom
Abstract
We leverage the Neural Tangent Kernel and its equivalence to training infinitely- wide neural networks to devise co-AE: autoencoder with infinitely wide bottle- neck layers. The outcome is a highly expressive yet simplistic recommendation model with single rrper-paramete' and a closed-form solution. Leveraging AE's simplicity, we also develop DISTIIII-CF for synthesizing tiny, high-fidelity data summaries which distill the most important knowledge from the extremely large and sparse user-item interaction matrix for efficient and accurate subsequent data-usage like model training, inference, architecture search, etc. This takes a data-centric approach to recommendation, where we aim to improve the quality of logged user-feedback data for subsequent modeling, independent of the learning algorithm. We particularly utilize the concept of differentiable Gumbel-san -ammpligg handle the inherent data heterogeneity, sparsity, and structuredness, while being scalable to datasets with hundreds of millions of user- -te- interactions. Both of our proposed approaches significantly outperform their respective state-of-the- art and when used together, we observe 96 105% of co-AE's performance on the full dataset with as little as 0.1% of the original dataset size, leading us explore the counter-intuitive question: Is more data what you need for better recommendation?
Introduction
The Neural Tangent Kernel (NTK) has recently advanced the theoretical understanding of how neural networks learn [2, 20). Notably, performing Kernelized Ridge Regression (KRR) with the NTK has been shown to be equivalent to training infinitely-wide neural networks for an infinite number of SGD steps. Owing to KRR's closed-form solution, these networks be trained in fast and efficient manner whilst not sacrificing expressivity. While the application of infinite neural networks is being explored in various learning problems [48, 3], detailed comparative analyses demonstrate that deep, finite-width networks tend to perform significantly better than their infinite-width counterparts for variety of standard computer-vision tasks [3]].
On the contrary, for recommendation tasks, there always has been a debate of linear vs. non-linear networks [29, 65], along with the importance of increasing the width depth of the network [1139]] At high level, the general conclusion is that a well-tuned, wide and linear network can outperform shallow and deep neural networks for recommendation [50]. We extend this debate by introducing our model oo-AF, an autoencoder with infinitely wide bottleneck layers and examine its behavior on the recommendation task. Our evaluation demonstrates significantly improved results over state-of-thh-art (SoTA) models across various datasets and evaluation metrics.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A rising challenge in recommender systems research has been the increased cost of training models on massive datasets which can involve billions of user-item interaction logs, in terms of time, com- putational resources, as well as the downstream carbon footprint. Moreover, despite anonymization efforts, privacy risks have been associated with publicly released user data [38]. To this end, we further explore recommendation from data-centric viewpoint [60], which we loosely define as:
Definition 1.1. (Data-centric AI) Systematic methods for improving the collected data's 's quality, thereby shifting the focus from merely acquiring large quantities of data; implicitly helping in learning algorithu''s 's generalization, scalability, and eco-sustainability.
Previous work on data-centric techniques generally involve constructing terse data summaries of large datasets, and has focused on domains with continuous, real-valued features such as images [66, 4 which are arguably more amenable to data optimization approaches. Due the heterogeneity, sparsity, and remi-structurednes. of recommendation data, such methods are not directly applicable. Common approaches for scaling down such recommendation datasets typically include heuristics such as random, head-user, or k-core sampling. More principled approaches include coreset construction [57] that focus on optimizing for picking the set of data-points that are the most representative from a given dataset, and are generally shown to out-perform various heuristic sampling strategies. However, synthesizing informative summaries for recommendation datasets still remains a challenge.
Consequenly, we propose DISTILL-CF, a data distillation framework for collaborative filtering (CF) datasets that utilizes oc-AE in bilevel optimization objective to create highly compressed, informative, and generic data summaries. DISTILL-CC CFF employs efficient multi-step differentiable Gumbel-san Gumbel-sampling sampling [23] each step of the optimization to encompass the heterogeneity, sparsity, and semi- seni-structuredness of recommendation data. We further provide an analysis of the denoising effect observed when training the model on the synthesized versus the full dataset.
To summarize, in this paper, we make the following contributions:
We develop so-AE: an infinite-width autoencoder for recommendation, that aims reconstruct the incomplete preferences in user's 's item consumption sequence. We demonstrate its efficacy on four datasets, and demonstrate that co-AE outperforms complicated SoTA models with only a single fully-connected layer, closed-form optimization, and a single hyper-paramen parameter. We believe our work be the first demonstrate that an anfinite-width network can outperform their finite-widis SoTA counterparts for practical scenarios like recommendation.
We subsequently develop DISTILL-CF: a novel data distillation framework that can synthesize tiny yet accurate data summaries for variety of modeling applications. We empirically demonstrate similar performance of models trained on the full dataset versus training the same models on 2 3 orders smaller data summaries synthesized by DISTILL-CF. Notably, DISTILL- CF and oo-AE are complementary for each other's practicality. as co- AE's closed- form solution enables DISTILL-CF scale datasets with hundreds of millions of interactions; whereas, DISTILL-CF's succinct data summaries help improving Ac-AE's restrictive training complexity. and achieving SoTA performance when trained on these tiny data summaries.
Finally, also note that DISTILL-CE CF has strong data denoising effect, validated with counter-intuitive observation - there is noise in the original data, models trained on less data synthesized by DISTILI--FF original dataset. Such observations, along with the strong data compression results attained by DISTILL-CF, reinforce our data-centrie viewpoint to recommendation, encouraging the community to think more about the quality of collected data, rather than its quantity.
be better than the same model trained on the entire
2 Related Work
Autoencoders for recommendation. Recent approaches to implicit feedback recommendation involve building models that can re-construct an incomplete user preference list using autoencoders [32, 59, 54, 33]. The CDAE model [64] first introduced this idea and used a standard denoising autoencoder for recommending new items to users. MVAE [32] later extended this idea to use variational autoencoders, and provided a principled approach to perform variational inference for this model architecture. More recently, EASE [59] proposed using a shallow autoencoder and estimates only an item-item similarity matrix by performing ordinary least squares regression on the relevance
matrix, resulting in closed-font optimization.
2

---

Infinite neural networks. The Neural Tangent Kernel (NTK) [20] has gained significant attention because of its equivalence to training infinitely-wide neural networks by performing KRR with the NTK. Recent work also demonstrated the double-descent risk curve [4] that extends the classical regime of train vs. test error for overparameterized neural networks, and plays a crucial role in the generalization capabilities of such infinite neural networks. However, despite this emerging promise of utilizing NTK for learning problems, detailed comparative analyses [43, 31, 2] for computer vision tasks demonstrate that finite-width networks are still significantly more accurate than infinite-width ones. Looking at recommendation systems, [65] performed a theoretical comparison between Matrix Factorization (MF) and Neural MF [18] by studying their expressivity in the infinite-width limit, comparing the NTK of both of these algorithms. Notably, their settings involved the typical (user-ID, item-ID) imput to the recommendation model, and observed results that were equivalent to random predictor. [48] performed a similar study that performed matrix completion using the NTK of single layer fully-connected neural network, but assumed meaningful feature-prion pritoe available beforehand.
Data sampling & distillation. Data sampling is ubiquitous sampling negatives while contrastive learning [51, 25], sampling large datasets for fast experimentation [57]. sampling for evaluating expensive metrics [30], etc. In this paper, we primarily focus on sampling at the dataset level, principled approaches of which can be categorized into: (1) coreset construction methods that aim to pick the most useful datapoints for subsequent model training 17, 27, 53, 26]. These methods typically assume the availability of a submodular set-function f V + RR+ V S X for given dataset X (see [6] for systematic review on submodularity), and use this set- functiinion f as proxy to guide the search for the most informative subset; and (2) dataset distillation: instead of picking the most informative data-points, dataset distillation techniques aim to synthesize data-points that can accurately distill the knowledge from the entire dataset into a small, synthetic data summary. Originally proposed in [62], the authors built upon the notions of gradient-basee hyper-paramen optimization [34] to synthesize representative images for model training. Subsequently, a series of works [67, 66, 40, 41] propose various subtle modifications to the original framework, for improving the cample-comllexities of models trained on data synthesized using their algorithms. Note that such distillation techniques focused continuous data like images, which are trivial to optimize in the original data-distillation framework. More recently, [24] proposed a distillation technique for synthesizing fake graphs, but also assumed to have innate node representations available beforehand, prohibiting their method's application for recommendation data.
3 A- Infinite-width Autoencoders for Recommendation
Notation. Given a recommendation dataset D = { (user,, item;, relevancee=, consisting of n user-item interactions defined over set of users 24 set of items I, and operating over binary relevance score (implicit feedback); we aim to best model user preferences for item recommendation. The given dataset DD can also be viewed as an interaction matrix, X E TR 4X| where each entry X,, is missing. Note that X is typically
either represents the observed relevance for item i by user extremely sparse, i.e., n < [24] x [Z]. More formally, we define the problem of recommendation as accurate likelihood modeling of P(i u, D) Vu E U, Vi E I.
Model. AE takes an autoencoder approach to recommendation, where the all of the bottleneck layers are infinitely-wide. Firstly, to make the original bi-variate problem of which item to recommend to which user more amenable for autoencoders, we make simplifying assumption that a user can be represented only by their historic interactions with items, i.e., the much larger set of users lie in the linear span of items. This gives two kinds of modeling advantages: (1) we no longer have to find a unique latent representation of users; and (2) allows AE to be trivially applicable for any user not in U. More specifically, for given user vector of historical interactions with items X,, E RIZZ, which is simply the Th row in X. We then employ the Neural Tangent Kernel (NTK) [20] of an autoencoder that takes the bag-of-itent representation of users input and aims to reconstruct it. Due to the infinite-width correspondence [20, 21. performing Kernelized Ridge Regression (KRR) with the estimated NTK is equivalent to training an infinitely- wide autoencoder for an infinite number of SGD steps. More formally, given the NTK, K R x IR| + R over an RKHS H of single-layer autoencoder with an activation function a (see [47] for the NTK derivation of a fully-connected neural network), we reduce the recommendation problem to KRR follows:
represent it by the sparse, ofgg-of-ordh
3

---

arg min
Σ ueu
1I((Xul x.12 A- 1111f
(1)
s.t. f(Xula) = Σ aj K(,) K(Xu; X.) o x xX, X
Where À is a fixed regularization hyper-parameter, a == [ai;; 02 aiul E R are the set of dual parameters
are interested in estimating, ā represents the dual activation of a [14], and d' represents its derivative. Defining gramian matrix K E IR ||]|[| where each element can intuitively be seen as the similarity of two users, í.e., K,, = K u,y Xu,). the optimization problem defined in Equation (1) has closed-form solution given by
(K + X1) 11 X. We can subsequently perform inference for any novel user as follows: P(: u, D) = softmax (f(X, á)). We also provide o0-AE's training and inference pseudo-codes in Appendix A, Algorithms 1 and 2.
Scalability. We carefully examine the computational cost of oc oc-AE's training and inference. Starting with training, oc-AE has the following computationalll-exppee
steps: (1) computing the gramian matrix K; and (2) performing its inversion. The overall training time complexity thus comes out to algorithm [12] for matrix inversion, whereas the memory complexity ([44) [Z] + (u(2) for storing the data matrix X and the gramian matrix K. As for inference for single user, both the time and memory requirements are (((4)) (Z]). Observing these computational complexities, we note that co-AE can difficult scale-up to larger datasets naively. To this effect, we address oc'EEs scalability challenges in DISTILL-CF (Section 4), by leveraging a simple observation from support vector machines: not all datapoints (users in our case) are important for model learning. Additionally, in practice, we can perform all of these matrix operations on accelerators like GPU/TPUs and achieve a much higher throughput.
be 0[24]2 [Z] + (2)66676 if we use the Coppersmith-Wing
DISTILL-CF
Motivation. Representative sampling of large datasets has numerous downstream applications. Consequently, in this section we develop DISTILL-CF: a data distillation framework to synthesize small, high-fidelity data summaries for collaborative filtering (CF) data. We design DISTILII-CH with the following rationales: (1) mitigating the scalability challenges in oo-AE by training only on the terse data summaries generated by DISTILL--F; (2) improving the sample complexity of existing, finite-width recommendation models; (3) addressing the privacy risks of releasing user feedback data by releasing their synthetic data summaries instead; and (4) abating the downstream CO2 emissions of frequent, large-scale recommendation model training by estimating their parameters only on much smaller data summaries synthesized by DISTILL-CF.
Challenges. Existing work in data distillation has focused on continuous domain data such as images [40, 41, 67, 66], and are not directly applicable to heterogeneous and nemi-structured domains such as recommender systems and graphs. This problem is further exacerbated since the data for these tasks is severely sparse. For example, in recommendation scenarios, a vast majority of users interact with very few items [22]. Likewise, the nodes in a number of graph-based datasets tend to have connections with very small set of nodes [68]. We will later show how our DISTILL-CE framework elegantly able to circumvent both these issues while being accurate, and scalable to large datasets.
Methodology. Given a recommendation dataset D, we aim to synthesize a smaller, support dataset D* that can match the performance of recommendation models Φ /4 x I IR when trained D versus DD We take inspiration from [40], which also a data distillation technique albeit for images. Formally, given recommendation model Φ, held-out validation set Dval, and differentiable loss function 7 : R x + IR that measures the correctness of a prediction with the actual user-item relevance, the data distillation task can be viewed as the following bilevel optimization problem:
arg min D
E [0 (u, 2), ; s.t. S = arg min
E (udrr)D
((((u,)),)) (2)

---

This optimization problem has an outer loop which searches for the most informative support dataset D* given fixed recommendation model ø', whereas the inner loop aims to find the optimal recommendation model for fixed support set. In DISTILII-CF. we use co-AR as the model of choice at each step of the inner loop for two reasons: (1) as outlined in Section 3, co-AF has a closed-form solution with single hyper-parameter X, making the inner-loop extremely efficient; and (2) due to the infinite-width correspondence [20, 2], oo-AE is equivalent to training an infenitely-wide autoencoder on D*, thereby not compromising on performance.
For the outer loop, we focus only on synthesizing fake users (given a fixed user budget ()) through learnable matrix X* R which stores the interactions for each fake user in the support dataset. However, to handle the discrete nature of the recommendation problem, instead of directly optimizing for X DISTILI--CF instead learns continuous prior for each user-item pair, denoting the importance of sampling that interaction in our support set (similar to the notion of propensity [56, 58D. We then sample X ~ X* to get our final, discrete support set.
Instead of keeping this sampling operation post-hoc, i.e., after solving for the optimization problem in Equation (2); we perform differentiable Gumbel-sampling [23] on each row (user) in X at every optimization step, thereby ensuring search only over sparse and discrete support sets. A notable property of recommendation data is that each user can interact with a variable number of items (this distribution is typically long-tailed due to Zipf's law [63]). We circumvent this dynamic user sparsity issue by taking multiple Gumbel-samples for each user, with replacement. This implicitly gives DISTILL-CF the freedom control the user and item popularity distributions in the generated data summary X by adjusting the entropy the
X
Having controlled for the discrete and dynamic nature of recommendation data by the multi-step Gumbel-samee -smpling trick, we further focus maintaining the sparsity of the synthesized data. To do so, in addition to the outer-loop reconstruction loss, we add an Ll-penalty over X² to promote and explicitly control its sparsity [16, Chapter 3J. Furthermore, tuning the number of Gumbel samples we take for each fake user, gives more control over the sparsity in our generated data summary. More formally, the final optimization objective in DISTILL-CE can be written as:
Xx log( (rg(Kxxx=> .0) + (1 Xx) log(1 K Xx,x= as] A2 -2 ||80|11
arg min X
E andl
s.t. a= (Kx + =Kx AT)-1 Xx x= (sstbal((
Where, A2 represents an appropriate regularization hyper-parameter for minimizing the L1-norm of the sampled support set X=, KXY represents the gramian matrix for the NTK of co-AE over X and Y as inputs, T represents the temperature hyper-parameter for Gumbel-sampling, y denotes the number of samples to take for each fake user in X", and a represents an appropriate activation function which clips all values over to be exactly 1, thereby keeping X' binary. We use hard-tanh in our experiments. We also provide DISTILI-CF'S pseudo-code in Appendix A, Algorithm 3.
Scalability. We now analyze the time and memory requirements for optimizing Equation (3). The inner loop's major component clearly shares the same complexity as x0-AE. However, since the parameters of co-AE (a in Equation (1)) are now being estimated over the much smaller support set X the time complexity reduces to O(12 (ZI) and memory to O(u [Z]), where μ typically only lies in the range of hundreds for competitive performance. On the other hand, for performing multi- step Gumbel-sampling for each synthetic user, the memory complexity a naïve implementation would be O(y- μ Zl) since AutoGrad stores all intermediary variables for backward computation. However, since the gradient of each Gumbel-samplin and can be computed individually, using jax. custom_ pestom_vjp..oo we reduced its memory complexity to Op ) adding nothing to the overall inner-loop complexity.
step is independent of other sampling steps
For the outer loop, we optimize the logistic reconstruction loss using SGD where we randomly sample a batch of users from 'A and update X directly. In totality, for an { number of outer-loop iterations, the time complexity to run DISTILI-CF is O (n² b b-46. /z)-ZZ)), and O(b - (u+bb) b) IIZ)) for memory. In our experiments, we note convergence in only hundreds of outer-loop iterations, making DISTILL-CF scalable even for the largest of the publicly available datasets and practically useful.
5

---

Table 1: Comparison of co-AE with different methods various datasets. All metrics are better when higher. Brief set data statistics can be found in Appendix B.3, Table 2. Bold values represent the best in given row, and underlined represent the recond-best. Results for co-AE the Netflix dataset (marked with a +) consist of random user-sampling with max budget of 25K = 5.4% users, and results for DISTILL- CF + AE have user-budget of 500 for all datasets.
Light GCN
Dataset
Metric
PopRec
MF
NeuMF
EASE
MVAE
no-ATE
only
D0-AA
AUC
0.8436
0.8445
0.8475
0.8525
0.8141
0.6673
0.8507
0,8539
0.8584
H 10
14.35
14.36
18.36
18.35
27.12
26.31
17.94
27.09
28.27
Amazzn Magazime
NDCG@10
8.42
8.33
13.1
13.6
22.57
22.84
12.18
23.06
23.81
NDCG@ 100
19.38
19.31
21.76
21.13
29.92
28.27
30.75
31.75
10
6.85
6.73
9.24
9.00
13.2
12.96
8.81
13.22
13.76
0.8332
0,9457
IR 10
13.07
12.93
24.63
23.25
27.43
28.54
21.7
31,51
31,16
30.38
29.63
58.61
ML-IM
NDCG@10 10
13.84
13.74
25.65
24.44
28.85
29.88
22.14
32.82
32.52
NDCG@ 10D
19.49
19.13
35.62
33.93
38.29
40.16
33.82
42.53
41.29
PSP@ 10
1.10
2.41
2.26
2.72
3.22
0.8945
0.9258
0.9523
HR 10
11.06
10.71
12.69
12.79
15.98
17.93
15.36
23,56
22.98
Douban
HR 100
17.07
16.63
20.29
19.69
22.38
25.41
22.82
28.37
27.20
NDCG@10
11.63
11.24
13.21
13.33
16.68
16.17
24.94
24.20
NDCG@ 100
12.63
12.27
14,96
14.39
17.20
19.55
17.32
23,26
22.21
10
AUC
0.9249
0,9234
0,9234
0,9244
0.9418
0.9495
0.96633
0.9728
HR e
12.14
11.49
11.69
11.06
26.03
20.6
29.69
29.57
Netflx
HR 100
28.47
27.66
27.72
26.76
Timed
50.35
44.53
50.88*
49.24
NDCG@10
12.34
30.59*
NDCG 100
17.79
16.95
17.17
16.40
35.09
29.22
36.59*
35.58
3.75
Experiments
Setup. We use four recommendation datasets with varying sizes and sparsity characteristics. A brief of data statistics can be found in Appendix B.3, Table 2. For each user in the dataset, we randomly split their interaction history into 80/10/1055 train-test-validation splits. Following recent warnings against unrealistic dense preprocessing of recommender system datasets [55, 57], we only prune users that have fewer than 3 interactions to enforce at least one interaction per user in the train, test, and validation sets. No such preprocessing followed for items.
Competitor methods & evaluation metrics. We compare oo-AE E with various baseline and SoTA competitors as surveyed in recent comparative analyses [1, 13]. More details on their architectures can be found in Appendix B.1. We evaluate all models on variety of pertinent ranking metries. namely AUC, HitRate (HR @ k). Normalized Discounted Cumulative Gain (nDCG kk), and Propensity-score Precision (PSP@k), each focusing on different components of the algorithm performance. A notable addition to our list of metrics compared to the literature is the PSP metric [22], which we adapt to the recommendation use case as an indicator of performance on tail items. The exact definition of all of these metries can be found in Appendix B.5.
Training details. We implement both ox-AE and DISTILL-CF using JAX [8] along with the Neural- Tangents package [44] for the relevant NTK computations. We re-use the official implementation of LightGCN, and implement the remaining competitors ourselves. To ensure fair comparison, we conduct hyper-parameter search for all competitors on the validation set. More details on the hyper-parameters for co-AE, DISTILL-CF, and all competitors can be found in Appendix B.3, Table 3. All of our experiments are performed on single RTX 3090 GPU, with random-see -eed initialization of 42. Additional training details about DISTILL-CF can be found in Appendix B.4.
I 'Ou implementation oo- oo-AE is available at OOur implementation for DISTILL-CE MF available https::/githhub.comm/ https://gttuubbooom/ovee///////s ac_cf

---

Amazon Magazine
ML-1M
Douban
Netflix
Heee0
HROL0
HRC10
Hiee
2003
sampled
sampled
Users sampled
% sampled
PSPeto
53.0
2003
ibo Users sampled
sampled
Users sampled
sampled
Interaction RNS
Head User
SVP-CF User
User RNS
DISTILL-CE
Figure I: Performance of 20-AB with the amount of users (log-scale) sampled according different sampling strategies over the HR&10 and PSP@10 metrics. Results for the Netflix dataset have been clipped due to memory constraints. Results for the remaining metrics can be found in Appendix B.6, Figure 13.
NUCGeLLO 10
Hieer
1.3
16.3
2003
2005
20.0
3006
20.0
105.0
%
Users sampled
Users sampled
Users sampled
Users sampled
Interaction RNS
Head User
SVP-CF User
User RNS
DISTILL-CE
Figure 2: Performance of the EASE model trained on different amounts of users (log-scale) sampled by different samplers on the ML-1M dataset.
Does Ac-AE E outperform existing methods? We compare the performance of 3c-AE with various baseline and competitor methods in Table 1. We also include the results of training oc-AE on data synthesized by DISTILL-CF with additional constraint of having budget of only μ 500 synthetic users. For the sake of reference, for our largest dataset (Netflix), this equates to a mere 0.1% of the total users. There
few prominent observations from the results in Table 1. First, o0-AE significantly outperforms SoTA recommendation algorithms despite having only a single fully-connected layer, and also being much simpler to train and implement. Second, we note that AE-AE trained on just 500 users generated by DiSTILL-CF is able to attain 96 105% of co-EE's performance on the full dataset while also outperforming all competitors trained on the full dataset.
How sample efficient is co-AE? Having noted from Table that oc-AE is able to outperform all SoTA competitors with little 0.1 %% of the original users, now aim better understand oo-AE's sample complexity, iee. the amount of training data co-AE needs in order to perform accurate recommendation. In addition to DISTILL-CF, we use the following popular heuristics for down-sampling (more details in Appendix B.2): interaction random negative sampling (RNS); user RNS; head user sampling: and a coreset construction technique, SVP-CF user [57]. We then train co-AE on sampled data for different sampling budgets, while evaluating on the original test-set. We plot the performance for all datasets computed over the HR 10 and @SP@ 10 metrics in Figure 1. We observe that while all heuristic sampling strategies tend to be closely bound to the identity line with slight preference to user RNS, oo-AE when trained on data synthesized by DISTILL-CF tends to quickly saturate in terms of performance when the user budget is increased, even on the log-scale. This indicates DISTILI--FF' superiority in generating terse data summaries for oo-AF, thereby allowing it to get SoTA performance on the largest datasets with as little 500 users.
7

---

80% Users
40% Users
10% Users
5% Users
2% Users
% drop in HR810
% Noise
% Noise
% Noise
Noise
% Noise
Head User
SVP-CF User
User RNS
DISTILL. CF
Figure 3: Performance drop of the EASE model trained on data sampled by different sampling strategies when there is varying levels of noise in the data. Performance drop is relative training on the full, noise-free ML-1MM dataset. Results for the remaining metrics can be found in Appendix B.6, Figure 14.
ML-1M
ML-1M (0.5% noise)
ML-1M (1% noise)
ML-1M (2% noise)
ML-1M (5% noise)
Hoejo
13
100
12
100
100
55
100
106
Users sampled
N Users sampled
Users sampled
% Users sampled
Users sampled
User-RNS FASE
User-RNS RNS oC-AE
DISTIL-CF FASE
DiSTIa-CC oc-AF
Figure 4: Performance of co-AE data sampled by DISTILL-CF and User-RNS when there is noise in the data. Results for EASE have been added for reference. All results are on the ML-IM IL dataset. Results for the remaining metrics can be found in Appendix B.6, Figure 15.
How transferable are the data summaries synthesized by DISTILL--FF? order to best evaluate the quality and universality of data summaries synthesized by DISTILL-CF, we train and evaluate EASE [59] on data synthesized by DISTILL-CF. Note that the inner loop of DISTILL-CE CF still consists of oc-AE, but we nevertheless train and evaluate EASE to test the synthesized data's universality. We re-use the heuristic sampling strategies from the previous experiment for comparison with DISTILL- CF. From the results in Figure 2, we observe similar scaling laws as oc-EE's for the heuristic samplers as well as DISTILL-CF. The semantically similar results for MVAE [32] are presented in Appendix B.6, Figure 10 for completeness. This behaviour validates the re-usability of data summarics generated by DISTILL-CEF because they transfer well to SoTA fintt-width models, which not involved in DISTILL-CFT user synthesis optimization.
How robust are DISTILL- CF and xo-AE to noise? User feedback data is often noisy due to various biases (see [10] for detailed review). Furthermore, due to the significant number of logged interactions in these datasets, recommender systems are often trained on down-sampled data in practice. Despite this, to the best of our knowledge, there is no prior work that explicitly studies the interplay between noise in the data and how sampling it affects downstream model performance. Consequently, we simulate simple experiment: we add 3% noise in the original train-set -> sample the noisy training data - train recommendation models on the sampled data - evaluate their performance on the original, noise-free test-set. For the noise model, we randomly flip a% of the total number of items in the corpus for each user. In Figure 3, we compare the drop in HR @ 10 the EASE model suffers for different sampling strategies when different levels of noise are added to the MovieLens- 1M dataset [15]. We make a few main observations: (1) unsurprisingly, sampling noisy data compounds the performance losses of learning algorithms: (2) DISTILL-CF has the best trade-off compared to other sampling strategies, with an increasing inject more noise into the original data; and (3)
poise:samlling:perfoome performance gap relative other samplers as we down-sample noisy data more aggressively, head user sampling improves relative to other samplers, simply because these head users are the least affected by our noise injection procedure.
Furthermore, to better understand oc-AE's denoising capabilities, we repeat the aforementioned noise-injection experiment but now train AE on down-sampled, noisy data. In Figure 4, we track

---

the change in co-AE's performance as function of the number of users sampled, and the amount of noise injected before sampling. We also add the semantically equivalent results for the EASE model for reference. Firstly, we note that the full-data performance-gan gap between co-AF: and EASE significantly increases when there is more noise in the data, demonstrating 20-AEss robustness to noise, even when its not specifically optimized for it. Furthermore, looking at the 5% noise injection scenario, we notice two counter-intuitivy observations: (I) training EASE on tiny data summaries synthesized by DISTILL-CF is better than training it on the full data; and (2) solely looking at data synthesized by DISTILL-CF for EASE, we notice the best performance when we have lower user sampling budget. Both of these observations call for more investigation of data-centric viewpoint to recommendation, i.e., focusing more on the quality of collected data rather than its quantity.
Applications to continual learning. Continual learning (see [45] for detailed review) is an important area for recommender systems, because these systems are typically updated regular intervals. A continual learning scenario involves data that is split into multiple periods, with the predictive task being: given data until the ش period, maximize algorithm performance for prediction on the + 1)th period. ADER [35] is SoTA con- tinual learning model for recommender systems, that injects the most informative user sequences from the last period to
9.0
8.8-
HR020
8.6
8.4-
8.2
50
1000
150
200
# Added users per Period
Joint
ADER
combat the catastrophic forgetting problem [52]. An intuitive application for DISTILI.-CF is to synthesize succinct data sum- maries of the last period and inject these instead. To compare these approaches, we simulate continual learning scenario by splitting the MovieLens- IM dataset into 17 equal sized epochs, and perform experiments with MVAE [32] for each period. Note that in DISTILL-CF, we still use oo-AE to synthesize data summaries (inner loop). We also compare with two baselines: (1) Joint: concatenate the data from all periods before the current; and (2) Individual: use the data only from the current period. As can from Figure 5, DISTILL-CF consistently outperforms ADER and the baselines, again demonstrating its ability to generate high-fidelity data summaries.
Indiv.
DISTILL-CP
Figure 5: DISTILL-CF for contin- ual learning.
Conclusion & Future Work
In this work, we proposed two complementary ideas: AO-AE, an infinite- autoencoder for modeling recommendation data, and DISTILL-CF for creating tiny, high-fideligg data summaries of massive datasets for subsequent model training. To our knowledge, our work is the first to employ and demonstrate that infitite-width neural networks can beat complicated SoTA models on recommendation tasks. Further, the data summaries synthesized through DISTILL-CF outperform generic samplers and demonstrate further performance gains for co-AE as well finite-width SoTA models despite being trained on orders of magnitude less data.
Both our proposed methods are closely linked with one another: oo- AE'AEs closed-loop formulation especially crucial in the practicality of DISTILL-CF, whereas DISTILL-CFT ability to distill the entire dataset's 's knowledge into small summaries helps AO to scale to large datasets. Moreover, the Gumbel sampling trick enables us to adapt data distillation techniques designed for continuous, real-valued, dense domains to heterogeneous, semi- stuuturtrd and sparse domains like recommender systems and graphs. We additionally explore the strong denoising effect observed with DISTILI-CFF, noting that in the case of noisy data, models trained on considerably less data synthesized by DISTILL- CF perform better than the same model trained on the entire original dataset. These observations lead us to contemplate much larger, looming question: Is more data what you need for recommendation? Our results call for further investigation on the data-centric viewpoint of recommendation.
The findings of our paper open up numerous promising research directions. First, building such closed-form
infinite networks beneficial for various downstream practical applications like search, sequential recommendation, or CTR prediction. Further, the anonymization achieved by synthesizing fake data summaries crucial for mitigating the privacy risks associated with confidential or PII datasets. Another direction is analyzing the environmental impact and reduction in carbon footprint as our experiments show that models can achieve similar performance gains when trained on much less data.

---

References
[1] Vito Walter Anelli, Alejandro Bellogin, Tommaso Di Noia, Dietmar Jannach, and Claudio Pomo. Top-n recommendation algorithms: quest for the state-of-the-art. arXiv preprint arXiv:2203.01 155, 2022.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang- On exact computation with an infinitely wide neural net, Advances Neural Information Processing Systems, 32, 2019.
[3] Sanjeev Arora, Simon Du, Zhiyuan Li, Ruslan Salakhutdinov. Ruosong Wang, and Dingli Yu. Harnessing the power of infinitely wide deep nets on small-data tasks. In International Conference on Learning Representations, 2020.
[4] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal, Reconciling modern machine learning practice and the bias-variance trade-off. arXiv preprint arXiv:1812.11118 2011.
[5] James Bennett and Stan Lanning. The netflix prize. 2007.
[6] Jeff Bilmes. Submodularity in machine learning and artificial intelligence. arXiv preprint arXiv:2202.00133 2022.
[7] Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming, Advances in Newral Information Processing Systems, 33: 1-8944944899,, 2020.
[8] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
[9] Dong-Ke Kyu Chac, Jihoo Kim, Duen Homg Chau, and Sang-Wook Wook Kim. Ar-cf: Augmenting virtual users and items in collaborative filtering for addressing cold-start problems. In Proceedings the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, page 1251-1260, New York, NY, USA, 2020. Association for Computing Machinery.
[10] Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. CoRR, abs/2010.03240,, 2020.
[11] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pages 7-10, 2016.
[12] Don Coppersmith and Shmuel Winograd. Matrix multiplication via arithmetic progressions. In Proceedings of the nineteenth annual ACM symposium Theory of computing, pages 1-6, 1987.
[13] M. Dacrema, P. Cremonesi, and D. Jannach. Are we really making much progress? worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM Conference on Recommender Systems, RecSys 119, 2019.
[14] Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks: The power of Initialization and dual view on expressivity. Advances neural information processing systems, 29, 2016.
[15] Maxwell Harper and Joseph A Konstan, The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiss) 2015.
[16] Trevor Hastie, Robert Tibshirani, Jerome Friedman, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
[17] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 339-648, 2020.
[18] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th International Conference World Wide Web, WWW '17, 2017.
[19] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
[20] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernl:: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
10

---

[21] H. Jain, V. Balasubramanian, B. Chunduri, and M. Varma. Slice: Scalable linear extreme classifiers trained 100 million labels for related searches. In Proceedings the 12th ACM Conference on Web Search and Data Mining, 2019.
[22] H. Jain, Y. Prabhu, and M. Varma. Extreme multi-label loss functions for recommendation, tagging. ranking and other missing label applications. In Proceedings of the SIGKDD Conference on Knowledge Discovery and Data Mining, 2016.
[23] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparamcterization with gumbel-softmax. Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[24] Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, and Neil Shah. Graph condensation for graph neural networks. In International Conference on Learning Representations, 2022.
[25] Yannis Kalantidis, Mert Bulent Sariyildiz,, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems, 33:21798-21809, 2020.
[26] Ehsan Kazemi, Shervin Minaee, Moran Feldman, and Amin Karbasi. Regularized submodular maximiza- tion at scale. Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5356-5366. PMLR, 18 24 Jul 2021.
[27] Krishnateja Killamsetty, Durga S. Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match Gradient matching based data subset selection for efficient deep model training. Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5464-5474. PMIR, 18-24 Jul 2021.
[28] Thomas N. Kipf and Max Welling, Semi- Superviiee Classification with Graph Convolutional Networks. In Proceedings of the 5th International Conference on Learning Representations, ICLR '17, 2017.
[29] Taeyong Kong, Taeri Kim, Jinsung Jeon, Jeongwhan Choi, Yeon-Chang Lee, Noseong Park, and Sang- Wook Kim. Linear, or non-linear, that is the question! In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 517-525, 2022.
[30] Walid Krichene and Steffen Rendle. On sampled metrics for item recommendation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, 2020.
[31] Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein Finite versus infinite neural networks: empirical study. Advances in Neurul Information Processing Systems, 112115556-11777 2020.
[32] Dawen Liang, Rahul G. Krishnan, Matthew D. Hoffman. and Tony Jebara. Variational autoencoders for collaborative filtering. In Proceedings of the 2018 World Wide Web Conference, WWW '1, 2018.
[33] Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang, and Wenwu Zhu. Learning disentangled representations for recommendation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' ('lché-BBcc, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[34] Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In International conference on machine learning, pages 2113-2122. PMLR, 2015.
[35] Fei Mi, Xiaoyu Lin, and Boi Faltings. Ader: Adaptively distilled exemplar replay towards continual learning for session-based recommendation. In Fourteenth ACM Conference on Recommender Systems. page 408 413, New York, NY, USA, 2020. Association for Computing Machinery.
[36] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint 1rXiv:1411.178 2014.
[37] A. Mittal, N. Sachdeva, Agrawal, Agarwal, Kar, and M. Varma. Eclare: Extreme classification with label graph correlations. In Proceedings of The ACM International World Wide Web Conference, 2021.
[38] Arvind Narayanan and Vitaly Shmatikov. Robust de anonymization of large sparse datasets. In 2008 IEEE Symposium on Security and Privacy (sp 2008), pages 111-125, IEEE, 2008
11

---

[39] Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman. Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krishnamoorthi. Ansha Yu, Volodymyr Kondratenko, Stephanie Pereira, Xianjic Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong, and Misha Smelyanskiy. Deep learning recommendation model for personalization and recommendation systems. CoRR, abs/1990.000099 2019.
[40] Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. In International Conference Learning Representations, 2021.
[41] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34, 2021.
[42] Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labelt reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP. JJNLPP), 2019.
[43] Roman Novak, Lechao Xiao. Yasaman Bahri, Jaehoon Lee, Greg Yang. Daniel A. Abolatia, Jeffrey Pennington, and Jascha Sohl-dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. In International Conference on Learning Representations, 2019.
[44] Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. In International Conference on Learning Representations, 2020.
[45] German Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks. 113:54-71, 2019.
[46] Y. Prabhu, A. Kag. S. Harsola, R. Agrawal, and M. Varma. Parabel: Partitioned label trees extreme classification with application to dynamic search advertising. In Proceedings of the International World Wide Web Conference, 2018.
[47] Adityanarayanan Radhakrishnan, Lecture 5: Ntk origin and derivation. 2022.
[48] Adityanarayanan Radhakrishnan, George Stefanakis, Mikhail Belkin, and Caroline Uhler. Simple, fast, and flexible framework for matrix completion with infinite width neural networks. Proceedings of the National Academy of Sciences, 119(16)=21 (1116))22110641111 1150641199 19, 2022.
[49] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme Bpr: Bayesian personalized ranking from implicit feedback. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI '09, 2009.
[50] Steffen Rendle, Walid Krichene, Li Zhang, and John Anderson. Neural collaborative filtering matrix factorization revisited. In Fourteenth ACM conference on recommender systems, pages 240-248, 2020.
[51] Joshua David Robinson, Ching- Chuang. Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. International Conference on Learning Representations, 2021.
[52] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning, Advances in Neural Information Processing Systems, 32, 2019.
[53] Durga S, Rishabh Iyer, Ganesh Ramakrishnan, and Abir De. Training data subset selection for regression with controlled generalization error. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research. pages 9202-9201111 92122 PMLR. 18 24 Jul 2021.
[54] Noveen Sachdeva, Giuseppe Manco, Ettore Ritacco, and Vikram Pudi. Sequential variational autoencoders for collaborative filtering. In Proceedings of the ACM International Conference on Web Search and Data Mining, WSDM '19, 2019.
[55] Noveen Sachdeva and Julian McAuley, How useful are reviews for recommendation? a critical review and potential improvements. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, 2020.
[56] Noveen Sachdeva, Su, and Thorsten Joachims. Off ooflpolicy bandits with deficient support. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, 2020.
12

---

[57] Noveen Sachdeva, Carole-Jean Wu, and Julian McAuley. On sampling collaborative filtering datasets. In Proceedings of the Fifleenth ACM International Conference Web Search and Data Mining, WSDM '22, page 842-850, New York, NY, USA, 2022. Association for Computing Machinery.
[58] T. Schnabel, A. Swaminathan, A. Singh, N. Chandak, and T. Joachims. Recommendations as treatments: Debiasing learning and evaluation. In Proceedings of The 33rd International Conference on Machine Learning, 2016.
[59] Harald Steck. Embarrassingly shallow autoencoders for sparse data. In The World Wide Web Conference. pages 3251-3257. 2019.
[60] Eliza Strickland. Andrew ng: Farewell, big data. IEEE Spectrum, Mar 2022.
[61] M. Toneva, A. Sordoni. R. Combes, A. Trischler, Y. Bengio, and G. Gordon. An empirical study of example forgetting during deep neural network learning. In ICLR, 2019.
[62] Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint 2018.
[63] Carole-Jean Wu, Robin Burke, Ed Chi, Joseph Konstan, Julian McAuley, Yves Raimond, and Hao Zhang. Developing recommendation benchmark for miperf training and inference. CoRR, abs/2003.07 207700030036,,.. 2020.
[64] Yao Wu, Christopher DuBois, Alice Zheng, and Martin Ester. Collaborative denoising auto auto- encoddrs for top-n recommender systems. In Proceedings of the ninth ACM international conference on web search and data mining, pages 153 162, 2016.
[65] Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar. and Kannan Achan. Rethinking neural vs. matrix-factorizztioo collaborative filtering: the theoretical perspectives. In International Conference on Machine Learning, pages 1 11514-11524. PMLR, MMLR, 2021.
[66] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pages 23674-12685 PMLR, 2021.
[67] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2021.
[68] Wenqing Zheng, Edward W Huang, Nikhil Rao, Sumeet Katariya, Zhangyang Wang, and Karthik Sub- bian. Cold brew: Distilling graph node representations with incomplete missing neighborhoods. In International Conference on Learning Representations, 2022.
[69] Feng Zhu, Chaochao Chen, Yan Wang, Guanfeng Liu, and Xiaolin Zheng. Dtedr: A framework for dual-target cross-domain recommendation. In Proceedingy of the 28th ACM International Conference on Information and Knowledge Management, pages 1533-1542, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We discuss both c--AE and DISTILL-CF in detail in Section and Section 4, along with validating their performance through experiments in Section 5
(b) Did you describe the limitations of your work? [Yes] We talked about the lack of scalability of using xc-AE naïvely in Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
13

---

(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as URL)? [Yes] Discussed throughout the setup sub-section in Section 5, and provided detailed instructions in Appendix B
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Discussed in the setup sub-section in Section 5, and provided hyper-parameters in Appendix B.3
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] Noted the seed we used in our experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Discussed in the training details subsection in Section
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
Released our anonymous code mentioned in Section 5
(d) Did you discuss whether and how consent was obtained from people whose data you're ro
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "kdAknoshysl", "reviewer": "Reviewer_3xoT", "summary": "This paper develops an infinite-width autoencoder for recommendation. The proposed recommendation model contains a single hyper-parameter and only needs a closed-form solution. Leveraging -AE's simplicity, this paper develops Distill-CF for large data. According to the experiments on several datasets, the proposed model shows better performance. ", "questions": "N/A.", "limitations": "N/A.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths.\nThe proposed model is novel and interesting. \nCompared with several baselines, the proposed model performs better. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hna1LzPld76", "reviewer": "Reviewer_ScS5", "summary": "This paper leverages Neural Tangent Kernel and demonstrates the effectiveness of infinite-width autoencoders in recommendation systems. Furthermore, the authors also propose a data distillation framework to synthesize a small support dataset which aims to match the performance of algorithms trained on the full dataset via the bilevel optimization. Experiments are conducted on four public datasets to evaluate the performance of infinite-width autoencoders and the data distillation framework. ", "questions": "Please refer to the detailed comments", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Provide an interesting view in enhancing recommendation systems by synthesizing a small but informative dataset, which could inspire more related work.\n2. The proposed approach shows promising performance in improving recommendation performance, and meanwhile it is more robust to noise compared to SOTA.\n\nWeaknesses:\n1. To enhance the readability of the paper, it is highly recommended to provide more background information and preliminary content on NTK and KRR to help the audiences better understand the proposed approach as these are pretty new topics in recommendation systems. \n2. If Distill-CF is claimed to be an general data distillation framework effective for CF datasets, it would be necessary to show how it performs while with other collaborative filtering models besides $\\infty$-AE, e.g.,  how the synthesized data summaries work with other autoencoder-based recommendation systems?\n3. The numbers of users are on different scales (e.g., 3k vs 476k). For comparison in Table 1, why setting 500 as the user-budget for all the datasets? Is it setting the user-budget based on the percentage of user number can help the audiences better understand the results?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "S7jx3tgUQSL", "reviewer": "Reviewer_s1k3", "summary": "This paper proposed an autoencoder model where Neural Tangent Kernel is applied to estimate an infinitely-wide neural model named $\\infty$-AE. Theoretically, infinitely-wide networks could bring highly expressive performance but still struggle with the large size of data in the recommendation. The author then proposed a learnable method for dataset summarization to tackle the problem. A lot of experiments are conducted to prove that the sampling method is effective and robust.", "questions": "In addition to the weakness mentioned above, the following questions are expected to answer. \n\n- It's a little confusing that \"the synthesized data was never be optimized (in the inner-loop of DISTILL-CF)\" in line 279. Does it mean that $X^s$ in equation 3 is never optimized (but it notes the \"inner-loop\", which may indicate that the EASE is never optimized according to the definition of inner loop in line 185)? Maybe it should be \"outer-loop\" here. Perhaps you mean that the data distilled from well-trained Distill-CF can train any different models.\n- In figure 3, is there any reason why the SVP-CF method suffers the most severe drop?\n- In figure 4, the vertical axis is set as HR@10 instead of 'drop in HR@10' used in figure 3. It is not so convincing that $\\infty$-AE is more robust than EASE, maybe the drop in $\\infty$-AE is higher than EASE. And only RNS and Distill-CF methods are used, will $\\infty$-AE get similar results when using \"Head User\" or \"SVP-CF\" method? \n- In continual learning, the data from previous periods should be unreachable again, so the \"joint\" method can not be regarded as continual learning.\n- It is better to explain how to combine MVAE with Distill-CF in continual learning experiments. Is that similar to the setting in figure 2, where the synthesized data is never optimized?\n- In figure 5, the ADER method gets even worse performance than individual methods, which is not so convincing. \n- In figure 6, the results on dataset ml-1m are totally different from another two datasets, where the deeper network gets much better performances. \n- In Appendix A Algorithm 2, the input $\\hat{X}_u$ should be noted in Input part: which may be \"history vector of user to be predicted\".\n", "limitations": "Not discussed ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n    - The paper applied NTK method to gain an infinitely-wide networks which is more expressive.\n    - The proposed data summarization method is actually a model-agnostic method, where the model is used to predict the user-item score in order to optimize the prior-matrix $X^S$.\n   - The combination of $\\infty$-AE and Distill-CF provide a simple but effective method to get a tiny model without loss of performance.\n\n\n- Weakness\n   - The Distill-CF method could be combined with different neural models which can be optimized alternately, such as NeuMF, EASE, LightGCN et al. However, the author didn't try it to verify the general effectiveness of Distill-CF.\n   - Many baselines may not fined-tuned. For example, the baseline EASE is not fine-tuned, since the regularization coefficient is of vital importance. \n   - The reason why the proposed method works well when only a small number of summarized data is used is not explained from a theoretical perspective. \n   - Why the use of summarized data can even perform better than the use of all data in many cases?\n  - The formulation of Gumbel softmax should be better described, otherwise, it is difficult to follow.\n  - Several hyper-parameters should be fine-tuned in the proposed methods.\n  - How to do inference? Only use the summarized data? \n  - It is challenging to optimize the summary of the dataset, due to the use of the hard tanh function, and Gumbel sampling trick as well as the inverse of a parameter-dependent square matrix.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6e-JVGVJYc", "reviewer": "Reviewer_NjNV", "summary": "The paper studies a novel recommendation framework which combines two complementary ideas. The infinite-width autoencoder, $\\infty$-AE, models the recommendation data, while DISTILL-CF creates a small set of data \"summaries\" (synthetic examples constructed from the real ones) used further for model training. The presented empirical results look very promising, and the introduced ideas are worth attention. ", "questions": "As this is an additional review, I do not have any direct questions to the authors. Nevertheless, I would like to give some recommendations:\n- Try to improve writing to make the paper less dense, \n- Properly introduce, define, and cite important concepts,\n- Properly justify your choices and claims,\n- Extend the discussion on performance on tail items,\n- Try to answer the question, why the small number of synthetic instances suffices to obtain good results,\n- Check the code for possible bugs in the experiments.", "limitations": "- Limited experimentation with tail items\n\nAlthough Table 1 and Figure 1 report the results for a metric somehow suited for tail items (PSP might be used for this purpose, but in general this is not the best choice because of its flaws; see, for example, \"On Missing Labels, Long-tails and Propensities in Extreme Multi-label Classification\", KDD 2022), the other experiments report HR@k which rather focuses on head items. I could overlook something, but an extended and more precise discussion should be given in the final version of the paper\n\n- Self-criticism  \n\nI did not find an honest discussion about the flaws of the method. I would appreciate a more critical thinking from the authors\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- A though-provoking and novel (according to my knowledge) framework for recommendation combining relatively new, state-of-the-art concepts such as infinite-width networks, data summaries, and Gumbel sampling trick,\n- Promising empirical results in terms of predictive and computational performance,\n- Relatively wide experimental studies.\n\nWeaknesses\n- The paper is quite dense, therefore does not read very well; one can be easily lost,\n- The paper is not self-contained; many concept are not properly introduced or cited (e.g., dual activation is neither defined nor cited; if not wrong, this paper should be used as reference: \"Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity\"),\n- An extended discussion around the question \"Is more data what you need for recommendation?\" should be given in the paper. The results given in Paragraph \"How robust are DISTILL-CF and $\\infty$-AE to noise?\" might be an effect of a bug in the experiments. Another reason could be the focus on head items as HR@k rather ignores tail items. Figure 1 reports promising results for PSP@k, but this might not be the best metric to use for tail items (see below). \n\nMinor comments:\n- \"infinitely-wide bottleneck layers\" sounds contradictory: This design choice needs more discussion to justify the approach. Is the regularization \"responsible\" for the bottleneck?\n- It is written: \"We can subsequently perform inference for any novel user as follows (...)\". The novel users will have no history, so their representation will be empty. This part needs additional comments.\n- I am not familiar with the recent sota approaches for recommendation, so it might be that the experimental results are less impressive than given,\n- Function $f$ in (1) should be properly defined to make it clear that its value is a vector,\n- $D^{val}$ seems to be not defined,\n- Use \"\\log\" in math formulas for logistic function.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept