Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies a novel recommendation framework which combines two complementary ideas. The infinite-width autoencoder, $\infty$-AE, models the recommendation data, while DISTILL-CF creates a small set of data "summaries" (synthetic examples constructed from the real ones) used further for model training. The presented empirical results look very promising, and the introduced ideas are worth attention. 
strengths_and_weaknesses: Strengths:
- A though-provoking and novel (according to my knowledge) framework for recommendation combining relatively new, state-of-the-art concepts such as infinite-width networks, data summaries, and Gumbel sampling trick,
- Promising empirical results in terms of predictive and computational performance,
- Relatively wide experimental studies.

Weaknesses
- The paper is quite dense, therefore does not read very well; one can be easily lost,
- The paper is not self-contained; many concept are not properly introduced or cited (e.g., dual activation is neither defined nor cited; if not wrong, this paper should be used as reference: "Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity"),
- An extended discussion around the question "Is more data what you need for recommendation?" should be given in the paper. The results given in Paragraph "How robust are DISTILL-CF and $\infty$-AE to noise?" might be an effect of a bug in the experiments. Another reason could be the focus on head items as HR@k rather ignores tail items. Figure 1 reports promising results for PSP@k, but this might not be the best metric to use for tail items (see below). 

Minor comments:
- "infinitely-wide bottleneck layers" sounds contradictory: This design choice needs more discussion to justify the approach. Is the regularization "responsible" for the bottleneck?
- It is written: "We can subsequently perform inference for any novel user as follows (...)". The novel users will have no history, so their representation will be empty. This part needs additional comments.
- I am not familiar with the recent sota approaches for recommendation, so it might be that the experimental results are less impressive than given,
- Function $f$ in (1) should be properly defined to make it clear that its value is a vector,
- $D^{val}$ seems to be not defined,
- Use "\log" in math formulas for logistic function.
questions: As this is an additional review, I do not have any direct questions to the authors. Nevertheless, I would like to give some recommendations:
- Try to improve writing to make the paper less dense, 
- Properly introduce, define, and cite important concepts,
- Properly justify your choices and claims,
- Extend the discussion on performance on tail items,
- Try to answer the question, why the small number of synthetic instances suffices to obtain good results,
- Check the code for possible bugs in the experiments.
limitations: - Limited experimentation with tail items

Although Table 1 and Figure 1 report the results for a metric somehow suited for tail items (PSP might be used for this purpose, but in general this is not the best choice because of its flaws; see, for example, "On Missing Labels, Long-tails and Propensities in Extreme Multi-label Classification", KDD 2022), the other experiments report HR@k which rather focuses on head items. I could overlook something, but an extended and more precise discussion should be given in the final version of the paper

- Self-criticism  

I did not find an honest discussion about the flaws of the method. I would appreciate a more critical thinking from the authors
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper develops an infinite-width autoencoder for recommendation. The proposed recommendation model contains a single hyper-parameter and only needs a closed-form solution. Leveraging -AE's simplicity, this paper develops Distill-CF for large data. According to the experiments on several datasets, the proposed model shows better performance. 
strengths_and_weaknesses: strengths.
The proposed model is novel and interesting. 
Compared with several baselines, the proposed model performs better. 


questions: N/A.
limitations: N/A.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposed an autoencoder model where Neural Tangent Kernel is applied to estimate an infinitely-wide neural model named $\infty$-AE. Theoretically, infinitely-wide networks could bring highly expressive performance but still struggle with the large size of data in the recommendation. The author then proposed a learnable method for dataset summarization to tackle the problem. A lot of experiments are conducted to prove that the sampling method is effective and robust.
strengths_and_weaknesses: - Strengths
    - The paper applied NTK method to gain an infinitely-wide networks which is more expressive.
    - The proposed data summarization method is actually a model-agnostic method, where the model is used to predict the user-item score in order to optimize the prior-matrix $X^S$.
   - The combination of $\infty$-AE and Distill-CF provide a simple but effective method to get a tiny model without loss of performance.


- Weakness
   - The Distill-CF method could be combined with different neural models which can be optimized alternately, such as NeuMF, EASE, LightGCN et al. However, the author didn't try it to verify the general effectiveness of Distill-CF.
   - Many baselines may not fined-tuned. For example, the baseline EASE is not fine-tuned, since the regularization coefficient is of vital importance. 
   - The reason why the proposed method works well when only a small number of summarized data is used is not explained from a theoretical perspective. 
   - Why the use of summarized data can even perform better than the use of all data in many cases?
  - The formulation of Gumbel softmax should be better described, otherwise, it is difficult to follow.
  - Several hyper-parameters should be fine-tuned in the proposed methods.
  - How to do inference? Only use the summarized data? 
  - It is challenging to optimize the summary of the dataset, due to the use of the hard tanh function, and Gumbel sampling trick as well as the inverse of a parameter-dependent square matrix.

questions: In addition to the weakness mentioned above, the following questions are expected to answer. 

- It's a little confusing that "the synthesized data was never be optimized (in the inner-loop of DISTILL-CF)" in line 279. Does it mean that $X^s$ in equation 3 is never optimized (but it notes the "inner-loop", which may indicate that the EASE is never optimized according to the definition of inner loop in line 185)? Maybe it should be "outer-loop" here. Perhaps you mean that the data distilled from well-trained Distill-CF can train any different models.
- In figure 3, is there any reason why the SVP-CF method suffers the most severe drop?
- In figure 4, the vertical axis is set as HR@10 instead of 'drop in HR@10' used in figure 3. It is not so convincing that $\infty$-AE is more robust than EASE, maybe the drop in $\infty$-AE is higher than EASE. And only RNS and Distill-CF methods are used, will $\infty$-AE get similar results when using "Head User" or "SVP-CF" method? 
- In continual learning, the data from previous periods should be unreachable again, so the "joint" method can not be regarded as continual learning.
- It is better to explain how to combine MVAE with Distill-CF in continual learning experiments. Is that similar to the setting in figure 2, where the synthesized data is never optimized?
- In figure 5, the ADER method gets even worse performance than individual methods, which is not so convincing. 
- In figure 6, the results on dataset ml-1m are totally different from another two datasets, where the deeper network gets much better performances. 
- In Appendix A Algorithm 2, the input $\hat{X}_u$ should be noted in Input part: which may be "history vector of user to be predicted".

limitations: Not discussed
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper leverages Neural Tangent Kernel and demonstrates the effectiveness of infinite-width autoencoders in recommendation systems. Furthermore, the authors also propose a data distillation framework to synthesize a small support dataset which aims to match the performance of algorithms trained on the full dataset via the bilevel optimization. Experiments are conducted on four public datasets to evaluate the performance of infinite-width autoencoders and the data distillation framework. 
strengths_and_weaknesses: Strengths:
1. Provide an interesting view in enhancing recommendation systems by synthesizing a small but informative dataset, which could inspire more related work.
2. The proposed approach shows promising performance in improving recommendation performance, and meanwhile it is more robust to noise compared to SOTA.

Weaknesses:
1. To enhance the readability of the paper, it is highly recommended to provide more background information and preliminary content on NTK and KRR to help the audiences better understand the proposed approach as these are pretty new topics in recommendation systems. 
2. If Distill-CF is claimed to be an general data distillation framework effective for CF datasets, it would be necessary to show how it performs while with other collaborative filtering models besides $\infty$-AE, e.g.,  how the synthesized data summaries work with other autoencoder-based recommendation systems?
3. The numbers of users are on different scales (e.g., 3k vs 476k). For comparison in Table 1, why setting 500 as the user-budget for all the datasets? Is it setting the user-budget based on the percentage of user number can help the audiences better understand the results?

questions: Please refer to the detailed comments
limitations: N/A
review_title: null

Meta-review: metareview: 
This paper proposes an infinite-width autoencoder for recommendation, which trained using the NTG framework by the kernelized ridge regression algorithm. The approach struggle with the large size of the data. To this end, the authors propose a method for data set summarization, called Distill-CF, for synthesizing tiny high-fidelity data summaries.


The paper received a mixed evaluation from the reviewers.

The strengths of the paper mentioned by the reviewers were: 
- A simple model with only one hyper-parameter and a closed-form solution
- A though-provoking and novel framework for recommendation
- Good performance in the experiments, relatively wide experimental study

On the other hand, the identified weaknesses were:
- The author did not verify the general effectiveness of Distill-CF beyond coupling with infinite AE, so it is not clear where is the actual gain
- Technical issues with the experiments
- Some issues with the readability
confidence: Less certain
Decision: Accept