Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a new compression method, Parameter Shared Setup (PSS), and its training method to compress the large-scale recommendation embedding table. The proposed new compression can achieve higher compression rate on larger dataset with almost no decrease in quality and training speed. It can achieve even 10000 × compression on criteo-tb DLRM model, which is much better than previous method. It provides both theoretical and experimental proof.


strengths_and_weaknesses: Strength.
1. The proposed method can compress the embedding table in memory of the order of O(logn). Larger dataset will benifit more from logn. Criteo-tb DLRM model can be compressed 10000 times.
2. Previous methods show that higher compression rate will suffer from logner training time. But in this paper, the larger number of epochs needed for convergence can be compensated by the faster iteration, which keeps the total training time comparable with others.
3. This paper provide both theoretical and experimental explanation for the performance.

Weakness.
The speed experiment needs more comparison  between different compression methods. Now the author only consider the comparison among different compression rate by their own method. More comparison is required to show the speed performance. The dense gradients should be effective on ROBE, if ROBE also requires training and can compress it 1000 times. Now the speed experiment only supports the longer epochs and faster single integration but can not show the advantage of the new methods.
questions: As mentioned above, more speed experiment comparing with other baselines should be included.
limitations: The compression should consider how to increase the number of data point without retraining. That is also valuable especially for recommendation
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper studies how random projections on embedding matrices could lead to heavy compressions of practical deep models. They picked a well-associable Criteo-TB dataset with binary loss functions and discovered that most SOTA baselines such as DLRM could be compressed by 10k times without observable performance degradation. The paper also argue that previous failures on random projection could be due to a lack of sufficient training epochs. Some theoretical background was introduced as well as a few other papers on related implementation details.
strengths_and_weaknesses: Strengths:
* Originality: While Johnson Lindenstrauss Transform tells us that random projections from sparse vectors could just work, showing that the simple idea actually works is a good piece of knowledge to have.
* Originality: The paper is based on well-known datasets and SOTA methods. The results appear associable.

Weaknesses:
* Significance: In my experience, random projections do not tend to work well in retrieval problems with cross-entropy loss. This paper is only limited to binary prediction losses in some variants of the same dataset. The authors may want to discuss limitations of their work, esp. with respect to retrieval problems such as BPR loss in recommender systems or XE loss in masked-language modeling.
questions: Sections 2.1-2.3 on theoretical analysis exist without a purpose. Many things should be discussed. For example:
* Line 107. Norm of recovered embedding from PSS. Why does it matter to have l2-loss in addition to inner-product loss? Do you expect the embedding vectors to be unit-normed?
* Line 123. The appearance of singular value is not motivated. Would scale of the embedding vectors matter? What actual values do you observe in trained models?
* In general, what actual knowledge could a reader gain besides saying that "random projection in general is a good idea"?

I did not go into the details of Section 2.5. According to the authors, the most successful model runs without sparse hashes, so I don't know how I should appreciate Sparse-JL. Also, what does QR decomposition do here?
limitations: The paper did not discuss limitations beyond the presented variants of a single dataset. I would suspect that random projection may not generally work well in retrieval-type of problems, such as search and masked language modeling.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on the problem of compressing large embedding tables in recommender systems. The authors provide a theoretical analysis of the memory requirements and approximation error of compressing embedding tables under the defined parameter sharing setup. They also show the increase of compression rate accelerates the training of each iteration while leading to more iterations until model convergence. The conclusion is that compressed models can be trained in the same overall time and are faster for inference compared with uncompressed ones.
strengths_and_weaknesses: S1. This paper is mostly well-organized and clearly presented. Figures are helpful, e.g. Figure 1 does help to clarify the tradeoffs of training larger embedding tables.

S2. The authors conduct experiments on real-world datasets. The results are sufficient to show the compensation of faster training speed for slower convergence of compressed models.

S3. The conclusions about compressed models are intuitive and are beneficial to the application of compressed models in real systems.

W1. The theoretical analysis of the memory requirements and approximation error is based on the defined parameter sharing setup. It is not clear whether parameter-sharing methods have become state-of-the-art methods of embedding compression. Thus the impact of the conclusions may be limited.

W2. Section 3 introduces several implementation choices for PSS, but the originality is not clear since the choices used by existing PSS methods are not mentioned.

W3. Due to the above two points, the contribution of this paper is not quite substantial and should be better summarized.
questions: Q1. Please see W1, what do you think of it?

Q2. Please see W2, what are the existing implementation choices for PSS methods?

Q3. Please see W3, can you summarize the main contribution of the paper?
limitations: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: Overall, the paper is a good contribution to the model compression research, which achieved 10000x compression using parameter sharing, compared previous best 1000x. Compressing large models is an important direction and I believe this is a good progress. While there were some disagreements on the significance of the contribution, I feel the authors' response reasonably addressed them.
confidence: Certain
Decision: Accept