Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work proposes a new approach, based on projective clustering, for compressing the embedding layers of DNNs for natural language modeling tasks. The authors show that the trade-off between compression and model accuracy can be improved by considering a set of k subspaces rather than just a single subspace. Methods for compressing DNNs is an active area of research and this paper presents a promising approach to do so as well as interesting results. 

Rating: The paper presents interesting ideas for compressing embedding layers. However, since this is an empirical paper, I would expect a more comprehensive set of empirical results and a better comparison with other related methods. Overall, the paper seems not very mature in its current form, hence my rating is 'Ok but not good enough - rejection'.

Pros
----
* The proposed method is appealing due to its simplicity and the idea of considering multiple subspaces for embedding is plausible in the context of compressing embedding matrices of NLP models.

* The results show improvements as compared to using just a single subspace. 

* The framework provides several ideas for future works. 

Cons
-----
* Typically, the SVD takes the form A = UDV, where U and V are the left and right singular vectors and the diagonal entries of D are the singular values. From the discussion it is not clear whether you factor the singular values into U, or whether you simply ignore the singular values? Also, how do you enforce the orthogonality constraints on U and V during the fine tuning stage? Have you considered a simpler low-rank factorization A = EF in your experiments, where no orthogonality constraints on E and F are imposed?

* It would be good to see the progression for k={2,3,4,5} in Figure 5 and 6. Further, the ensemble approach in Figure 6 hasn't been discussed in detail anywhere in the paper. It is not exactly clear to me how you are computing the ensemble.

* It would be very helpful to see some Tables that shows the total number of weights, accuracy, k, j, etc., in order to better understand the performance.  

* How do you determine k and j in practice? Are you using some heuristic or are you simply doing a grid search?

* I would like to see how your method compares to ALBERT and whether a modified ALBERT (as you suggest in your future work section) is doing better.
 
* I would be interesting to see if you approach is also useful for compressing a fully connected layer in different settings. This should be easy to test and could be reported in the Appendix. 

Minor comments:
--------------
* It is nice to see that you have many generalizations an extensions in mind, but this section appears very lengthy to me. 

* compression rater -> compression rates
review_title: Interesting work

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper extends the idea of using subspace clustering to compress the neural nets by considering multiple subspaces and projecting each point to its closest subspace. The paper needs more investigation on the related works. Basically, the idea and the technique is not novel. See the related literature below:
[1] Trittenbach, Holger, and Klemens Böhm. "One-Class Active Learning for Outlier Detection with Multiple Subspaces." Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2019.
[2] Liu, Risheng, et al. "Fixed-rank representation for unsupervised visual learning." 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2012.
[3] Xu, Dong, et al. "Concurrent subspaces analysis." 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05). Vol. 2. IEEE, 2005.
[4] Feng, Jianzhou, et al. "Learning dictionary via subspace segmentation for sparse representation." 2011 18th IEEE International Conference on Image Processing. IEEE, 2011.
---------------------------------------------
Pros:
•	Smoothly readable. 
•	The contribution section is described thoroughly and properly. 
•	Providing the codes for reproducing results.
---------------------------------------------
Cons:
Abstract:
•	The abbreviations like NLP or SVD should be defined first, then used. 
•	Assuming that the reader already has corresponding field knowledge about systems such as GLUE, DistilBERT, or RoBERTa and mentioning them in the abstract may be bold.
•	Details of the methods such as the use of Aj matrix or k>1 subspace should not be mentioned in the abstract but rather in the contribution or introduction section accordingly.
•	The last sentence “Open Code for ….” Should not be mentioned in the abstract but in the code description section. 
•	The figures 1-3 in the paper look not well organized, which makes the proposed simple idea to be extremely complex.
Results:
•	It would be better to discuss the comparable results more thoroughly. 
•	Model compression literature should be reviewed and the typical methods should be compared with in the experiments.
Discussion and Conclusion:
•	Only discussion of the results is provided in this section and the conclusion is not provided explicitly. 


Future Work:
•	Better not to start the section with numbered items right away. Better to have a starting sentence first. 
Appendix B
•	Titled results before fine-tuning and includes figures with no explanation. Provide proper description and discussion for each subfigure. 

review_title: Notitle

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper applies projective clustering to the embedding layer of deep networks with large model sizes such as RoBERTa. The idea of finding more than one subspaces to factorize the embedding weight matrix has nice intuition and insights. I vote for accepting.

Strengths:
1. The paper has convincing evidence showing the reduction in percent of accuracy drop when applying projective clustering to the embedding weight vectors.
2. The paper has illustration figures that clearly show the intuition of the approach as well as how the compression is achieved.

Weaknesses:
1. It would be better if more baselines can be included in the experiment comparisons. In particular, Since Step 2-3 of the proposed MESSI pipeline (page 4) is partitioning of all the input neurons and computing SVD for each partition, I would be really interested in seeing the comparison of projective clustering vs simpler clustering methods such as k-means to partition the input neurons, in the evaluation.
2. The authors discussed extensions such as using $L_1$ error and $L_1$ distance, but no experiments were performed for the extensions. Some experiment results will be better to establish the flexibility of the framework of projective clustering in model compression tasks.

Questions during rebuttal period: 
1. Please provide some results regarding the weaknesses above, especially the result of more baseline methods.
2. Is projective clustering the only way to find clusters in multiple subspaces? What are some alternatives? For example, in subspace clustering, all the data points can be projected to the same subspace and form clusters; we may run subspace clustering for multiple times to get clustering results in different subspaces.
review_title: Nice application of projective clustering to model compression

Meta-review: The paper proposes to use projective clustering to compress the embedding layers of DNN. This is a novel interesting idea which can  impact the area of Knowledge distillation. There were some concerns about the empirical study which was addressed to some extent  by the authors during the rebuttal.
Decision: Accept (Poster)