Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: SVD of matrices in a language model is used to reduce model size. Instead of standard SVD that minimizes the MSE of reconstructing the matrix elements, a modified version that weights elements is used. Specifically, importance of each parameter (matrix element) is estimated based on the magnitude of gradients w.r.t. to that element. To have closed-form solution for weighted SVD, average row weights are used instead of element weights.
main_review: The paper is well-written and easy to follow. 

Strengths:
+ the approach does not require re-training (doing pre-training again)
+ experimental results on several tasks show only a small drop in model quality
+ the approach can further reduce the size of already-reduced-size models (e.g. TinyBERT) 

Weaknesses:
- the reduction in the number of parameters is modest compared to methods that re-train the network in a compact representation, like ALBERT
- Results in Table 2 look incomplete: why is only one task reported for each of the models? Why different task is used for each different model? It would be better to pick e.g. three tasks and use them for all four models.
- the work does not provide much insight into the problem that motivates the approach (e.g. why "group 10" would affect "important parameters" more than "group 9")

Questions:
- how common is the behavior in Fig 1? It would be interesting to see evidence of it on at least one more dataset/task. It might also be useful to use log scale for the axis, currently it is not possible to see if e.g. removing group 9 leads to lower/similar/higher performance drop vs group 8 (i.e., is group 10 special, if yes, why?).
- what is the average/median magnitude of singular values in group 9 vs in group 10? 
- have you tried minimizing eq. (5) using SGD? Do you expect substantial gap between (5) and the simplified version in eq. (6)?


---
Post-rebuttal:
Thank you for the clarifications and additional experiments, I've updated my score to 6.
summary_of_the_review: The paper provides a simple method to reduce (modestly) the size of an already trained language model, without the need for costly re-training. The evidence in favor of the method is empirical, with little in terms of theoretical or experimental exploration of the motivating phenomenon: the overlap between "important parameters" and "parameters poorly captured by SVD".
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The proposed paper focuses on a new technique to compress weight matrices in Machine Learning, e.g., weight matrices of layers in DNN. 
One idea is to simply exploit truncated SVD. While this minimizes the Euclidean norm of the error, it doesn't necessarily lead to a lower task error since the truncated part might still have a noticeable effect in the resolution power of the model. The authors suggest to solve an alternative minimization problem which takes empirical Fisher information under consideration. Experiments on a few datasets imply that the proposed method can achieve better results than simply using truncated SVD of the weight matrix.
main_review: 
-) Written English needs improvement.

-) The idea is interesting and the authors explain the main intuition adequately clear. 
The authors explain the advantages and limitations properly.

-) How easy/fast is to generate the fine-tuned model? In other words, what is the overall 
computational cost to create a fine-tuned model and apply FWSVD versus generic training 
and SVD?

-) Unfortunately it is not clear to me whether the proposed method comes with any guarantees. 
I understand that minimizing the Euclidean norm of the approximation of W does not necessarily 
imply lower classification (or other tasks) error, however I am not sure the proposed method does 
either. Is that so? For example, will the new method always be better than SVD? One needs to 
account for the cost of pre-processing costs too. 

-) In addition to my comment above, Figure 6 seems to complicate the discussion. The legend reads 
as "Unlike SVD, the performance drop of our FWSVD perfectly imitates the ideal trending shown in Figure 1.", 
however I can see that on the left subfigure "Rank group 8"  leads to a higher performance drop than "Rank group 7" which 
is exactly what the authors wanted to avoid in the first place. Furthermore, on the right subfigure, FWSVD 
actually introduces higher reconstruction error, which does not even reduce monotonically. I understand that 
the goal is to achieve higher accuracy in the classification task and not the matrix reconstruction part, 
but this seems to complicate the optimization part of the model, e.g., how do the authors make sure 
they do not overfit or spend more time than necessary computing the low-rank approximation?

-) The above experiment is explained better in Section 5.5.1 but the following sentence is confusing 
"The results suggest that FWSVD’s objective (Equation 6) aligns with the task objective better by sacrificing the reconstruction error."

-) Section 5.2.2 is rather unclear; can you please provide more details, perhaps in the appendix? Without 
proper explanation in this step, the experiments can not be judged fairly. 

-) Please make the y-axis in Figure 6 run in logarithmic scale. 

-) While generally providing better results than SVD for the experiments shown in this paper, the proposed 
method does not offer dramatic improvements in accuracy, nor it is guaranteed to do so. It would be very 
helpful if the authors could provide additional experiments about other benefits of the method, if applicable, 
in the appendix (e.g., is the new method faster than previous approaches based on SVD etc.)

Minor comments:

-) Equation (1) and related text is confusing. The "\approx" symbol should be "=" since matrix W is of rank 'k'. I suggest the authors to simplify discussion by using 'r' instead of 'k' and simply mention that matrix W is of rank k\geq r.

-) "we propose a simplification by making the same row of the W matrix to share the same importance". Did you mean to say that each weight located in row 'i' will use the average weight of its row?
summary_of_the_review: The proposed idea is interesting and seems to lead to improvements over truncated SVD, however these improvements 
are not major. Moreover, the proposed algorithm is less general than truncated SVD.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the compression of the weights of a neural network via a modification of the standard paradigm of using low rank approximations via SVD on the weight matrices. Instead, they propose a weighted SVD objective where the weights are computed based on the Fisher information of the (trained) parameter with respect to the dataset. They simplify the method so it is more computationally tractable by having the same row of each weight matrix share the same weight. This translates to solving SVD on a modified reconstruction objective (multiply by a diagonal). Here the Fisher information for parameter w is approximated by the average squared l2 norm of the gradient with respect to w over the empirical data distribution.

They then conduct several experiments with this approach in the problem domain of compressing language models. They evaluate their method on compressing task specific models as well as the problem of compressing an already compressed model, and demonstrate their approach is generally better.


main_review: Strengths: 
1. Solid experiments demonstrating the proposed method outperforms other existing approaches. 
2. Novel weighting scheme for SVD for low-rank weight compression (though should double check this more thoroughly).



Some comments: 

1. I would suggest using the phrase "weighted SVD" early on in the introduction (e.g., exactly when you introduce your new method).
2. It might be good to address the existing literature on compressing trained neural networks which also goes beyond simply trying to minimize the Frobenius norm of the difference between the weights. For example, the paper https://arxiv.org/abs/1505.06798 also goes beyond simply considering the reconstruction objective on the weights, and includes the nonlinearity as well in the reconstruction objective. Your work obviously is different enough to stand on its own, but it might be good to make a mention of this work and others (e.g. do more of a lit search / related work on low rank compression).
3. It might be good to find the best hyperparameters for each setting to truly do a fair comparison (avoiding hyperparameter tuning isn't really a fair comparison, IMO -- but this depends on how much compute you have available). 
4. You forgot to bold the best performer in line 3 of Table 2 (in this case, the original compact model).

summary_of_the_review: The paper introduces a new approach to weighting the low-rank compression of neural nets, and empirically demonstrates that it outperforms other methods. The approach isn't too surprising, but it does appear to be novel (though I would recommend doing a more thorough lit review and make the related work section larger, since I feel certain that there are many more related papers doing similar things (I mention one in the main review). I think the experiments are overall good and demonstrate the point nicely, which is essentially that the important thing in compressing networks is not the parameter compression -- but rather, the compression of the function specifically on the relevant distribution. Overall I vote to accept.

UPDATE:
I read the comments, my opinion is unchanged.
review_title: null

Meta-review: The paper studies the problem of task-specific model compression obtained from fine-tuning large pre-trained language models. The work follows the line of research in which model size is reduced by decomposing the matrices in the model into smaller factors. Two-step approaches apply SVD and then fine-tuned the model on task specific data. The present work makes the observation that after the first step (the SVD compression) the model can dramatically lose its performance, due to the mismatched optimization objectives between the low-rank approximation and the target task. The work provides evidence backing this claim. The paper proposes to address this problem by weighting the importance of parameters for the factorization according to the Fisher information. Experimental evaluation shows that the proposed method can achieve better results than variants that use truncated SVD of the weight matrices.

The paper is well written and easy to read. The method is simple and effective and can be applied to in a wide range of settings. The authors provided a thorough response which clarified several points. This led Reviewer Kuwu to increase the score to 6.

All three reviewers agree that the main observation in the work is interesting and informative for researchers and practitioners working on the problem.

Reviewer jnTC points out that the paper would have been stronger if it included theoretical exploration of the reasons behind the "importance of low SVs" phenomenon.

Reviewer Kuwu and jnTC consider the results marginally novel. Reviewer Kuwu considered the significance of the reported results to be limited, and put the work marginally above the acceptance threshold. Reviewer jnTC disagrees with this view, considers and appreciates the generality of the method and the fact that it can work well even for compressed models, while improving in accuracy by a few percent over competing approaches which result in similar parameter counts. The AC agrees with Reviewer jnTC.

Overall all reviewers consider the paper borderline but recommend accepting the paper. The AC overall the topic important (reducing the footprint of language models), the method simple and well motivated. The empirical evaluation is very thorough and shows clear gains across a large number of settings.
Decision: Accept (Poster)