Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents an approach called ShapeShifter to compress large language models. The paper follows the approach of reducing number of parameters by using factorized matrix representations. The primary idea is to represent the matrix as a sum of Kronecker products, instead of standard low-rank factorizations. The paper theoretically shows that for non-square matrices, the proposed approach allows for increased reduction in parameters for the same decomposition rank. For square matrices, it allows for increased expressiveness of the decomposition for the same rank. Experimental results on machine translation tasks show that the proposed approach performs much better than the state-of-the-art model compression approaches and is just slightly worse than the large models.
main_review: The paper is very well written and easy to understand. The primary idea itself is quite novel, although the use of Kronecker products as a matrix factorization technique has been around for a long time but this paper shows its application in compression large neural network which is quite interesting.  The authors also provide ample theoretical and experimental evidence to show that their method is indeed better than the state-of-the-art.
limitations_and_societal_impact: 1) Since the proposed compression still lags behind the large model it would be interesting to see some analysis on what exactly is this model not able to express, for instance how well does a rank r Kronecker product matrix represent a matrix from the original large model.
2) The paper does not talk about any reduction in training time or prediction time, it would be good to see some discussion on that as well.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes Shapeshifter, a parameter efficient Transformer using low-ranked factorized representations and some reshaping tricks to achieve fast, space efficient and expressive linear layers. 
main_review: I think this paper is a great contribution to existing literature that tries to make Transformers more parameter efficient. 

Results look great and promising, the method also looks interesting and simple (in a good way). 

The results look very good, and Shapeshifter models seem to be able to substantially compress Transformers by almost 20 times in some cases while maintaining competitive performance. Results look very promising. The method also looks simple enough to have widespread potential impact. 

A particular nit I had is about the complexity. I think there is no mention or evaluation of flops/throughput/speed of the model. I was wondering if the authors could report that. Parameter counts are only one aspect of efficiency and it would be good to know how fast/slow the model is. 

Another concern is also that experimental comparisons are quite limited and only conducted on MT tasks. It would be great if authors can provide more evidence that this works on another task other than MT alone. 

Overall, I like the idea and i think this could be pretty impactful. 


limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: A sizable effort has been made in recent history to obtain the accuracy of very large language models at a fraction of the parameter of the large models. Broadly these methods fall into various categories like model compression, factorized representation, distillation etc. This paper underlines another such attempt at obtaining a factorized representation of matrices to reduce the parameter space. The authors justify the use of Kronecker products to factorize parameter space, by showing that stacked layers of low rank matrices increase expressiveness. They also show the effectiveness of their proposed method by achieving a 4 to 14-fold reduction in parameters without sacrificing accuracy.
main_review: Contributions:
One of the key contributions of this paper is showing how unlike low rank matrix factorization, stacking multiple layers of Kronecker factors, has an effect on the kind of matrices that can be represented. In the process they prove / make the following key observation:

-- An orthogonal matrix can be  represented as a product of O(n^2) 2-variants givens matrices (known)

-- Any 2-variant Givens matrix can be represented as a product of at most three component matrices where each component matrix is a sum of two pairs of Kronecker products. (theorem 2). 

-- This result extends to any n-by-m matrix in general. 

-- If instead one uses the sum of r Kronecker products of  matric pairs then the number of such matrices that are needed to represent an orthogonal matrix is O(n^2/r).

-- Multiple experiments show that as compared to other factored representations such as low-rank, PHM the shapeshifter method achieves a better tradeoff in terms of parameter and BLEU score. 

Weakness

-- Using Kronecker products to reduce/factorize the parameters has been explored in multiple works (and has been cited by the authors), thus reducing the novelty of shapeshifter. 



Questions for the Authors:

-- Given that in line 144 -145 the multiplication step uses many operations such as transpose and reshape, it would have been good to see on synthetic examples the tradeoff between memory (perhaps parameter size) and time for shapeshifter and other baselines. I would assume that a lower parameter representation is achieved at the cost of higher computational requirements.

-- The previous question also points towards another relationship i.e. between computational cost (training time) and parameter reduction. Have the authors done any experiments on it or given it any thoughts?

Minor points

-- Instead of the result table, the tradeoff would be better visible with a plot of parameter size and BLUE score. 

limitations_and_societal_impact: The authors have not discussed any adverse societal impact of their research, and justified it as they are trying to reduce the size of large language model, which should provide solutions to some of the limitations of large language models.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Shapeshifter reparameterizes dense kernels into a product of two lower-rank matrices. Shapeshifter introduces reshapes and transposes to make this reparameterization more efficient. The paper provides a theoretical justification for its expressivity and strong experimental results in machine translation.
main_review: Originality: I haven't seen this exact factorization in previous literature, so I believe the idea is novel. It's a relative straightforward extension of low-rank factorization. I consider the simplicity a positive, however. TensorNetwork (https://github.com/google/TensorNetwork/tree/master/tensornetwork/tn_keras) is another generalization of low-rank factorization that I would have liked to see a comparison to.

Quality: Claims are supported with theoretical justification and experimental results. The decomposition into Kronecker products provides good intuition. As the authors note, "the [theoretical] results are limited to idealized multilayer models with linear activation". Given that I hoped to see more experimental results. Maybe at least language modeling and compare against BERT (GLUE, SQuAD)? Some practical matters were left unaddressed. What's the effect on the reshapes and transposes on training and inference speed? It's common in larger models to parallelize the dense layer across multiple accelerators. Is that possible here?

Clarity: Paper clearly explains the ideas and experiments in a way that would be easy to reproduce.

Significance: Given the ease of implementation, I could see this method attracting widespread use as a way to reduce parameter count. More experimental results and benchmarks are needed and above practical matters would need to be addressed for researchers and practitioners to feel confident in using this technique, however.
limitations_and_societal_impact: I don't believe there are any potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an algorithm they call ‘Shapeshifter’ to reduce the number of parameters needed for Transformer-based models while preserving expressiveness and performance. The proposed algorithm is based on the idea of low-rank factorizations of weight matrices, but uses a different factorization method involving sums of Kronecker products that leads to greater expressiveness. They show theoretical analysis to prove their factorization can accurately represent any $n \times m$ matrix or stack of matrices (in an idealized case), and experimental results using transformer-based models for translation on English-German, English-French and English-Romanian. They demonstrate superior or comparable performance with vastly fewer parameters compared to some other model compression techniques.
main_review: This paper utilizes applies concepts proposed in previous work in a novel way to create a general method for reducing parameters needed in a wide variety of deep architectures. The proposed method is far more general than the work that came before it, and the reduction in parameter size it offers is applicable throughout almost any application in deep learning. I think it is a very strong paper overall, and a very useful contribution to the field.


The main criticisms I would make of this paper relate to the results and related work. First, I think it is odd that knowledge distillation methods such as DistilBERT are mentioned in the introduction, but not discussed in the related work or compared against in the results. I do understand that the proposed method is somewhat orthogonal to knowledge distillation and could perhaps even be used in concert with it. Nevertheless, the majority of successes in minimizing the size of transformer models in the literature thus far have used knowledge distillation (DistilBERT, TinyBERT, etc…), and a simple comparison of your method’s effectiveness against these existing approaches in preserving accuracy while reducing parameters would help make a stronger claim for the method’s usefulness. Without this comparison it is hard to judge if this using this method would lead to an actual improvement over existing techniques for shrinking model size. My one hesitation about giving this paper a higher score is due to my uncertainty over how it compares to these models.


Second, I think the results section could be better structured. There are 5 tables containing results from 5 different datasets, but multiple tables contain results from the same datasets and there are multiple different datasets used for the same language pairs. Many of the tables also lack captions and in places it is unclear how to interpret them (e.g. why are the results with the DeLighT architecture and PHM architecture both in table 2, with no explanation other than a horizontal line about what the difference between the first set of results and the second set of results is). These aren’t critical issues, but I found that I had to read through the results section three or four times before I understand what each of the tables was supposed to be representing.


Finally, I had one or two questions I wanted to ask about the architecture. First, at the end of section 4 you claim “Compared to the equivalent sum of Kronecker-products representation, using the matrix factorization representation avoids the explicit summation of terms, resulting in simpler, more efficient implementations”. Is it actually faster? It seems like it should have the same number of operations to me. Second, you mention that the $m \times n$ matrix is reshaped into a $\sqrt{mn} \times \sqrt{mn}$ matrix. Do you discuss how you handle the case where $\sqrt{mn}$ is not an integer?

limitations_and_societal_impact: Little to no discussion of either as far as I can tell. It might be a good idea to add some discussion about the situations where this approach is and is not suitable compared to other model compression approaches.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method to reduce the number of parameters by weight factorization. Unlike traditional low-rank factorization (n x r and r x m), this work decomposes the matrix into two sqrt(mn) x r matrices. The algorithm includes the reordering of elements to recover the desired shape. The authors find connections to Kronecker products and theoretically support the expressiveness of the proposed method. Experiments on NMT tasks show that Shapeshifter can achieve better parameter efficiency (fewer parameters with comparable performance). The results are compared to two similar works: PHM and DeLighT.
main_review: Overall, the idea is simple and efficient. The paper is well-written and easy to follow. I also appreciate that the authors explained experiments in detail and split the parameter savings for each component in every table.

I have a few concerns and suggestions:

(1) In section 5.3 and table 5, the expressiveness is showed by comparing with low-rank factorization. (“Despite more than two times… increased expressiveness of a stack of low-rank … layers” (line 340-343)) However, the parameter reduction almost comes from the embedding, so how can we know that different factorization in layers is the reason of better performance? I suggest using the same embedding factorization method (or no factorization for the embedding) to match the number of parameters and only change layers above for fair comparison. Also, there should be some explanation of the relationship between the number of parameters and expressiveness.

(2) I am not much familiar with the theoretical part, but does the analysis still holds after reshaping and transpose? Maybe these operations change the rank of the matrix?

(3) As mentioned, (line 156-157) I agree that Shapeshifter implementation should be faster than PHM (and possibly slower than naïve low-rank factorization). It will strengthen the paper if there are some inference speed comparisons.

(4) I am just curious that low-rank factorization (both naïve and Shapeshifter) ‘from-scratch’ is better than ‘decompose-after-training’. If we do not think about the training cost and the latter performs better, why should we train with a factorized form?

Minor issue: in table 5, “Ro-En” should be changed to “En-Ro”.

-------------
After reading the authors' responses and other reviews, I think my concerns are well addressed. I appreciate authors, especially for extra experiments, and I'll change my score from 5 to 6.
limitations_and_societal_impact: The authors well addressed the limitations of theoretical analysis. Because the idea mainly focuses on efficiency, I do not think this paper needs more discussion about the social impact.
review_title: null

Meta-review: The paper attempts to improve parameter efficiency of transformer models for sequence modelling. In this regard, the authors propose a novel replacement of all matrices in the transformer by sum of Kronecker products (a type of low-ranked factorized representations) and some reshaping tricks. Further, the expressivity of such Kronecker-based linear layers is analysed. Empirical results on translation looks promising (better model-size vs performance trade-off as compared to other compression approaches for transformers). We thank the reviewers and authors for engaging in an active discussion, which resulted in clearing a lot of the concerns (e.g. speed/flops) and a lot of constructive feedback were provided to improve the paper. The authors provided new empirical results as part of the discussion, please include them in the final version of the paper as they add great value and understanding to the model as a whole.

- Please remove "fast" from abstract in the final version as the proposed method is slower than baseline in both training and inference.
Decision: Accept (Poster)