Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies how to initialize via spectral initialization and regularize DNNs via Frobenius decay. The benefits of spectral initialization and Frobenius decay are demonstrated via many experiments. This paper focuses on the empirical study of both SI and FD.  

My major concern is on its technical contribution. The spectral initialization is a scheme commonly used for low-rank models. It is well understood that spectral initialization is better than random initialization in low-rank literatures. Moreover, the proposed Frobenius decay is simply a penalization on the squared norm of the whole factorization. That is, it replaces the weight-decay $\|W\|_F^2$ in deep nets by plugging in the low-rank formulation of $W$. The behaviors of these two components in the experiments are expected. These two components have not provided much new insight. 

~~~~~
This major concern is relieved after rebuttal. 

review_title: Technical contribution

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper studies how the initialization and regularization of the factorized layers $W=U\Pi_i M_i V^T$. It focused on the two known techniques, spectral initialization (SI) and Frobenius decay (FD) (i.e., regularize the weight $W$ rather than the factorized terms $U$, $V$, and $M_i$'s), to initialize and regularize such layers for training.
From the technical point of view, the paper is not novel, however, its strength is motivating the need for SI & FD, and providing various empirical studies in different applications such as model compression and self-taught distillation.
Strengths:
1. The paper is generally well-written and clear. The majority of claims are either cited appropriately or shown via extensive simulations.
2. Good motivations, supported by simulations (Figures 1, 2) and claim 3.1
3. Extensive simulations for different tasks: compressed model training, knowledge distillation, and multi-head attention, in different areas computer vision, question-answering, transformer training, and BERT.

Weaknesses:
The paper can be seen as merely applying known methods to different areas and observing its performance. It lefts many questions unanswered and raises some other concerns;
1. SI initialization is reasonable when training a compressed model from a "pre-trained" model since the decomposition error is minimized $W\approx UV^T$. However, when training from scratch, to my understanding, the method first generates a normal $W\sim N(0, \sigma^2 I)$, and uses SVD to find $U$ and $V$. SI + FD can be interpreted as assuming a Gaussian prior on $W$ and enforcing that prior on $W$ during training. One immediate question would be why such a prior is a good assumption in general? From VGG experiment (Table 1), one can conclude that specific initialization is not an important factor in training. Probably, for other simpler initializations for $U$ and $V$ (e.g., $U$ and random Rademacher and $V$ random normal, or even one of them being some truncated identity matrix) the FD will work too. The claims can be interpreted as 'If Gaussian prior is necessary for the parameters, SI+FD is the way for training."
2. Figure 1 shows the bound in equation (2) is tight for the compression. Is there any similar result for the distillation (i.e., $r>max(m,n)$)?
3. How to extend SI to have factorization $W=U(\Pi_{i=1}^d M_i)V^T$ for $d>0$?


There are a few minor points/typos that the paper can be improved on:
- The RHS equality in equation (2) is not trivial. It needs a proper reference or proof.
- In Table 1, Dynamic Sparsity for CIFAR100 at compression 0.02 performs better than SI&FD. The bold text in the table incorrectly implies that SI&FD is better.

review_title: A good empirical study of known techniques for initilization and regularization

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper discusses about applying low-rank matrix and tensor factorization of weight and applying weight decay on them. This type of low-rank regularization is an important mechanism in deep learning models and already many researchers have shown interest in this topic, hence this paper would interest many researchers in the community.

The technical aspects of the paper seem to be correct. The experimental results are very encouraging since good improvements are shown with popular datasets. Also, the paper covers compression using ideas related to the state of the art tensor factorization methods.

The most important component in learning with factorization methods is specifying the appropriate rank of matrices and tensors. When using factorization methods problems such as matrix/tensor completion, the performance will strongly depend on the rank. It is not clear how the rank of weight matrices would affect the performance of the deep learning model. This paper does not address this issue. Furthermore, there are no theoretical results shown that take the rank into consideration. Can the authors add an experiment to show how different factorization with respect to ranks affects the performance? And/or can they give a theoretical result explaining how the rank relates to the improvement of the learning model? 


review_title: An important research but few details are missing 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies initialization and regularization in factorized neural networks (reparameterize a weight matrix by the product of several weight matrices). The authors proposed spectral initialization, that is to initialize the factorized matrices using the SVD of the un-factorized matrix. The authors also proposed Frobenius decay that is to regularize the Frobenius norm of the product of the factorized weight matrices. The motivation is to simulate the routines for non-decomposed counterparts. The authors empirically showed the effectiveness of spectral initialization and Frobenius decay in different applications: compressed model training, knowledge distillation, and multi-head self-training. 

I think it’s important to study the initialization and regularization for factorized neural networks. A priori, it needs different initialization and regularization methods due to different architecture compared with its non-decomposed counter-part. This paper gave very simple and natural solutions and was able to show its effectiveness in experiments. 

I also have some questions as below:
1. In the experiments in section 5 (knowledge distillation), default initialization is used instead of spectral initialization. I wonder if SI leads to a bad performance here. If that’s the case, it requires more explanation of why SI fails in this setting.
2. In Figure 1, it seems FD is a stronger regularizer compared with default weight decay. It seems if the regularization coefficient is carefully tuned for each regularizer, the benefits of FD is actually not very significant. Also, what’s "no decay (normalized)"?
3. In section 2, the definition of the factorized CNN is not very clear to me. It might be good to give more detailed definitions here. 
4. Spectral initialization requires computing SVD of the weight matrix. If the matrix dimension is high, this step can be very time-consuming. I wonder if there is any more efficient way to construct the factorized matrices so that their product is still as i.i.d. Gaussian matrix. Because we don't need to compute the SVD for an arbitrary matrix, what we need is only to make sure that the product of the factorized matrixes is distributed as i.i.d. Gaussian. 
review_title: Empirical result studying spectral initialization and Frobenius decay on factorized NN

Meta-review: This paper applies spectral initialization and weight decay to neural nets with factorized layers. Although these ideas have been extensively studied in other areas, formalizing and applying them to deep neural nets is of potential interest to the community. The simulation results are nice, especially the experiments on compression methods (comparison to sparse pruning e.g. lottery tickets) and Transformers. I recommend acceptance. 

Decision: Accept (Poster)