Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper the authors aim to understand the structure of convolution filters, in order to initialize them better. First, the authors estimate the covariance matrix of the filters based on pre-trained models, and sample new filters from a Gaussian distribution with this covariance. Then, based on a visual inspection of the filters in trained networks, the authors propose a simple initialization method. The method seems to work rather well in practice, even when the filters are not trained.
strength_and_weaknesses: ***Strengths***:

The initialization method proposed in the paper is interesting and the empirical results are encouraging. The method is very simple to implement.


***Weaknesses***:


**Limited scope:** The experiments in the paper are limited to mainly one architecture called ConvMixer (and also the ConvNeXt architecture). Also, only CIFAR-10 and ImageNet datasets are used. I think it will be interesting to see if the proposed initialization method can work for other architectures and various datasets.

**Comparison to other methods:** The authors mainly compare to the standard uniform initialization (default in PyTorch). I think it will be interesting to compare to other methods (discussed in the related work section). Another relevant work: 

Li et. al. FILTER SHAPING FOR CONVOLUTIONAL NEURAL NETWORKS, ICLR 2017

Although other methods are probably more complex, it will be interesting to see the test accuracy that they can achieve, and what is the time complexity of different methods.

**Visual inspection:** I think the authors should better explain what is the structure that we see in Figures 1 and 21. The filters in these figures look rather different, especially in the last layer, although the authors claim that “Covariance matrices from a ConvMixer trained on ImageNet exhibit similar structure to those of ConvMixers trained on CIFAR-10”. For example, the checkerboard structure in figure 21 doesn’t appear in figure 1. 
I think the authors should better explain the intuition behind the proposed covariance $\hat{\Sigma}=M \odot (C-0.5S)$. How is the checkerboard structure captured here ?
The authors say that “the sub-blocks of the covariances often have a static negative component in the center, with a dynamic positive component whose position mirrors that of the block itself.” Is it possible that the “dynamic positive component” is the variance, i.e. $[\Sigma_{i,j}]_{\ell,m}$ for $i=\ell,j=m$, and thus it is large ?

**Sensitivity to hyperparameters of the method:** Although the method is quite robust to the hyperparameters of the method (eq. (11)) for CIFAR-10, it is rather sensitive for ImageNet, as we can see in figures 18,19 (and tables 5,6).

***More comments and questions:***

- Can you try to explain or give any intuition why for narrower models 32 filters work better than 256 ? or for shallower models 8 layers better than 32 ?
- The authors don’t tune the hyperparameters of the method for each model and claim that “hyperparameter tuning is optional”. I think it will be interesting to see what can be squeezed from this method by optimal tuning for each model. 
- What do you mean by “each data point runs through a full cycle of the LR schedule” in figures 8,11 ?
- In eq. (9): 1/2 is missing
- Will be good to explain how exactly linear interpolation is performed for the shallower models.
- For the shallower models “2- and 4-deep models are also quite effective” – but no results for 2 layers in figure 4.
- In figure 4, for narrower models, there is no comparison to uniform, and no 200 epochs.
- Will be good to explain the “triangular learning rate schedule”.
- I think it will be good to give a quick description of ConvMixer and ConvNeXt architectures.

clarity,_quality,_novelty_and_reproducibility: - The paper is well written and easy to read.
- I think the main contribution of the paper is the initialization method in section 3. However, I think this contribution is quite modest, given the fact that other techniques exist and the authors don’t compare the different methods.
- Code is not submitted with the paper.
summary_of_the_review: Overall, the paper proposes an interesting method with nice experiments, but I think more work should be done to provide clearer explanations and intuition, with more architectures and comparisons to prior work, before it can be published.

-- POST AUTHOR FEEDBACK:

Thank you for the detailed response and the new results. I am happy to increase my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies the structure of depthwise convolutional filters in pre-trained models. Empirically, it shows that computing the covariance of these filters and using it for initialization can bring benefits in terms of trainability and final performance, and such statistics can be transferred from a smaller to a larger model. A new initialization is proposed which further proposes benefits and doesn't require pre-training a network.
strength_and_weaknesses: The paper is clear and the motivation / intuition for studying the structure of pre-trained filters is presented well.

Empirical observations are somewhat surprising, showing that transferring from a smaller to a larger network provides more benefits than using covariances computed from the original model.

The main visualizations and comparative results are, unfortunately, based on CIFAR-10 and not on ImageNet. Although some ImageNet results are shown, it would be better if Fig 1, 2 and 3 were based on ImageNet. Some of the main claims from Fig 2, e.g. that transferring from smaller to larger models provides benefits, is also based on what seem to be small improvements (comparing groups C and B).

The proposed initialization seems to be very strongly based on the visualizations of CIFAR-10 trained networks. Although it shows benefits, some comparison against other initialization schemes would be valuable (dirac, orthogonal, zero-init for convs with skip connections, etc).
clarity,_quality,_novelty_and_reproducibility: The paper is clear and the ideas and results are novel.
summary_of_the_review: The idea is interesting and the visualizations help motivate the approach. The proposed initialization seems a bit too ad hoc and based on heuristics, and, while it yields faster training, the final performance under standard training budgets is similar to the commonly-adopted uniform initialization (bottom rows of Table 1).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an initialization scheme for convolutional filters. The main idea is to first build a filter covariance matrix from pre-trained networks and then sample initial filters based on this covariance. The paper also provides a learning-free closed-form alternative for filter initialization. The experiments specifically focus on ConvMixer and ConvNext networks, and show that using the proposed initialization scheme outperforms uniform initialization. They also show that in some cases, networks with fixed frozen filters can achieve competitive performance.
strength_and_weaknesses: Strengths:
- Building a closed-form initialization scheme based on the observations from pre-trained networks is notable.
- The authors evaluate their method in variaous variants and the visual presentation of the results helps the reader to better see the differences across the variants. 

Weaknesses:
- The closed-form covariance matrix computation is not completely learning-free, as the authors perform a grid search to find the optimal parameters.
- It is not clear why the authors use a quadratic schedule for the variance.
- The related works section can be improved by adding other recent initialization methods and comparing the results to them.

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well-written, but it can be improved by fixing the typos. 

Quality: The effectiveness of the proposed initialization method with multiple experiments is experimentally sound. However, the paper lacks theoretical justifications of why the proposed covariance matrices perform well, other than the intuition and the observed pattern in trained networks.

Novelty: The idea of using pre-trained network weights for initialization is not novel. However, the proposed closed-form multivariate scheme is novel and it can be useful.

Reproducibility: The code is not provided, but the implementation details provided can be used to reproduce the results.


summary_of_the_review: The main idea of the paper is interesting and useful to the community. 

Questions from the authors:
- It would be helpful to see the results of initializing with covariances from smaller models when two or more variables change simultaneously, for instance, from a model both narrower and shallower, to see if similar results will be obtained with more efficient pre-training.
- Would it be useful to use the proposed method for initializing networks with normal convolutions like ResNets?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the covariance structures of each convolution filter in deep learning models for classification problems. It builds a parametric model for these structures and proposes to directly sample a multivariate Gaussian distribution as an initialization for a family of CNNs. Numerical results show that in some cases, such initialization may achieve very high classification, without the need of learning. 
strength_and_weaknesses: This paper builds upon an interesting observation of the covariance structures of each convolution filter at multiple layers of CNNs. It proposes a parametric model for such covariance structures such that certain parameters may be fitted from trained models. 

Certain results of this paper could still be improved to support the point of the learning-free multivariate initialization scheme. It is still not very clear on the ImageNet case, if the learning-free approach could achieve a similar performance compared to the learning approach. In Table 1 Frozen vs. Thawed, we see that the results with the Frozen case are worse than those of the Thawed case. It is thus not clear why learning is not needed. One suggestion would be to fit the parametric model of covariances to some small ImageNet models, and then to see if they work well (as CIFAR is quite different to ImageNet, using the parameters from CIFAR might have some unpredictable risk). 
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to read. I have a few questions:
- What is a sub-block as mentioned in Figure 1?
- That the notation of [ ] in equation 3 means? Are you trying to estimate the covariance Simga_i
 from H vectors?
- How do you directly transfer CM-256/24 to CM-256/8 in Figure 3 (D in first subplot)? It is not very clear to me. 
- Do you observe edge-like filters in the first layer of CNNs? If so, are those filters look very different to the ones built from the multivariate initialization scheme? Could that explain the performance gap between Thawed and Frozen?
- There a few typos in the text, such as in the caption of Fig 4, the paragraph of eq (6).
- What does ‘free’ mean in the last sentence of the conclusion?
summary_of_the_review: REVISED: I have raised my score to 6 as I still find the significance of the paper could be improved by providing further discussion on the impact of the initialization of other layers beyond depthwise convolutional layers . This is not so clear by reading the paper. Nevertheless, the authors proposed an interesting method to model the covariance structure of these layers, which gives an explanation of the performance of CNN. This is novel and could be published.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the structure of depthwise convolutional filters in pre-trained ConvMixer and ConvNeXt models, which have large kernels. It shows empirically that computing the covariance of these filters and using it for initialization can improve the trainability and final performance, and that such statistics can be transferred from a smaller to a larger model. Based on this, a new initialization is proposed which doesn't require pre-training a network, and improves over several baselines. Interestingly, sometimes, this improvement can be achieved without training the depthwise convolutional filters at all. This empirical observations in this paper seem novel and interesting, and the suggested method seems practically useful. 

The reviewers had some concerns, but these were mostly addressed during the discussion period, e.g. with some additional experiments and ablation studies. 
justification_for_why_not_higher_score: The suggested method and empirical observations are nice, but do not seem ground-breaking.
justification_for_why_not_lower_score: All reviewers recommended accept [8,8,6,6] and there are no major concerns remaining.
Decision: Accept: poster