Understanding the Covariance Structure of Convolutional Filters
Authors: Asher Trockman
Devin Willmott
J Zico Kolter
Venue: ICLR 2023 poster (2023)
Keywords: initialization
init
covariance
gaussian
convolutional neural network
convmixer
convnext
transfer learning
spatial mixing
computer vision
convolution

Abstract: Neural network weights are typically initialized at random from univariate distributions, controlling just the variance of individual weights even in highly-structured operations like convolutions. Recent ViT-inspired convolutional networks such as ConvMixer and ConvNeXt use large-kernel depthwise convolutions whose learned filters have notable structure; this presents an opportunity to study their empirical covariances. In this work, we first observe that such learned filters have highly-structured covariance matrices, and moreover, we find that covariances calculated from small networks may be used to effectively initialize a variety of larger networks of different depths, widths, patch sizes, and kernel sizes, indicating a degree of model-independence to the covariance structure. Motivated by these findings, we then propose a learning-free multivariate initialization scheme for convolutional filters using a simple, closed-form construction of their covariance. Models using our initialization outperform those using traditional univariate initializations, and typically meet or exceed the performance of those initialized from the covariances of learned filters; in some cases, this improvement can be achieved without training the depthwise convolutional filters at all. Our code is available at https://github.com/locuslab/convcov.

Published as a conference paper at ICLR 2023
UNDERSTANDING THE COVARIANCE STRUCTURE OF CONVOLUTIONAL FILTERS
Asher Trockman' Devin Willmott', J. Zico Kolter Carnegie Mellon University and Bosch Center for Al Correspondence to: ashertêcs cmu. edu
ABSTRACT
Neural network weights are typically initialized at random from univariate distributions, controlling just the variance of individual weights even in highly- structured operations like convolutions. Recent ViT-inspired convolutional networks such as ConyMixer and ConvNeXt use large-kemeel depthwise convolu- tions whose learned filters have notable structure; this presents an opportunity study their empirical covariances. In this work, we first observe that such learned filters have struhy-structtred covariance matrices, and moreover, covariances calculated from a small network may be used to effectively initialize variety of larger networks of different depths, widths, patch sizes, and kernel sizes, indicating a degree of model-independence to the covariance structure. Mo- tivated by this finding, we then propose a learning-free ffre nultivariate initialization scheme for convolutional filters using simple, closed-form construction of their covariance. Models using our initialization outperform those using traditional univariate initializations, and typically meet or exceed the performance of those initialized from the covariances of learned filters; some cases, this improvement can be achieved without training the depthwise convolutional filters at all. Our wac.occuullbccc........ooo conveov.
find that
code is available at https:: htggtttttbbb.ggg/////////g com coocuusabbb
INTRODUCTION
Early work in deep learning for vision demonstrated that the convolutional filters in trained neural networks are often highly-structured, in some cases being qualitatively similar filters known from classical computer vision (Krizhevsky et al., 2017). However, for many years it became standard to replace large-filter convolutions with stacked small-filter convolutions, which have less room for any notable amount of structure. But in the past year, this trend has changed with inspiration from the long-range spatial mixing abilities of vision transformers. Some of the most prominent new convolutional neural networks, such ConvNeXt and ConvMixer, once again use large- filterfilter convolutions. These new models also completely separate the processing of the channel and spatial dimensions, meaning that the now-single-channel filters are, in some sense, independent from each other than in previous models such as ResNets. This presents an opportunity to investigate the structure of convolutional filters.
In particular, we seek to understand the statistical structure of convolutional filters, with the goal of more effectively initializing them. Most initialization strategies for neural networks focus simply on controlling the variance of weights, as in Kaiming (He et al., 2015) and Xavier (Glorot & Bengio, 2010) initialization, which neglect the fact that many layers in neural networks are highly-structured, with interdependencies between weights, particularly after training. Consequently, we study the covariance matrices of the parameters of convolutional filters, which we find have a large degree of perhaps wrrhaps-interpretabe trained models can be used to effectively initialize new convolutions by sampling filters from the corresponding multivariate Gaussian distribution.
structure. We observe that the covariance of filters calculated from pre-
We then propose a closed-for forseddfoommaoe and completely learning-fre free construction of covariance matrices for randomly initializing convolutional filters from Gaussian distributions. Our initialization is highly effective, especially for larger filters, deeper models, and shorter training times; it usually outper- forms both standard uniform initialization techniques and baseline technique of initializing by

---

Published as a conference paper at ICLR 2023
sampling from the distributions of pre-trained filters, both in terms of final accuracy and time-to- convergence. Models using our initialization often see gains of over 1% accuracy on CIFAR- 10 and short-training ImageNet classification; it also leads to small but significant performance gains on full-scale, R 80%-accuracy ImageNet training. Indeed, in some cases our initialization works so well that it outperforms uniform initialization even when the filters aren't I trained all. And our initialization is almost completely free to compute.
Related work Saxe et al. (2013) proposed to replace random i.i.d. Gaussian weights with random orthogonal matrices, constraint in which weights depend on each other and are thus, in some sense, "multivariate"; Xiao et al. (2018) also proposed an orthogonal initialization for convolutions. Similarly to these works, our initialization greatly improves the trainability of deep (depthwise) convolutional networks, but much simpler and constraint-free, being just random sample from multivariate Gaussian distribution. Zhang et al. (2022) suggests that the main purpose of pre- training may be to find a good initialization, and crafts a mimicking initialization based on observed, desirable information transfer patterns. We similarly initialize convolutional filters to be closer to those found in pre-trained models, but do so in a completely random and simpler manner. Romero et al. (2021) proposes analytic parameterization of variable-size convolutions, based in part on Gaussian filters; while our covariance construction is also analytic and built upon Gaussian filters, use them specify the distribution of filters.
Our contribution is most advantageous for large-filter convolutions, which have become prevalent in recent work: ConvNeXt (Liu et al., 2022b) uses
convolutions, and ConvMixer (Trock- man & Kolter, 2022) uses 9 x 9; taking the trend step further, Ding et al. (2022) uses 31 x 31, and Liu et al. (2022a) uses 51 x 51 sparse convolutions. Many other works argue for large-filter convolutions (Wang et al., 2022; Chen et al., 2022; Han et al., 2021).
Preliminaries This work is concerned with depthwise convolutional filters, each of which is parametrized by k X k matrix, where l (generally odd) denotes the filter's 's size. Our aim is to study distributions that arise from convolutional filters in pretrained networks, and to explore prop- erties of distributions whose samples produce strong initial parameters for convolutional layers. More specifically, we hope to understand the covariance among pairs of filter parameters for fixed filter size k. This is intuitively expressed as a covariance matrix 2 E R xx² with block structure: Σ has k x k blocks, where each block [Ma,]= E Reak corresponds to the covariance between filter pixel i.j and all other 2² - 1 filter pixels. That is, [Emgle.m [mg,mle, gives the covariance of pixels j and e, m.
In practice, we restrict our study to multivariate Gaussian distributions, which by convention are considered distributions over n-dimensional vectors rather than matrices, where the distribution W( E') has a covariance matrix E' E S4 where En = E/ represents the covariance between vector elements and j. To align with this convention when sampling filters, we convert from our original block covariance matrix representation to the representation above by simple reassignment of matrix entries, given by
(I) In this form, we may now easily generate a filter F E RR by drawing sample E Rt from (i,, E') and assigning Fi,j == fkitj- In this paper, we assume covariance matrices are in the block form unless we are sampling from a distribution, where the conversion between forms is assumed.
E/ j.ke- := EEndle, for S i, C, m S k.
Scope We restricted our study to the large-filter depthwise convolutions found in new ViT-style CNNs. namely the popular ConvMixer and ConvNeXt architectures. These networks consist of a patch embedding layer followed by alternating spatial- and channel-mixing steps. Both use depth- wise convolution for spatial mixing, but Cone uses pointwise convolution (equivalently. linear layers) for spatial mixing while ConvNeXt uses MLPs. ConvMixer uses no internal downsampling, while ConvNeXt includes several downsampling stages. Unlike normal convolutions, the filters in depthwise convolutions act each input channel separately rather than summing features over input channels. The depth of networks throughout the paper is synonymous with the number of depthwise convolutional layers. All networks investigated use a fixed filter size throughout the net- work, though the methods we present could easily be extended to the non- non-uniform case. Further, all methods presented do not concern the biases of convolutional layers.
2

---

Published as a conference paper at ICLR 2023
ConvMixer-2588, 1x1, (CIFAR-10)
Depit
Layer #11
Layer #5 of
Layer #8 of
Figure 1: In pre-trained mod- els. the covariance matrices of convolutional filters are highly- structured. Filters in earlier lay-
Unnormalized
ers tend be focused, becoming more diffuse as depth increases. Observing the structure of each block, we note that there is often a static, centered negative com- ponent and dynamic positive component that moves according the block's position. Often, covariances are higher towards the center of the filters.
Normaltzon Par-Block
2 THE COVARIANCES OF TRAINED CONVOLUTIONAL FILTERS AND THEIR TRANSFERABILITY ACROSS ARCHITECTURES
In this section, we propose a simple starting point in our investigation of convolutional filter covari- ance structure: using the distribution of filters from pre-trained models initialize filters in new models, a process we term covariance transfer. In the simplest case, we use a pre-trained model with exactly the same architecture the model to be initialized; we then show that actually transfer filter covariances across very different models.
Basic method. We use E 1 1..... D to denote the 7th depthwise convolutional layer of model with D layers. We denote the j I,..., filters of the 7th pre-trained layer of the model by Fig for a model with H convolutional filters in a particular layer (i.e., hidden dimension H) and /² to denote the filters of new, untrained model. Then the empirical covariance of the filters in layer i is
......
Σ, Covfvec(Fa);..... sec(Fi)))
with the mean μ4 computed similarly. Then the new model can be initialized by drawing filters from the multivariate Gaussian distribution with parameters 14, Σ;;
FJJ (W,, E,) for l.... H....,,,ee H,iel....
(3)
Note that in this section, distributions from which to initialize. However, we found that the mean can be assumed to be zero with little change in performance, and we focus solely on the covariance later sections.
the means of the filters in addition the covariances define the
Experiment design. We our initialization methods primarily on ConvMixer since it is simple and exceptionally easy to train on CIFAR-10. We use FFCV (Leclere et al., 2022) for fast data loading using our own implementations of fast depthwise convolution and RandAugment (Cubuk et al., 2020). To demonstrate the performance of our methods across a variety of training times, we train for 20, 50, or 200 epochs with a batch size of 512, and we repeat all experiments with three random seeds. For all experiments, we use a simple triangular learning rate schedule (see Appendix A.1) with the AdamW optimizer, a learning rate of ,01, and weight decay of .01 as in Trockman & Kolter (2022).
Most of our CIFAR experiments use a 20n- with either patch size or 2: a ConvMixer- H/D has precisely D depthwise convolutional layers with // filters each, ideal for testing our initial covariance transfer techniques. We train ConvMixers using popular filter sizes 3, 7, and 9, as well as 15. We also test methods ConvNeXt (Liu et al., 2022b), which includes downsampling unlike ConvMixer; we use a patch size of or 2 with ConvNeXt rather than the default to accomodate relatively small CIFAR-10 images, and the default 7 filters.

---

Published as a conference paper at ICLR 2023
For most experiments, we provide two baselines for comparison: standard uniform initialization, the standard in PyTorch (He et al., 2015), as well as directly transferring the learned filters from a pre-trained model to the new model. In most cases, expect new random initializations to fall uniform and direct transfer initializations. For our covariance transfer
between the performance experiments, trained variety of reference models from which compute covariances; these all trained for the full 200 epochs using the same settings as above.
Frozen filters. Cazenavette et al. noticed that ConvMixers with 3 3 filters perform well
even when the filters are frozem: that is, the filter weights remain un-
changed over the course of training, receiving no gradient updates. As we are initializing filters from the distribution of trained filters,
CM CM2/12/12 1x1 Patches Grad. Step
we suspect that additional training may not completely neces- sary. Consequently, in all experiments we investigate both models with thawed filters as well as their frozen counterparts. Freezing filters removes of the two gradient calculations from depthwise convolution, resulting in substantial training speedups as kernel size increases (see Figure 2). ConvMixer-512/12 with kernel size x 9 is around 20% faster, while 15 x 15 is around 40% faster. Fur- ther, good performance in the frozen filter setting suggests that an initialization technique is highly effective.
CUDA Yimw (msi
300
Rrezon Thowed
200
300
5
11
13
15
Flbeer Sice
Figure 2: The backward pass is faster with frozen filters.
2.1 RESULTS
The simplest case of covariance transfer (from exactly the same architecture) a fairly effective ini- tialization scheme for convolutional filters. In Fig. 3, note that this case of covariance transfer (group B) results in somewhat higher accuracies than uniform initialization (group A), particularly for 20- epoch training; it also substantially improves the case for frozen filters. Across all trials, the effect of using this initialization is higher for larger kernel sizes. In Fig. 8, we show that covariance transfer uniform initialization quickly fades.
(gold) initially increases convergence, but the advantange As expected, covariance transfer tends to fall between the performance of direct transfer, where we directly initialize using the filters of the pre-trained model, and default uniform initialization (see group D in Fig. 3 and the green curves in Fig. 8).
However, acknowledge that is not appealing to pre-train models just for initialization tech- nique with rather marginal gains, so we explore the feasibility of covariance transfer from smaller models, both in terms of width and depth.
Narrower models. We first see if it's possible to train narrower reference model to calculate filter covariances to initialize wider model; for example, using a ConvMixer-32/8 to initialize a 255/ In Figure 4, we show that the optimal performance surprisingly comes from the covariances of a smaller model. For filter sizes sizes greater than 3, the covariance transfer performance increases with width until width 32, and then decreases for width 256 for both the thawed and frozen cases. We plot this method in Fig. 3 (group C), and note that almost uniformly exceeds the performance of covariance transfer from the same-sized model. Note that the method does not change; the covariances are simply calculated from a smaller sample filters.
Shallower models. Covariance transfer from shallow model to a deeper model is somewhat more complicated, as there is no longer a one-to-one mapping between layers. Instead, we linearly interpolate the covariance matrices the desired depth (see Appendix A.1 for more details). Surprisingly, we find that this technique is also highly effective: for example, for a 32-layer-deep ConvMixer, the optimal covariance transfer result is from an 8-layer-decy CinvMixer, and 4-deep models are also quite effective (see Figure 4).
Different patch sizes. Similarly, is straightforward to transfer covariances between models with different patch sizes. We find that initializing ConvMixers with 1 x patches from filter covariances of CinvMixers with x 2 patches leads to a decrease in performance relative to using a reference model of the correct patch size; however, using the filters of a 1 x patch-size ConvMixer to initialize a 2 x patch size ConvMixer increases performance (see group vs. group B in Fig. 9). Yet, in both cases, the performance is better than uniform initialization.
4

---

Published as a conference paper at ICLR 2023
ConvMixer-256/8 Patch Size 2x2
100: RccuPac 113
0.90
2.85
A Uniform
Cov, transfer from CM 32
Cov. transfer from 23-256/88
Direct transfer from GM-2566
Thawed
3x3
7x7
Sxc
15x15
Monv 256/24 Patch Size
0.05
T46l AicuRACy (1K)
0.40
32/24
Thawed
Epoche:
15x15
ConvMixer-256P Patch
0.94
Teat Acaneeyy
0.92
0.88
Cos.
E.Ours
Cov. transfer from OM 2368
Direct transffer from CM-256/6
Thowed
Frozen
Opoche: 20 50 200
3x3
Fiiter Size
Figure 3: CIFAR- 10 accuracy for uniform initialization (A), baseline covariance transfer (B-D), and our custom initialization results (E).
Covariance
ConvMixer-({. [2, 4,8, 32, 256)/8
ConvMixer-
0.95
Tel Ay (86)
1632
2
256
2
Tharee Frozen
9x
15:15
ConvMixer (56/
256/32 (Patch 2x2)
Bal. Rccurayy (18)
0.55
152432
16 sa
1634
1524
0.90
3.85
Trazen
Eprcitc
3.80
9x9
25:15
Kornel
Figure 4: CIFAR-10 experimental results from initializing via covariances from narrower (top) and shallower (bottom) models. The numeric annotations represent the width (top) and depth (bottom) of the pre-trained model we use to initialize. U represents uniform initialization.
Different filter sizes. Covariance transfer between models with different filter sizes more chal- lenging, as the covariance matrices have different sizes. In the block form, we mean-pad or clip each block the target filter size, and then bilinearly interpolate over the blocks reach a correctly-ste covariance matrix. This technique is still better than uniform initialization for filter sizes larger than 3 (which naturally has very little structure transfer), especially in the frozen case (see Fig. 9)
Discussion. We have demonstrated that it is possible initialize filters from the covariances of pre-trained models different widths, depths, patch sizes, and kernel sizes; while some these techniques perform better than others, they almost all better than uniform initialization. Our ob-
5

---

Published as a conference paper at ICLR 2023
servations indicate that the optimal choice of reference model is narrower or shallower, and perhaps with a smaller patch size or kemnel size. We also found that covariance transfer from ConvMixers trained on ImageNet led to greater performance still (Appendix A). This suggests that the best co- variances for filter initialization may be quite unrelated to the target model, ie., model independent.
3 D.I.Y. FILTER COVARIANCES
Ultimately, the above methods for initializing convolutional filters via transfer are limited by the necessity of trained network from which form filter distribution, which must be accessible at initialization. We thus use observations on the structure of filter covariance matrices to construct our own covariance matrices from scratch. Using our construction, we propose a depth-dependent but simple initialization strategy for convolutional filters that greatly outperforms previous techniques.
Visual observations. Filter covariance matrices in pre-trained Cixerixers and ConvNeXts have great deal of structure, which we observe across models with different patch sizes, architectures, and data sets; see Fig. 1 and 32 for examples. In both the block and rearranged forms of the covariance matrices, we noticed clear repetitive structure, which led an initial investigation on modeling covariances via Kronecker factorizations; see Appendix A for experimental results. Beyond this, we first note that the overall variance of filters tends to increase with depth, until breaking down towards the last layer. Second, we note that the blocks of the covariances often have static negative component the center, with dynamic positive component whose position mirrors that of the block itself. Finally, the covariance of filter parameters is greater in their center, i.e., covariance matrices are at first centrally- focused and become more diffuse with depth. These observations agree with intuition about the structure of convolutional filters: most filters have the greatest weight towards their center, and their parameters are correlated with their neighbors.
Constructing covariances. With these observations in mind, we propose a construction covari- ance matrices. We fix the (odd) filter size k E N+, let 1 E RR** be the all-ones matrix, and, building block for our initialization, use unnormalized Gaussian-like filters Zo RRkx with single variance parameter o, defined elementwise by
for 1 S i,jssk k. 2 + + 12
(Z,).. exp
(4)
2a
Such construction produces filters similar to those observed in the blocks of the Layer #5 covari- ance matrix Fig. 1.
To capture the dynamic component that moves according the position of its block, we define the block matrix C E x2 with k x blocks by
[Ct,]] Shift Zo, (,,
(5)
where the Shift operation translates each element of the matrix i and j positions forward in their respective dimensions; see Appendix E for details. We then define two additional components, both constructed from Gaussian filters: a static component S = 1 & Z, E R***k² xk² and a blockwise mask component M = Z ② 1 E R4 x which encodes higher variance as pixels approach the center of the filter.
Using these components and our intuition, we first consider Σ = M o (C 1).) where @ is an elementwise product. While this adequately represents what we view to be the important structural components of filter covariance matrices, it does not satisfy the property [Ei,le,, = [em, (i.e,, covariance matrices must be symmetric, accounting for our block representation). Consequently, instead calculate its symmetric part, using the notation as follows to denote a "block- trock-transpose".
B = Σ Emgle, = | for 1 < i,j, m mmmmm k.
(6)
Equivalently, this is the perfect shuffle permutation such that (X@Y) Y with X, Rixt First, we note that " C due to the definition of the shift operation used in Eq. 5 (see Ap- pendix E). Then, noting that B M and B = S by the previous rule, define construction

---

Published as a conference paper at ICLR 2023
2
Figure 5: Our convolutional covariance matrix construction with a = 2
Variance Schedule
Figure 6: How our initialization changes with depth. Variance increases quadratically with depth according to a schedule which can be chosen through visual inspection of pre-trained models through grid search. Here we use the parameters Oo .5, Uo .5, Gg 3.
Depth
of Σ to be the symmetric part of Σ (where C, S, M are implicitly parameterized by the of Zo):
E )-)(
(C -C- [ (M @ (C - }] (M @ (C - - = M @ (C (C--5))- - -C- M
(7)
= ] (C
(8)
(9)
While E is now symmetric (in the rearranged form of Eq. 1), is not positive semi-definite, but can easily be projected to 54", as is often done automatically by multivariate Gaussian procedures. We illustrate our construction in Fig. 5, and provide an implementation in Fig. 30.
Completing the initialization. As explained in Fig. we observed that in pre-trained models, the filters become more "diffuse" as depth increases; we capture this fact in our construction by increas- ing the parameter with depth according to simple quadratic schedule; let be the percentage depth, i.e., d = +-1 for the 7th convolutional layer of model with D total such layers. Then for layer i, we parameterize our covariance construction by variance schedule:
(dd = do u + da
(10)
where do, Vo Ga jointly describe how the covariance evolves with depth. Then, for each layer 1-1 1,..., , D, we compute カーΣ and initialize the filters as Fl,g ~ (0, (o))) for j E l,...,
We illustrate our complete initialization scheme in Figure
RESULTS
In this section, we present the performance of our initialization within ConvMixer and ConvNeXt on CIFAR-10 10 and ImageNet classification, finding it to be highly effective, particularly for deep models with large filters. Our new initialization overshadows our previous covariance transfer results.
Settings of initialization hyperparameters do. Un, and a, were found and fixed for CIFAR-10 experiments, while two such settings were used for ImageNet experiments. Appendix C.I contains full details on our (relatively small) hyperparameter searches and experimental setups, as well as empirical evidence that our method is robust to a large swath of hyperparameter settings. Additional experiments on more datasets and baseline initializations may be found in Appendix B.
4.1 10 RESULTS
Thawed filters. In Fig. 3, we show that large-kernel models using our initialization (group E) outperform those using uniform initialization (group A), covariance transfer (groups B, C), and
7

---

Published as a conference paper at ICLR 2023
even those directly initializing via learned filters (group D). For 2 x 2-patch models (200 epochs), relative to uniform, our initialization causes up to a 1.1% increase in accuracy for MonxMixer-256/8,, and up to 1.6% for Co1vM1xer-256/24. The effect size increases with the the filter size, and is often more prominent for shorter training times. Results are similar for 1 x 1-patch models, but with smaller increase for 7 x 7 filters (0.15% vs. 0.5%). Our initialization has the same effects for ConvNeXt (Fig. 7). However, our method works poorly for 3 x 3 filters, which we believe have fundamentally different structure than larger filters; this setting is better-served by our original covariance transfer techniques.
In addition improving the final accuracy, our initialization also drastically speeds up convergence of models with thawed filters (see Fig. 8), particularly for deeper models. A ConvMixer- 2266/6 with 2x 2 patches using our initialization reaches 90% accuracy approximately 50% fewer epochs than uniform initialization, and around 25% fewer than direct learned filter transfer. The same occurs, albeit to a lesser extent, for initialization parameters for both patch sizes to demonstrate robustness to parameter choices.
patches but note that for this experiment we used the same
Frozen filters. Our initialization leads to even more surprising effects in models with frozen filters. In Fig. 3, we see that frozen- filter 2 x 2-patch models using our initialization often exceed the performance of their uniform, thawed-filter counterparts by significant mar- gin of 0.4% - 2.0% for 200 epochs, and an even larger margin of 0.6% 5.0% for 20 epochs (for large filters). That is, group E (frozen) consis- tently outperforms groups A-D (thawed), and in some cases even group E (thawed), especially for the deeper 24-layer ConvMixer. While this effect breaks down for 1 patch models, such frozen-filter models still see accuracy increases of 0.6% 3.5%. However, the effect can still be seen for 1 x 1-patch ConvNeXts (Fig. 7). Also note that frozen- filter models can be up to 40% faster to train (see Fig. 2), and may be more robust (Cazenavette al.).
CanvNext-aae atto
Woc.ecoee (180)
0.8
Uriffrm
Direct Irandeer
83
transter 11B8 wicktt
Epochs: 20 a 50 200
Patch
Figure 7: Our init also improves ConvNeXt's ac- curacy on CIFAR-I0 (group E vs. A).
ConvMixer-256/16
Patch Size: 2x2, Filter Size: 9x9
ConvMixer-256/16
Patch Size: 1x1, Filter Size: 9x9
95
Test Accuracy
90
Test Accuracy
Filter Initi Initiaiiztionn
93
Filter Initializat on Uniform
Uniform
Cow, transfer
Cov. transfer
Direct transfer
91
Direct transfer
Ours
Ours
84
100
100
Epochs
Epochs
Figure 8: Convergence plots: each data point runs through full cycle of the LR schedule, and all points are averaged over three trials with shaded standard deviation.
4.2 IMAGENET EXPERIMENTS
Our initialization performs extremely well on CIFAR-IO for large-kernel models, almost always helping and rarely hurting. Here, cxplore if the performance gains transfer to larger-scale Ima- geNet models. We observe in Fig. 32, Appendix F that filter covariances for such models have finer- grained structure than models trained on CIFAR-10, perhaps due using larger patches. Nonethe- less, our initialization leads to quite encouraging improvements in this setting.
Experiment design. We used the "A1" training recipe from Wightman et al. (2021), with cross- entropy loss, fewer epochs, and a triangular LR schedule as in Trockman & Kolter (2022). We pri- marily demonstrate initialization for 50-epoch training, the difference between initializations is most pronounced for lower training times. We also present two full, practical- -ccale 150-epoch experiments large models. We also included covariance transfer experiments in Appendix F.
8

---

Published as a conference paper at ICLR 2023
Table 1: ImageNet-Ik I accuracy from various architectures and initializations. "Ours" denotes proposed initialization. Bold indicates best within architecture and category (frozen or thawed).
Model
THAWED
FROZEN
Architecture
Filter Size
Patch
Uniform
Ours
Ours
Uniform
Ours
Ours
Size
Epochs
5 .5
25 1.0
.15 .25
.15 .25 1.0
Con-MMixe-5112/1
14
67.03
67.41
67.34
60.47
64.43
64.12
ConvMixer-511224
14
50
67.76
69.60
69.52
62.50
66.57
66.38
Con-Mixxe-511232
14
65.00
68.78
68.84
55.79
66.59
66.32
ConvMixer-1024/11
14
50
73.55
73.62
73.75
68.96
71.48
71.30
ConvMixer-
74.19
75.33
75.50
69.65
73.42
74.31
Com-Mixcr-1024/32
14
50
72.18
74.98
74.95
64.94
73.00
73.12
ConvMixxe-512/11
7
72.05
71.92
72.32
67.25
68.91
68.92
ConwNeXt-AAto
4
50
69.96
67.84
68.06
51.43
64.52
64.43
ConwNeXt-Tiny
4
50
75.99
76.08
77.11
64.17
74.62
75.21
ConvMixer-1536624
14
150
80.11
80.28
ConvNeXt-Tiny
4
150
79.74
79.81
Thawed filters. On models trained for 50 epochs with thawed filters, our initialization improves the final accuracy by 0.4% 3.8% (see Table I). For the relatively-shallo which we tuned the initialization parameters, we see gain of just 0.4%; however, when increasing larger gains of 1.8% and 3.8%. respectively, and similar trend the depth to 24 32, 1024 models. Our initialization also boosts the accuracy of the 18- among the wider ConvMixer- layer ConvNeXt-Tine from 76.0% 77.1%; however, it decreased the accuracy of the smaller, 12-layer ConyNeXt-Atto. This is perhaps unsurprising, seeing as our initialization seems to be more helpful for deep models, and we used hyperparameters optimized for model with substantially different patch and filter size.
1in on
Our initialization is also beneficial for more-practical 150-epoch training, boosting accuracy by around 0.1% on both ConvMixer-15366 the effect is small, this demonstrates that our initialization is still helpful even for longer training times and very wide models. We expect that within deeper models and with slightly more parameter tuning, our initialization could lead to still larger gains full-scale ImageNet training.
and ConvNeXt-Tiny (see Table 1, bottom rows). While
Frozen filters. Our initialization is extremely helpful for models with frozen filters. Using our ini- tialization, the difference between thawed and frozen-filter models decreases with increasing depth, i.., it leads to 2 - 11% improvements over models with frozen, uniformly-initaalzze filters. For ConvMixer- 1w-Mixxer-102432.. the accuracy improves from 64.9% to 73.1%, which over 1% better than the corresponding thawed, mmiformll-initializzn model, and only 2% from the best result using our ini- tialization. This mirrors the effects we saw for deeper models on our earlier CIFAR-10 10 experiments. We similar effect for ConvNeXt- Tiny, with the frozen version using initialization achiev- ing 75.2% accuracy vs. the thawed 76.0%. In other words, our initialization so effectively captures the structure of convolutional filters that it is hardly necessary to train them after initialization; one benefit of this is that substantially speeds up training for large-filter convolutions.
CONCLUSION
In this paper, we proposed a simple. closed-form, and learning-free initialization scheme for large depthwise convolutional filters. Models using our initialization typically reach higher accuracies more quickly than uniformly-inttiaizzed models. We also demonstrated that our random initializa- tion of convolutional filters is so effective, that many cases, networks perform nearly as well (or even better) if the resulting filters do not receive gradient updates during training. Moreover, like the standard uniform initializations generally used in neural networks, our technique merely samples from particular statistical distribution, and it is thus almost completely computationally free. In summary, our initialization technique for the increasingly popplar large-kernel depthwise convolution operation almost always helps, rarely hurts, and is also free.

---

Published as a conference paper at ICLR 2023
REFERENCES
George Cazenavette, Joel Julin, and Simon Lucey. Rethinking the role of spatial mixing.
Yukang Chen, Jianhui Liu, Xiaojuan Qi, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Scaling up kernels 200:::222..055555 2022.
in 3d cnns. arXiv preprint
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Xiaohan Ding, Xiangyu Zhang. Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11963-11975, 2022.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, and Jingdong Wang. On the connection between local attention and dynamic depth-wise convolution. In International Conference on Learning Representations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Commuicaatinn of the ACM, 60(6):84 00.88899900200 2017.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. ffev. https: / g thub. com/ wiwffe//bseeeeee f fcv/, 2022. commit f253865.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Mykola Pech- enizkiy, Decebal Mocanu, and Zhangyang Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. arXiv preprint arXiv:220.003620 2022a.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11996-119999 11986, 2022b.
David W Romero, Robert-Jan Bruintjes, Jakub Tomezak, Erik J Bekkers, Mark Hoogendoorn, and Jan C van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. arXiv preprint arXiv:21 00.22110.080 10.08059, 2021.
Andrew M Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 2013.
Asher Trockman and Zico Kolter. Patches are all you need? 2022.
arXiv preprint arXiv:2201.09792.
Zeyu Wang, Yutong Bai, Yuyin Zhou, and Cihang Xie. Can cnns be more robust than transformers? 2022.
arXiv preprint
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure timm. arXiv preprint 1rX:v:2110.00476, 2021.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and mean field theory of enns: How to train 10,000-layer vanilla con- volutional neural networks. In International Conference on Machine Learning, pp. 5393-5402. PMLR, 2018.
Yi Zhang, Arturs Backurs, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, and Tal Wagner. Unveiling transformers with lego: a synthetic reasoning task. arXiv preprint 2206002206.,4311, 2022.
10

---

Published as a conference paper at ICLR 2023
A ADDITIONAL CIFAR RESULTS
Cov. transfer across patch sizes: ConvMlixar-25e 256/8 Patch Size: 2x2
Cov. transfer across fiter sizes: ConvMlxar-256R 56/8 Patch Sze:: 2x2
0.00
0.85
a Cov CH 2565
Cow. 236/8
CM-22
Cov. CM 32.8 2x2
Thawed
T0r CM 2568 1xl CM-23588
Epocts:
Figure 9: Initializing via covariances from models with different patch (left) and filter sizes (right)- Left: Lowercase denotes initializing from patch size x 1, and uppercase 2 2. Right: Annotations denote the reference filter size, U is uniform.
0.85
Esccher
Unifomm
Dinect CM (CIMAR)
JmNees)
0.80
9x
ConvMixer-256i1 25612 Patch
0.95
Jeurney
0.90
Unitemm
CM-32 (CIFAR)
from CM-512 (mhin)
Cov.
0.85
Figure 10: Using filter distributions from pre-trained ImageNet models to initialize models trained on CIFAR-10 is also effective (represented by groups E and F, with hatch marks).
ConvMixer 256/8
Patch Size: 2x2, Filter Size: 9x9
ConvMi xer 256/8
Potch Size: 1x1, Filter Size: 9x9
94
94
Test Accuracy (95)
92
Test Accuracy (%)
93
90
Filter Initialization
92
Filter initializati on
88
Cov, transter
Cov: transter
Ours
Ours
Epochs
Epochs
Figure 11: Convergence plots: each data point runs through a full cycle of the LR schedule, and all points are averaged over three trials with shaded standard deviation.
Covariance structure. As a first step towards modeling the struc- ture of filter covariances, we replaced covariances with their Kronecker- factorized counterparts using the rearranged form of the covariance ma- trix defined Eq. (1), i.e., this slightly improved performance over unfactorized covariance transfer (see Fig. 12), suggesting that filter covariances are not only eminently transferrable for initialization, but that their core structure may be sim- pler than meets the eye. Kronecker factorizations were computed via gradient descent minimizing the mean squared emror.
ConvMixer-2566B Patch Size 2x2
@ A where E St. Surprisingly,
covariance weeer factorizen ic kranecker factortted IC (
Figure
12: Approx.

---

Published as a conference paper at ICLR 2023
Empirical Covariance
Our Covariance
Layer
Learned Filters
Sampled Filters
Sadpled Filters
Empirical Covariance
Our Covariance
Layer
Learned Filters
Sampled Filters
Sampled Filters
Empirical Covariance
Our Covariance
Layer #8
Learned Filters
Sampled Filters
Sampled Filters
Figure 13: Filters learned generated for ConvMixer-25677 with 2 patches and x 9 filters trained on CIFAR-10: learned filters (left), filters sampled from the Gaussian defined by the empiri- cal covariance matrix of learned filters (center), and filters from initialization technique (right).
12

---

Published as a conference paper at ICLR 2023
A.1 ADDITIONAL EXPERIMENT DETAILS
epochs 130
Ir_sched lambda t: op. in-iiterp((t), \
Leariing Rale
0.0075
IG, =pochs.2//5, epochs.4//5, epechsl,
Io, Ir_max, 1s_mex/20.0, 011 [0]
0.0050
5 for spach in range tepacha)
y) enumerate :loader):
8
lepsched oot.param_grrouptoo (1cm1e)
11/len (Loader))
0.0000
60
80
100
Epochs
Figure 14: Implementation and visualization of the piecewise triangular learning rate schedule used for all experiments. We used the implementation from Trockman & Kolter (2022).
def linear inniirrinteerplationnn
depth, :o_depth):
cove: 1ist of covartance matrices irpayrray)
fron_depth: depth of sodel from which
transferring
# te_depth: depth of model we are transferring to
from_knota up ap.erange (0, from_depth) [from_depth 5 to to_knota = ng.arange to_depth) (to_depth -
ret
8
for knot in tt_kneta:
for i range (len(trooooo if knot >- fren_kncts and knot <- kron_knots
s_weight (knot frem_kncts(E+1]
from_knots + 111 b_waight (knot from_ (n)tatt)aa / (fron_knots (1+1] (ron_
B
ret ta_ cova[i] b_weight [1+]] break zeturn ret
15
Figure 15: Implementation of linear interpolation used for transferring covariances from shallower to deeper models. We treat the depth of layer as a percent the full depth of the network, iee. our notion of depth is always in the interval [0, 1].
13

---

Published as a conference paper at ICLR 2023
B ADDITIONAL BASELINES AND DATASETS
ConwMixxer-26688 Patch Size 2x2
0.95
Taxt Acurncy (894
0.05
A uniform
Zoros
Ours
8 DiSc
arthogand
0.80
3x3
7x7
9x0
15x15
Fiker Size
Figure 16: Our initialization technique outperforms alternatives (uniform, dirac, all-zero, and or- thogonal initialization) for filter sizes larger than 3, especially for short-duration training. Note that for dirac and all-zero initialization (B, C) in the frozen case, accuracies are Rs 60% and thus are not on the graph. Results CIFAR-10.
ConvMixer-64j4 Patch Size 2x2 (Data set: SVHNI
Tex Accnney (150
0.900
0.925
0.900
0.875
Zoros
0.850
Orthagonnl
Thawad
Frozen
Epochs: 23 a 50 200
Fiter
Our initialization also generally outperforms or matches baselines on the Street View House Numbers (SVHN) dataset. Since this task is relatively easy, we used much smaller Con4Mixer-64/4. The advantage our method is most noticeable for short-duration (20 epoch) training and the frozen-filted setting.
Figure 17:
ConvMixer- 128/8 Patch Size 2x2 (Dataset: EuroSATI
0.975
Tea Mutccuurcc 1h50
3,900
Kimfom-
Orthogonal
Thawed
Frozen
Epochs:
Figure 18: Our initialization is again competitive with baselines the EuroSAT dataset, where used a random 75/25 train/test split. We found our models converged quickly on this task, and thus used fewer epochs for all experiments (10, 20, and 50). Our initialization approximately matches uniform initialization and outperforms other methods in the 50-epoch setting, but performs better in the 10- and 20- epoch settings. It is also particularly effective for frozen filters.
ConvMixer-25668 Patch Size 2:2 (Dataset: CIFAR 100)
0.70
Tec: Recureyy hre
9.65
Orthogonal
Thawed
Freze
Epochs: 20 0050 200
9.0
Figure 19: Our initialization is particularly advantageous the more-difficult CIFAR- 100 dataset, with 50-epoch training meeting or even surpassing 200-epoch uniformly-inin -inttaliinittalizzee training.
14

---

Published as a conference paper at ICLR 2023
ConvMixer:26/116 Patch Size 4x4 (Dataset: Tiny ImageNet)
Test Accurscy 088)
0.50
0.45
Uniform
Zeros
E Ours
Dirac
Orthogonal
Thawed
Frozen
Epochs:
200
7x7
9x3
Filter Size
ConvMixer-512/16 Patch (Dataset: Tiny ImageNet)
0.60
25'
0.55
0.50
A Uniform
C Zenos
Dur
Orthogonal
Thowed
Frozen
Epochs: 20
50
200
0.45
7x7
9x3
Figure 20: Our initialization is also helpful on Tiny ImageNet. Here we increased the patch size to 4x4 to accomodate the increased input size of 64x64 (for computational efficiency).
Dataset: ConvMixer-256/16
ConvMixer-25611
Size:
94
Test Accuracy (16)
92
Test Accuracy (%)
65
90
Alter Initialization
Filter Initialization
Uniform
Uniform
88
Dinc
60
Zeros
Zeros
86
Orthogonal Ours
55
Orthagonal
84
50
60
80
100
80
100
Epocts
Epochs
Figure 21: Compared to other initialization techniques, ours results in faster convergence when training 20n/Mixer-256/16 CIFAR-10 10 and CIFAR-100. That is, often train models for fewer epochs when using our initialization to achieve results comparable to those from using other initialization techniques.
Dataset: CIFAR 10
Datasel: CIFAR
ConvMbxer
Patch Size: 2:2, Filter Size: 7x7
CanvMixer 256/8
Patch Size: 2x2, Filter Size: 7x7
100
R Initialization
90
Test Accuracy (96
80
Test Accuracy (%)
Ours
60
70
40
60
Initialization Unitorm
20
40
200
1000
5000
10000
15000
20000
25000
30000
Training Points
Training
(a)
(b)
Figure 22: We investigated how our initialization affects data efficiency by training on random (smaller) subsets of CIFAR-10. Each trial averaged over such subsets. In the very low-data setting (a), we trained for 300 epochs to compensate for the smaller number of iterations overall. In (b), we trained for 100 epochs. In some cases, using our initialization comparable doubling the number of training points.
15

---

Published as a conference paper at ICLR 2023
C HYPERPARAMETER GRID SEARCHES & EXPERIMENTAL SETUP
CIFAR-10 hyperparameter search. We chose an initial setting of our method's three hyperpa- rameters via visual inspection, and then refined them via small-scale grid searches. For CIFAR-I0 experiments, we searched over parameters for Con-Mixer-256/8 with frozen 9 x 9 filters trained for 20 epochs, and chose dy .08, 2/ .37, a 2.9 for 2 x 2-patch models, and found the optimal parameters for 1 x 1-patch models to be approximately doubled. However, note that our initialization is quite robust different parameter settings, with the difference from our doubling choice being less than 0. 0.1% (see Figure 23). We used the same parameters across all kernel sizes, as well as for ConvNeXt, choice which likely sub-optimal; our search only serves as a rough heuristic.
ImageNet- I hyperparameter search. We did a small grid search using a Conv-Mixer-512/11 with 14 14 patches and filters trained for 10 epochs ImageNet-1k 1k (see Appendix F), from which we chose two candidate settings:
11,, fi = .5, dg = .25 for frozen-filter models and do .15, Uo 0.25, a₂ 1.0 for thawed models. We use these parameters for all the ImageNet experiments, even for models with different patch and kernel sizes (e.g. ConvNeXt). This demonstrates that hyperparameter tuning optional for our technique; its transferability not surprising given our results in Sec. 2.
C.I CIFAR-10 GRID SEARCHES
Figure 23: Grid search over initialization parameters do, Ua Gg for ConyMixer-25888 with 9 x frozen filters and 2 patches trained for 20 epochs on CIFAR-10. Note that the performance of uniform initialization is only =85%, i.c., almost all choices result in some improvement.
16

---

Published as a conference paper at ICLR 2023
Figure 24: Grid search over initialization parameters do, Va Og for ConwMixer-258/8 with frozen filters and 1 x 1 patches trained for 20 epochs on CIFAR-10. Note that the performance of uniform initialization is only =s88%, i.e., almost all choices result in some improvement.
Figure Grid search over initialiation parameters do. Vo Gr for aon on CIFAR-10 with frozen filters and 1 x patches trained for 20 epochs on CIFAR-10. Note the baseline performance with uniform initialization is around 80%, i.e., compared to ConvMixer there are more potentially disadvantageous parameter combinations.
17

---

Published as a conference paper at ICLR 2023
Figure 26: Grid search over initialiation parameters do, fly G ConvNeXt-atto on CIFAR-10 10 with frozen filters and 1 patches trained for 20 epochs, using the "sawtooth" variance schedule (see Fig 27) to account for downsampling layers. While this perhaps shows better robustness to parameter changes than Fig. 26, the effect could also be due to effectively dividing the parameters by two.
ConvMixer Depth
ConvNeXt "Depth"
1.0
0.6
0.6
0.4
0.4
0.0
0'2
0.4
0.0
0'2
0.4
0.8
ConvMixer Variance
ConvNeXt Init. Variance
2
1.0
1
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 27:
Proposed stepwise variance schedule for ConvNeXt, i.e., model including downsam- pling layers. In our experiments, we saw no advantage to using this scheme.
18

---

Published as a conference paper at ICLR 2023
C.2
IMAGENET GRID SEARCHES
ScA
6:7
513
919
31.0
12.2
52.
223
510
512
523
11.0
522
914
32A
12.
523
$3.7
91.9
52.1
22.4
52.1
51.7
520
P23
21.
514 11.7
22.
21.7
02
52.
52.2
$2A
51.6
5..0
12
52.6
12
51.4
51.
N2 52.
52
-514
03
31.
32.4
$1.
510
55
522 52.
52
011
0.25
0.5
2.0
4.0
6.2
0.1
0.23
63
10
20
4.6
6.2
11
5.25
25
3.1
4.0
5.0
Acodertrr
Naturic Asculasian
Figure 28: Frozen filters: Grid search over initialization parameters for ConwMixer-512/12 with 14 x 14 patches and 9 x 9 filters, 10 epochs.
Veriencer 6.100
55.53
55 50
55.56
55.35
55.22
55255
54.6
11.
11
551
54.
55
112
14
943
542
543
50.7
34.75
15.1
543
54.
$6.93
15.4
15.4
C23
$6.25
555
66.1
14:44
53
$4.03
55.4
15.4
55 $4.7
2,4
-553
15.5
55.2
15
14.0
6.5 20
26
0.5
Figure 29: Thawed filters: Grid search over initialization parameters for ConvMixer-512/12 with 14 x 14 patches and 9 x 9 filters, 10 epochs.
19

---

Published as a conference paper at ICLR 2023
D IMPLEMENTATIONS
D.1 TERSE IMPLEMENTATION
1 def ConvCov (x, sl:
Gauss (k, mx, my, 5):
np.
1+)))
zeros (1K, k)
for 1, J in mp.ndindex(k,,)
for i, j in EP ndinddxexxx
Cik+iskvitk, keickej-k]] 1k, 5 2,1 - Gausa tk, /21/22///22))///// ap, ones (1k, ap.kzon (1, np.pkkonff2
13-xx-k//2-13
(i-my-k//2-1138 k//2)
= (fex x//2) resti, 31 ap ap.exp.......e............ (-D.5x)) return res res. reshspe ik, k)
7
zeturn 0.5 (8 x (0 SI + C
Figure 30: Implementation of our convolution covariance construction NumPy.
der Initialize (wconV, s0, C, kg, scoav.ehape 0.5
2
s0
d++2
cov ConvCov (ks, 51 reshapp (ka,)v4) ttranspeep (0, 1, .reshape (k5vv),)x2)
filters np. www.randoom.e www.wmmawtiaaaaaamm aaweaaatettaaaeeeeaeaaaeeeeeeeee aate nornal (np. (ks*k2), cov, ze, aconv.data
tensorr rew.hsaeeerrrrraeeeeeeeee.. ks, ksl, daype-woons daype, device-wconv deviice)
Find depthwise convolutional layers
convs Ix for in model. moduless isinstence (x, nn. Conv2d)
and len weight. shape) == and v.veight Saaapt
Initialize them according variance achedule
for i, conv in enumerate (convs):
Initialize (conv, weight, / (convo)))) 1), 0.16, 0.32, 2,88)
Figure 31: Code to use our covariance construction and variance schedule to initalize depth- wise convolutional layers in PyTorch. wconv is the weight of a depthwise convolutional layer (nn. Conv2d), and d E [0, 1] is its depth as a fraction of the total depth.
SHIFT FUNCTION DEFINITION PROOF
For given matrix Z Rexk (say, a Gaussian kernel centered at 0, 0 the top left of the filter), assume the shift operator is defined as follows:
Shift(Z, 8x, σy)1, = ((i+-6a)) R.(j++55))
(11)
Then, if
[Ci,]] = (,,t(
(12)
and the operation (.) is defined by
DB = Σ
Eaglem = =(ilmmaa.
(13)
and
[Ct.le. = Shift( i,i,j)em Z(i+e) mad (2))+m)) mod
(14)
1Cemlis = Shift(Z, E, m)), Z(e+s) mod k. mod k
(15)
this shows that (Cislem (Ce,mli,s for all S i.j, l, m S k (i.e., C is "block-symmetrice" which shows = CB.
20

---

Published as a conference paper at ICLR 2023
F ADDITIONAL IMAGENET EXPERIMENTS
ConvWixer-51211. Patchh Size 7x7, Kernel Size 9x9 (ImageNet-1k)
Depth
Layer
U Unnormann aomaaaaae
Covariance matrices from a ConvMixer trained on Im- ageNet exhibit similar structure to those of ConvMixers trained on CIFAR-10; however, later layers tend to have more struc- ture, including "checkerboard"
Figure 32:
Normalzzeed Per Brr--Bikck
pattern in each block.
21

---

Published as a conference paper at ICLR 2023
F.1 10-EPOCH IMAGENET RESULTS
ConvMixer-512/12: Patch Size 14, Kernel Size 9
Thawed
Frozen
Uniform init
54.5
47.4
Stats from CM-512/12
55.5
53.4
Stats from CM-64/12
55.2
52.7
Filters transferred from CM-512/12
55.1
54.4
Our init (.15, .3, .5)
55.4
52.2
Our init (.15, .25)
55.5
52.4
Table 2: ConvMixer performance ImageNet-Ik Ik training with 10 epochs. Our initialization performs comparably to loading covariance matrices from previously-trained models (which were trained for 150 epochs).
ConvMiieer512212: Patch Size 7, Kernel Size 9
Thawed
Frozen
Uniform init
61.87
56.73
Stats from CM-512/12
62.56
60.79
Stats from CM-64/12
62.72
60.86
Filters transferred from CM-512/12
62.81
61.83
Our init (.15, .3, .5)
62.49
58.94
Our init (.15, 5, .25)
62.59
59.31
Table 3: ImageNet 10-epoch training
Con-Mixer-512224: Patch Size 14, Kernel Size 9
Thawed
Frozen
Uniform init
50,40
43.00
Stats from CM-512/12
53.03
51.45
Stats from CM-64/12
53.16
51,25
Filters transferred from CM-512/12
52.87
52.12
Our init (.15, .5)
53,80
51.16
Our init (.15, 5, .25)
53.76
50.81
Table ImageNet 10-epoch training
ConvNeXt-Atto
Thawed
Frozen
Uniform init
31.37
23.63
Stats from the same arch
33.44
40.41
Stats from 1/8"-width arch
29.81
31.47
Filters transferred from same arch
31.68
40.48
Our init (.15, .3, .5)
37.64
34.59
Our init (.15, .25)
31.34
34.23
Our init (.15, .25, 1.0)
38.01
33.98
Table 5: ImageNet 10-epoch training
22

---

Published as a conference paper at ICLR 2023
ConvNeXt-Tiny
Thawed
Frozen
Uniform init
32.51
25.94
Stats from the same arch
42.78
41.54
Stats from 1/8thwidth arch
44.60
42.86
Filters transferred from same arch
31.01
45.32
Our init (.15, .5)
35.64
35.04
Our init (.15, .25) Our init (.15, .25, 1.0)
40.17
38.91
40.78
36.62
Table ImageNet 10-epoch training
23

---

Published as a conference paper at ICLR 2023
F.2 50-EPOCH IMAGENET RESULTS
ConvNeett-Ato
Thawed
Frozen
Uniform init
69.96
51.43
Stats from the same arch
68.83
66.71
Stats from 1/8th-winee -widt arch
68.69
66.31
Filters transferred from same arch
68.01
67.29
Our init (.15, .3, .5)
63.55
63.48
Our init (.15, .5, .25)
67.84
64.52
Our init (.15, .25, 1.0)
68.06
63,43
Table 7: ImageNet 50-epoch training
ConvMixer-512/12: Patch Size 14, Kernel Size 9
Thawed
Frozen
Uniform init
67.03
60.47
Stats from the same arch
67.13
65.08
Stats from -/8th arch
66.75
64.94
Filters transferred from same arch
67.28
66.11
Our init (.15, .3, Our init (.15, .5, .25) Our init (.15, 22, 1.0)
66.12
64.39
67.41
64.43
67.34
64.12 12
Table ImageNet 50-epoch training
Con>Mixer-512224: Patch Size 14, Kernel Size 9
Thawed
Frozen
Uniform init
67.76
62.50
Stats from the same arch
68.92
67.91
Stats from 1/8th-width -width arch Filters transferred from same arch
68.78
67.36
69,42
68.66
Our init (.15, .5) Our init (.15, .25)
69.05
66.20
69.60
66.57
Our init (.15, 22, 1.0)
69.52
66.38
Table 9: ImageNet 50-epoch training
24

---

Published as a conference paper at ICLR 2023
G ADDITIONAL CIFAR-10 TABLES
25

---

Published as a conference paper at ICLR 2023
Table 10: CIFAR- 10 results for ConvMixer-25500 22668 with patch size 2. Bold denotes the highest per group, and blue bold denotes the second highest.
88. 94 + .25
.03
.23
.12
00 + .055
.66
1 .24
93 02 = .09
91.02 +
92 01 +
Our
90. 22
02 02
85
92. 11
89 94 = .08
91 02.62 = .17
90 00 = .27
37
92 28
.09
90.90 12
92 47 =
Direct
90 08.08
01
92
88
Cov. transser ( widhh
89.34 # .40
1.33 #: .27
92 92:27 + .21
00
92.00 + .18
89.51 1 .06
90.61 + 11
FROZEN
89.36 +
01.03 +
+
8T.52 +
transfer
+ 11
15
.36
13
.12
01 1..344 1 .08
+ .15
.08
.25
4:
+
89 56
91 42
88 26
70.72
88. 9T
93
92
89
88, 89 = .41
91.12 =: 22
= .2
86. 73 = 22
44 = 23
+4.07 = 00
25 .39
33
21
33
05
Uniform
92. 38
N0.70
90. 04
85, 05
87 04
88. 64 = 133
91 03 + .07
92 = 11
90.70 00.79 = 244
92 48 = 08
93. .9
92. 22.17 23
Our
92 .4.94
90 02 14
188
18
82 97 18
26
20
02 62 26
80 19
90 79 24
= 26
Direst
91 09
90 48
91 01
91 8T
92
91
Cox, transfor
89 76 + .14
+: .02
90 6:3 = -10
92 88 = .12
26
19
THAWED
[g widtth)
92 4
U1 88 +
=
27 =
01 83
92. 07.01:2
90.8
+ 399
.21
.26
89 .1 00
91.77 26
91. 155
.13
51
116
21
90 08
92. 01
92. 83
91 60
Cov.
88.
90.
89 69 = -20
92 #: .03
08 + .16
830 66 = -21
= .10
.45
= 23
Uniform
91 81 =
86
26
91.74
92. 05
86.0 06.04
89. 94
91. 71
91
#
Eps
20
50
200
20
50
200
200
20
50
200
FiLL
15
26

---

Published as a conference paper at ICLR 2023
Table 11: CIFAR- 10 results for Minv--xxe-25500 256/24 with patch size 2. Bold denotes the highest per group, and blue bold denotes the second highest.
88 05 .00
.23
.10
.22
92 + .27
94 .09
06
.24
9416 12
90 09
92 97.93
92.
90
90
88 .5.95 # .11
1 =: .14
+ .23
89 34 1 .40
91 11 + .0
92. 92.53 = .15
1 22.2
+ .2
1
Direct
76
89 76
11.1945
91
Cov. transles ( width)
12
.03
89. 19 +: 12
09
92 60 .09
25
33
90.67 18
FROZEN
88 80 +
94 4
+
92. 37
88 78
01.
933
Cov, transfer
88 63 20
02. 02.02 .32
11
08
UT
01 86 288
11
90. 19.1922 32
87
84
88
87. 87.71 + 32
91.58 +: .18
# .05
+ .17
.7
16
Uniform
93. 71 +
90. 88 +
+
1
88 18.1841
85 81
80 04
91.78
80. 41.42
80 72
85. 10
88.
88: 42 + .14
.24
22
.18
.08
94 .18
22
.18
33
94.18 14.16-0
90 28
92 30
93 66
92.
03
= 228
= -9
89 81 = -16
.12
93. 13.16 -11
27
.03
+
Direct
89 22
92 56
94 37
90. 22
92. 20
02
THAWED
Cov. truusser g width)
= 113
007
11
= 14
17
21
29
12
18
88. 07
06.07
90.10
92 22
92.
94.
89
02.
93
86
88. 79 07
.44
.12
.15
91. 73 26
91 .6.36 11
.1.1
48
15
= 23
93. 08
89. .8.68
02
Cov.
92.
94.
89
92.
= 11
=: 15
88 49 # 46
+ .77
11
10
Uniform
94. 1: +
+
=
38.37
92.38
91 90
03.57
$7.90
06
95
74
03
88
92.3
03.
54
80.
92
#
Epp
20
50
200
20
50
200
29
50
200
20
50
200
Filt. Size
15
27

---

Published as a conference paper at ICLR 2023
Table 12: CIFAR- 10 results for ConvMixxr-25500 -26x8-226668888 with patch size 1. Bold denotes the highest per group, and blue bold denotes the second highest.
89.07 + .21
91. 04.47 4: .14
91.08 4: .15
93. 56 1 .14
58 + 12
1 .07
93 47 05
90.73 4:
02. 92.31 4.
45 +
Our
36
92. 73
+ .17
2222
.36
1: .03
02
92 116
98 .01
92 25.35 .13
98 08.01 27
90 64
91 92.92
9
01
03
Cov. transfer ( wicht)
90 = .03
=: .10
.38
.1.1
07
+ .17
.23
= 17
FROZEN
=
91 .6.6
92. 05
91 13
08
25
52
93. 08.08
90 70.70
91.7 91.72
92 02.79
90 37 + 008
27
06
02 .04
32
90 90.65 15
14
91. #:
92. 12.1222 +
91 83
Cov, Uniform
90.
02
89.33 + .09
90.97 +: .19
-2
89.84 = .04
91.31 .2
69.45 .02
91.96 .07
86.04 + .1
.17
= 22
91.71
20.20
89.
90.2
89. 84 = 27
91 - -10
91. = .35
80 = 23
94. 14.124 + .24
93 02.42.1 1 .07
94 11.13-5 13
=
+
02 +
Our
03
93
118
.2
.18
.07
07
17
03
28
10
+ 02
Direct
90 66
92 05.05
82 5-4
92 37
93 83
91 70
92 95
93 59
03
THAWED
Con. trunsfer 1g widthh
+ .2
+ .18
92. 58 + 09
92 02 + .05
1 13
03
+
90.78 78.786
92 09
22
08
92. 99
93 37
91
Cov: transfer
90 90:60 + .31
+ .23
4 .08
93.60 .16
92.34 1 11
+ 16
+
92.00
9/ 38
91. 94
93 83.00
91.74
91.00
93 26
92.
= .11
.06
.19
316
.15
23
36
23
00
16
Uniform
90.41 11
1..70
74
76
55
01
02.
91.
02.
93
Eps
20
50
200
20
50
280
200
20
50
200
FilL Size
15
28

---

Published as a conference paper at ICLR 2023
with patch size 1. Bold denotes the highest per group, and blue bold denotes the second highest.
Table 14: CIFAR-IC 10 results for ConvNeXt-atter with patch size 2. Bold denotes the highest per group. and blue bold denotes the second highest.
79.36 .07
85 46 1: 220
87.68 87.88990 1: .28
.14
.2
Our init
84 71
90.13
0/ 27
90
83 83.61 + .23
86 98.93 #: 20
Direct transfer
+ .57
8.9 02.32-2 1: .2.
:: .21
86. 39 #:
Direct
83
90
Cov. transfer ( widthh
79. 03.03 .07
84. 03.83.55 8: .0
1: .02
FRozeN
Cox, transfer C with))
86. 02.02 4: .37
= .24
FROZEN
84. 00 -
86 60
89 08
Cov. transfer
87.81 10
.11
Cov. trinsfer
80. 82 = 11
85. #: .45
4: .10
80 70
86 25
63.
80. + 888
.39
.08
72.18 + 300
77.02 +: .34
Unitorm
Uniform
85.18 1
87.10 +
79. 82 +
.78
.18
.35
81.13 + .54
86 06.07 .24
.2
Our init
03 +
Init
90. 54
20
80
92
Direct transfer
4: .75
= 40
83, 42 = 44
= 20
87 02 = .10
86. 18 =
89. 07
91 05
Direct
86, 21:28:2
THAWED
Cos; transfor ( widhh)
.25
87 .2 1. .27
+ .11
THAWED
Cov. transferr width)
81 22 =: .78
+ 116
18
Table 13: CIFAR 10 results for ConvNeXt-an attoo atto
60
86. 02
88. 14
84.
90
Cov. transfer
100
46
# 32
+ .21
87 40 + .27
77
82. 36
86:21
85
88,
90.1
Cov.
47 + .68
Uniform
72 96 = .70
83.71 = .00
8 46 +
$7.55 1
Uniform
066.28-22 =
00
86.
Fips
20
SD
200
Eps
20
50
2010
Filt. Sine
Filt. Size
7
29


Reviews:
[{"review_id": "tNOOnKa1DA", "reviewer": "Reviewer_tjLw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an initialization scheme for convolutional filters. The main idea is to first build a filter covariance matrix from pre-trained networks and then sample initial filters based on this covariance. The paper also provides a learning-free closed-form alternative for filter initialization. The experiments specifically focus on ConvMixer and ConvNext networks, and show that using the proposed initialization scheme outperforms uniform initialization. They also show that in some cases, networks with fixed frozen filters can achieve competitive performance.", "summary_of_the_review": "The main idea of the paper is interesting and useful to the community. \n\nQuestions from the authors:\n- It would be helpful to see the results of initializing with covariances from smaller models when two or more variables change simultaneously, for instance, from a model both narrower and shallower, to see if similar results will be obtained with more efficient pre-training.\n- Would it be useful to use the proposed method for initializing networks with normal convolutions like ResNets?", "strength_and_weaknesses": "Strengths:\n- Building a closed-form initialization scheme based on the observations from pre-trained networks is notable.\n- The authors evaluate their method in variaous variants and the visual presentation of the results helps the reader to better see the differences across the variants. \n\nWeaknesses:\n- The closed-form covariance matrix computation is not completely learning-free, as the authors perform a grid search to find the optimal parameters.\n- It is not clear why the authors use a quadratic schedule for the variance.\n- The related works section can be improved by adding other recent initialization methods and comparing the results to them.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "RBRKzHr-xe", "reviewer": "Reviewer_JJAa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper the authors aim to understand the structure of convolution filters, in order to initialize them better. First, the authors estimate the covariance matrix of the filters based on pre-trained models, and sample new filters from a Gaussian distribution with this covariance. Then, based on a visual inspection of the filters in trained networks, the authors propose a simple initialization method. The method seems to work rather well in practice, even when the filters are not trained.", "summary_of_the_review": "Overall, the paper proposes an interesting method with nice experiments, but I think more work should be done to provide clearer explanations and intuition, with more architectures and comparisons to prior work, before it can be published.\n\n-- POST AUTHOR FEEDBACK:\n\nThank you for the detailed response and the new results. I am happy to increase my score.", "strength_and_weaknesses": "***Strengths***:\n\nThe initialization method proposed in the paper is interesting and the empirical results are encouraging. The method is very simple to implement.\n\n\n***Weaknesses***:\n\n\n**Limited scope:** The experiments in the paper are limited to mainly one architecture called ConvMixer (and also the ConvNeXt architecture). Also, only CIFAR-10 and ImageNet datasets are used. I think it will be interesting to see if the proposed initialization method can work for other architectures and various datasets.\n\n**Comparison to other methods:** The authors mainly compare to the standard uniform initialization (default in PyTorch). I think it will be interesting to compare to other methods (discussed in the related work section). Another relevant work: \n\nLi et. al. FILTER SHAPING FOR CONVOLUTIONAL NEURAL NETWORKS, ICLR 2017\n\nAlthough other methods are probably more complex, it will be interesting to see the test accuracy that they can achieve, and what is the time complexity of different methods.\n\n**Visual inspection:** I think the authors should better explain what is the structure that we see in Figures 1 and 21. The filters in these figures look rather different, especially in the last layer, although the authors claim that “Covariance matrices from a ConvMixer trained on ImageNet exhibit similar structure to those of ConvMixers trained on CIFAR-10”. For example, the checkerboard structure in figure 21 doesn’t appear in figure 1. \nI think the authors should better explain the intuition behind the proposed covariance $\\hat{\\Sigma}=M \\odot (C-0.5S)$. How is the checkerboard structure captured here ?\nThe authors say that “the sub-blocks of the covariances often have a static negative component in the center, with a dynamic positive component whose position mirrors that of the block itself.” Is it possible that the “dynamic positive component” is the variance, i.e. $[\\Sigma_{i,j}]_{\\ell,m}$ for $i=\\ell,j=m$, and thus it is large ?\n\n**Sensitivity to hyperparameters of the method:** Although the method is quite robust to the hyperparameters of the method (eq. (11)) for CIFAR-10, it is rather sensitive for ImageNet, as we can see in figures 18,19 (and tables 5,6).\n\n***More comments and questions:***\n\n- Can you try to explain or give any intuition why for narrower models 32 filters work better than 256 ? or for shallower models 8 layers better than 32 ?\n- The authors don’t tune the hyperparameters of the method for each model and claim that “hyperparameter tuning is optional”. I think it will be interesting to see what can be squeezed from this method by optimal tuning for each model. \n- What do you mean by “each data point runs through a full cycle of the LR schedule” in figures 8,11 ?\n- In eq. (9): 1/2 is missing\n- Will be good to explain how exactly linear interpolation is performed for the shallower models.\n- For the shallower models “2- and 4-deep models are also quite effective” – but no results for 2 layers in figure 4.\n- In figure 4, for narrower models, there is no comparison to uniform, and no 200 epochs.\n- Will be good to explain the “triangular learning rate schedule”.\n- I think it will be good to give a quick description of ConvMixer and ConvNeXt architectures.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KQ1MH5gKYeL", "reviewer": "Reviewer_tQX5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the structure of depthwise convolutional filters in pre-trained models. Empirically, it shows that computing the covariance of these filters and using it for initialization can bring benefits in terms of trainability and final performance, and such statistics can be transferred from a smaller to a larger model. A new initialization is proposed which further proposes benefits and doesn't require pre-training a network.", "summary_of_the_review": "The idea is interesting and the visualizations help motivate the approach. The proposed initialization seems a bit too ad hoc and based on heuristics, and, while it yields faster training, the final performance under standard training budgets is similar to the commonly-adopted uniform initialization (bottom rows of Table 1).", "strength_and_weaknesses": "The paper is clear and the motivation / intuition for studying the structure of pre-trained filters is presented well.\n\nEmpirical observations are somewhat surprising, showing that transferring from a smaller to a larger network provides more benefits than using covariances computed from the original model.\n\nThe main visualizations and comparative results are, unfortunately, based on CIFAR-10 and not on ImageNet. Although some ImageNet results are shown, it would be better if Fig 1, 2 and 3 were based on ImageNet. Some of the main claims from Fig 2, e.g. that transferring from smaller to larger models provides benefits, is also based on what seem to be small improvements (comparing groups C and B).\n\nThe proposed initialization seems to be very strongly based on the visualizations of CIFAR-10 trained networks. Although it shows benefits, some comparison against other initialization schemes would be valuable (dirac, orthogonal, zero-init for convs with skip connections, etc).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DlMzhgepEPx", "reviewer": "Reviewer_Yta9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the covariance structures of each convolution filter in deep learning models for classification problems. It builds a parametric model for these structures and proposes to directly sample a multivariate Gaussian distribution as an initialization for a family of CNNs. Numerical results show that in some cases, such initialization may achieve very high classification, without the need of learning. ", "summary_of_the_review": "REVISED: I have raised my score to 6 as I still find the significance of the paper could be improved by providing further discussion on the impact of the initialization of other layers beyond depthwise convolutional layers . This is not so clear by reading the paper. Nevertheless, the authors proposed an interesting method to model the covariance structure of these layers, which gives an explanation of the performance of CNN. This is novel and could be published. ", "strength_and_weaknesses": "This paper builds upon an interesting observation of the covariance structures of each convolution filter at multiple layers of CNNs. It proposes a parametric model for such covariance structures such that certain parameters may be fitted from trained models. \n\nCertain results of this paper could still be improved to support the point of the learning-free multivariate initialization scheme. It is still not very clear on the ImageNet case, if the learning-free approach could achieve a similar performance compared to the learning approach. In Table 1 Frozen vs. Thawed, we see that the results with the Frozen case are worse than those of the Thawed case. It is thus not clear why learning is not needed. One suggestion would be to fit the parametric model of covariances to some small ImageNet models, and then to see if they work well (as CIFAR is quite different to ImageNet, using the parameters from CIFAR might have some unpredictable risk). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster