Initializing Models with Larger Ones
Authors: Zhiqiu Xu
Yanjie Chen
Kirill Vishniakov
Yida Yin
Zhiqiang Shen
Trevor Darrell
Lingjie Liu
Zhuang Liu
Venue: ICLR 2024 spotlight (2024)
Keywords: Deep Learning
Neural Networks
Weight Initialization
Small Models
Computer Vision
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Selecting weights from a pretrained large model to initialize a small model improves accuracy and reduces training time

Abstract: Weight initialization plays an important role in neural network training. Widely used initialization methods are proposed and evaluated for networks that are trained from scratch. However, the growing number of pretrained models now offers new opportunities for tackling this classical problem of weight initialization. In this work, we introduce weight selection, a method for initializing smaller models by selecting a subset of weights from a pretrained larger model. This enables the transfer of knowledge from pretrained weights to smaller models. Our experiments demonstrate that weight selection can significantly enhance the performance of small models and reduce their training time.  Notably, it can also be used together with knowledge distillation. Weight selection offers a new approach to leverage the power of pretrained models in resource-constrained settings, and we hope it can be a useful tool for training small models in the large-model era.

Published as a conference paper at ICLR 2024
INITIALIZING MODELS WITH LARGER ONES
Zhiqiu Xu', Yanjie Chen', Kirill Vishniakov', Yida Zhiqiang Shen',
Trevor Darrell', Lingjie Liu', Zhuang Liu" University of Pennsylvania PUC Berkeley *MBZUAI
4 MMeta Al Research
ABSTRACT
Weight initialization plays an important role in neural network training. Widely used initialization methods are proposed and evaluated for networks that are trained from scratch. However, the growing number of pretrained models now offers new opportunities for tackling this classical problem of weight initialization. In this work, we introduce weight selection, method for initializing smaller models by selecting a subset of weights from a pretrained larger model. This enables the transfer of knowledge from pretrained weights smaller models. Our experiments demonstrate that weight selection can significantly enhance the performance of small models and reduce their training time. Notably, it can also be used together with knowledge distillation. Weight selection offers a new approach to leverage the power of pretrained models in pnouurc--cossrringg settings, and we hope it can be a useful tool for training small models in the magee-moeel era. Code is available at https: / g thub.
wwwpaa/ww///ww/itt/et/e//// weight selection.
INTRODUCTION
The initialization of neural network weights is crucial for their optimization. Proper initialization aids in model convergence and prevents issues like gradient vanishing. Two prominent initialization techniques, Xavier initialization (Glorot & Bengio, 2010) and Kaiming initialization (He et al., 2015), have played substantial roles in neural network training. They remain the default methods in modern deep learning libraries like PyTorch (Paszke et al., 2019).
These methods were developed for training neu- ral networks from random initialization. At that time, was the common practice. However, the landscape has changed. A variety of pre- trained models are now readily available, thanks to collective efforts from the community (Wolf et al., 2019; Wightman, 2019). These mod- els are trained on large datasets like ImageNet- 21K (Deng et al., 2009) and LAION-5B (Schuh- mann et al., 2022) and are often optimized by experts. As a result, fine-tuning from these pretrained models (Kolesnikov al., 2020; Hu et al., 2022) is usually considered a preferred option today, rather than training models from scratch.
smaller classify flowers.
model
Inttiaiize
train
MTB
Figure 1: Large pretrained models offer new op- portunities for initializing small models.
However, these pretrained large models can be prohibitive in their resource demand, preventing their wide adoption for resource-constrainee settings, e.g., on mobile devices. For many pretrained model families, even the smallest model instance can be considered extremely large in certain contexts. For example, masked autoencoders (MAE) (He et al., 2022) and CLIP (Radford et al., 2021) both provide ViT-Base (Dosovitskiy et al., 2021), a 80M-parameter architecture, as their smallest pretrained Transformer model. This is already too large for applications on edge devices, and the smallest LLaMA (Touvron et al., 2023) model is even another 100 times larger, with 7B parameters. With few small pretrained models available, developers would have to train them from scratch target datasets to suit their needs. This approach misses the opportunity to utilize large pretrained models, whose knowledge is learned from extensive training on large data.

---

Published as a conference paper at ICLR 2024
In this work, we tackle this issue by introducing weight initialization method that uses large pretrained models to train small models. Specifically, we introduce weight selection, where a subset of weights from a pretrained large model is selected to initialize smaller model. This allows for knowledge learned by the large model to transfer to the small model through its weights. Thanks to the modular design of modern neural networks, weight selection involves only three simple steps: layer selection, component mapping, and element selection. This method can be applied to any smaller model within the same model faamlly as the large model. Using weight selection for initializing a small model is straightforward and adds no extra computational cost compared to training from scratch. It could also be useful even for large model training, e.g., initializing LLaMA-7B with trained weights from LLaMA-30B.
We apply weight selection to train small models on image classification datasets of different scales. We observe significant improvement in accuracy across datasets and models compared with baselines. Weight selection also substantially reduces the training time required to reach the same level of accuracy, Additionally, it can work alongside another popular method for knowledge transfer from large models - knowledge distillation (Hinton et al., 2015). We believe weight selection can be a general technique for training small models. Our work also encourages further research on utilizing pretrained models for efficient deployment.
RELATED WORK
Weight initialization. Weight initialization is crucial aspect of model training. Glorot & Bengio (2010) maintain constant variance by setting the initial values of the weights using a normal distri- bution, aiming to prevent gradient vanishing explosion. He et al. (2015) modify it to adapt to ReLU activations (Nair& & Hinton, 2010). Mishkin & Matas (2016) craft the orthogonality in weight matrices to keep gradient from vanishing exploding. Saxe et al. (2014) and Vorontsov al. (2017) put soft constraints on weight matrices to ensure orthogonality.
There are methods that use external sources of knowledge like data distribution or unsupervised training to initialize neural networks, A data-dependent initialization can be obtained from performing K-means clustering or PCA (Krähenbühl al., 2015; Tang et al., 2017) on the training dataset. Larochelle et al. (2009), Masci et al. 1). Trinh et al. (2019), and Gani et al. (2022) show training on unsupervised objectives can provide better initialization for supervised training.
Utilizing pretrained models. Transfer learning (Zhuang et al., 2020) is a common framework for using model weights pretrained from large-scale data. Model architecture is maintained and the model is fine-tuned on specific downstream tasks (Kolesnikov et al., 2020). Knowledge distillation involves training a usually smaller student model to approximate the output of a teacher model (Hinton et al., 2015; Tian et al., 2019; Beyer et al., 2022). This allows the student model to maintain the performance of teacher while being computationally efficient. Another alternative approach for using pretrained models is through weight pruning (LeCun et al., 1990; Han et al., 2015; Li et al., 2017b; Liu et al., 2019). It involves removing less significant weights from the model, making it more efficient without significantly compromising performance.
Lin et al. (2020) transform parameters of a large network to an analogous smaller one through learnalle linear layers using knowledge distillation to match block outputs. Sanh et al. (2019) and Shleifer & Rush (2020) create smaller models by initializing with a subset of layers from a pretrained BERT (Devlin et al., 2018). This method requires the smaller model to have the same width as teacher's. s. Czyzewski et al. (2022) borrow weights from existing convolutional networks to initialize novel convolutional neural networks. Chen et al. (2016) and Chen al. (2021) transform weights from smaller networks as an effective initialization for larger models to accelerate convergence. Trockman et al. (2023) initialize convolutional layers with Gaussian distribution according to pretrained model's covariance. Similarly, Trockman & Kolter (2023) initialize self-attention layers according to observed diagonal patterns from pretrained ViTs. These two methods use statistics from but do not directly utilize pretrained parameters. The concurrent work, Xia et al. (2023), applies structured pruning to reduce existing large language models to their smaller versions and uses the pruned result as an initialization. Zhong et al. (2024) explores knowledge transfer between transformer-baste through parameters. Weight selection, in contrast, directly utilizes pretrained parameters, does not require extra computation, and is suitable for initializing any smaller variants of the pretrained model.
LLM
2

---

Published as a conference paper at ICLR 2024
3 WEIGHT SELECTION
Given pretrained model, our goal is obtain an effective weight initialization for a smaller-size model within the same model family. Borrowing terminology from knowledge distillation, we refer to the pretrained model teacher and the model we aim to initialize as student.
3.1 APPROACH
Modern neural network architectures often follow a modular approach: design a layer and replicate it build the model (He al., 2016; Vaswani al., 2017; Tolstikhin et al., 2021; Dosovitskiy et al., 2021; Liu et al., 2022). This design ethos promotes scalability: models can be widened by increasing the embedding dimension the number of channels each block, and deepened by stacking more layers. It also enables us to perform weight selection following three steps: layer selection, component mapping, and element selection.
uniform element selection
pretrained model (teacher)
layer selection
student initialization
Figure 2: Weight selection. initialize smaller variant of pretrained model, we uniformly select parameters from the corresponding component of the pretrained model.
Layer selection. Selecting layers from teacher is the first step. For each layer in student, layer from teacher selected as source for initialization. The procedure for layer selection is slightly different for isotropic architectures and hierarchical architectures. An isotropic architecture refers to the neural network where each layer exhibits a consistent and uniform layerwise design throughout the model. VIT (Dosovitskiy al., 2021) and MLP-mixer (Tolstikhin et al., 2021) belong isotropic architectures. hierarchical architecture characterized by multi-scale representations and hierarchy in embedding dimensions. Hierarchical architectures typically have stages with varying scales and embedding dimensions. For example, classic convolutional networks like VGG (Simonyan & Zisserman, 2014) progressively decrease spatial dimensions while increasing channel dimensions, capturing multi-scale features. Modern architectures like Swin- Transformer (Liu et al., 2021) I) and ConvNeXt (Liu al., 2022) also employ this hierarchical design.
For isotropic architectures, we select the first N layers from the teacher, where N represents the student's 's layer count, denoted as first-N selection. When dealing with hierarchical structures like ConvNeXt (Liu et al., 2022), first-N selection is applied at each individual stage. An alternative method uniform layer selection, where evenly-spaced layers in teacher are selected. Empirical results in Section 4.3 and Appendix B show that first-N layer selection is ideal for weight selection.
Component mapping. the second step, we map components between student and teacher. From the previous step, obtained layer mapping from teacher to student. The task is then reduced to initializing one student layer with one teacher layer. Thanks to the modular approach adopted by modern neural network design, layers in models of the same family have an identical set of components which only differ in their width. The process for matching the corresponding components thus natural one-to-one mapping.
Element selection. Upon establishing component mapping, the next step is to initialize student's component using its larger counterpart from teacher. The default method for element selection is

---

Published as a conference paper at ICLR 2024
uniform selection, where evenly-spen rvenyy-spaced elements are selected from teacher's 's tensor as shown in figure Details on uniform selection and other element selection methods will be introduced next part.
3.2 METHODS FOR ELEMENT SELECTION
In this part, we formulate element selection and introduce different selection criteria. Consider weight tensor W, from student that we seek to initialize with teacher's 's weight tensor W.. If Wa has shape t1, t.., En. then W., which is of the same component type with Wt, will also span ..., dimensions. Our goal is to select a subset W elements to initialize W.. Several possible methods for element selection are discussed as follows. We compare the performance of these element selection methods in Section 4.3. We find that as long as consistency is maintained (as explained in the paragraph Random selection with consistency, weight selection can achieve similar level of performance. We propose using uniform selection as default for weight selection in practice.
Uniform selection (default). For each dimension of W,, select evenly-spaced 8; slices of 4-- For example, to initialize a linear layer W. of shape 2 x 3 with a linear layer We of shape 4 x 6, we select 1st and 3rd slice along the first dimension, and 1st, 3rd, and 5th slice along the second dimension. We present pseudocode for uniform selection in Algorithm 1. The algorithm starts with a copy of teacher's weight tensor W, and iteratively performs selection on all dimensions of W, to reach the desired shape for student. Notably, in architectures that incorporate grouped components such as the multi-head attention module ViTs and the grouped convolution in ResNeXt (Xie et al., 2017) uniform selection absorbs information from all groups. For example, when applied to ViTs, uniform selection will select parameters from all heads in the attention block, which is likely to be beneficial for inheriting knowledge from the pretrained ViTs.
Algorithm Uniform element selection from teacher's 's weight tensor
Input: W
> teacher's weight tensor
Input:
desired dimension for student's weight tensor
Output: W, with shape s
I: procedure UNIOORMEEEMENTSELLCTTOVVW
2:
W, + Copy of W,
b student's weight tensor
3: n < length of Wa.shape
4:
for i = - do
5:
de +- [h.hhape[i]
6:
d, < s[i]
7: indices <- Select d, evenly-spaced numbers from to de
8: W. Select indices along W,'s it dimenssio
9: end for
10: return W.
11: end procedure
Consecutive selection. For each dimension of W,, select consecutive 8; slices out of 4, In contrast to uniform selection, for architectures with grouped components, consecutive selection selects some entire groups while omitting the contrast. For architectures without such grouped components, consecutive selection is equivalent to uniform selection.
Random selection (with consistency). For all weight tensors, and for each dimension of Wt, select the same sundomly-generatee
set of s; slices out of t. Through empirical experiments in Section 4.3, we find that consistency (selecting the same indices for all weight matrices) is key for weight selection to reach its best performance. Motivation for maintaining consistency stems from the existence of residual connections - neurons that are added in the teacher model should have their operations preserved in the student. Furthermore, maintaining consistency preserves complete neurons during element selection, since only consistent positions are selected. worth noting that uniform selection and consecutive selection inherently preserve consistency, which are both special instances of random selection with consistency.
Random selection (without consistency). Along every dimension of W., randomly select s; slices out of t,. Unlike random selection w/ consistency, this method does not require selecting the same indices for every weight tensor. We design this method to examine the importance of consistency.
4

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
4.1 SETTINGS
Datasets. We evaluate weight selection on 9 image classification datasets including ImageNet- IK (Deng et al., 2009), CIFAR-I0, CIFAR-I00 (Krizhevsky, 2009). Flowers (Nilsback & Zisserman, 2008), Pets (Parkhi et al., 2012), STL-10 (Coates et al., 201 I), Food-101 (Bossard et al., 2014)), DTD (Cimpoi et al., 2014), SVHN (Netzer et al., 2011) I) and EuroSAT (Helber et al., 2019; 2018). These datasets vary in scales ranging from 5K to 1.3M training images.
Models. We perform experiments on ViT-T/16 (Touvron et al., 2021a) and ConvNeXt-F (Liu et al., 2022), with ImageNet-21K pretrained ViT-S/16 and ConvNeXt-T as their teachers respectively, We obtain weights for ImageNet-2IK pretrained ViT-S/16 from Steiner et al. (2021) and ImageNet-21K pretrained ConvNeXt-T from Liu al. (2022). We present the detailed configurations in Table 1.
configuration
student
teacher
model
ViT-T
ConvNeXi-F
ViT-S
ConvNeXI-T
depth
12
2/2/6/2
12
3/3/9/3
embedding dimension
192
96 192 384 768
384
48 18/96/ 192 384
number of heads
6
number of parameters
5M
5M
22M
28M
Table 1:
Model Configurations. We perform main experiments on ConvNeXt and ViT, and use student that halve the embedding dimensions of their corresponding teacher.
Training. We follow the training recipe from ConvNeXt (Liu et al., 2022) with adjustments to batch size, learning rate, and stochastic depth rate (Huang et al., 2016) for different datasets. See Appendix A for details. To ensure fair comparison, we adapt hyperparameters to baseline (training with random initialization), and the same set of hyperparameters used for training with weight selection.
Random initialization baseline. We utilize the model-specific default initialization from timm library (Wightman, 2019), popular computer vision library with reliable reproducibility. Its default initialization of ViT-T and ConvNeXt-F employs a truncated normal distribution with standard deviation of 0.02 for linear and convolution layers. The truncated normal distribution, designed to clip initialization values, is adopted to develop modern neural networks (Liu et al., 2022).
4.2 RESULTS
Experiment results are presented in Table 2. Across all image classification datasets, weight selection consistently boosts test accuracy, especially for smaller datasets. Notably, weight selection addresses the well-known challenge of training ViT on small datasets, which likely contributes to the significant accuracy improvement for VIT. Training curves for ImageNet- 1K are shown in Figure 3.
Both models benefit from weight selection early on and maintain this advantage throughout training.
dataset (scale 4 random init weight selection
change
random init weight selection change
ImageNet-IK
73.9
75.6 ? 1.6
76.1
76.4
0.3
SVHN
94.9
96.5
? 1.6
95.7
96.9
1.2
Food-101
79.6
86.9
7.3
86.9
89.0
2.1
EuroSAT
97.5
98.6
1.1
98.4
98.8
1.4
CIFAR-10
92.4
97.0
14.6
96.6
97.4
10.8
CIFAR-100
72.3
81.4
79.1
81.4
84.4
13.0
STL-10
61.5
83.4
12..9
81.4
92.3
10.9
Flowers
62.4
81.9
19..5
80.3
94.5
1 14.2
Pets
25.0
68.6
143.6
72.9
87.3
1 14.4
DTD
49.4
62.5
113.1
63.7
68.8
5.11
VIT-T
ConvNeXt-F
Table 2: Test accuracy on image classification datasets. On all datasets, employing weight selection for initialization leads to improvement in test accuracy. Datasets ordered by their image counts. Weight selection provides more benefits when evaluated on datasets with fewer images.

---

Published as a conference paper at ICLR 2024
80
60
60
aceaney (30)
accatri (%)
minn loss
Joss
40-
random init test acc
random test
weight selection test acc
weight solection tost acc random init train loss
20
random init train Joss
20
weight selection train loss
weight selection triin loss
00
50
100
150
200
250
300
0
50
100
150
200
250
300
(a) VIT-T
(b) ConvNeXt-F
Figure Training curves on ImageNet-IK. When initialized using weight selection from ImageNet- 21K pretrained models, both VIT-T (from ViT-S) and ConvNeXi-F (from ConvNeXI-T) exhibit superior performance compared their randomly-inittalzzed counterparts.
4.3 COMPARISONS
Advantage over classic initialization. We compare weight selection and its variants with two widely-adopted initialization methods: Xavier initialization (Glorot & Bengio, 2010) and Kaiming initialization (He et al., 2015) and present results on CIFAR-100 as shown in Table 3. All variants of weight selection yield considerably better results than classic initialization methods.
Comparison of selection methods. Consistency the key for weight selection to reach its best performance. For both model architectures, uniform selection, consecutive selection, and random selection with consistency achieve a similar level of performance. Note that uniform selection and consecutive selection inherently maintain consistency. When removing consistency, a sharp drop in performance is observed. For random selection experiments, we report the median performance of three different selection results.
init timm default (trunc normal)
ViT-T
ConyNeXt-F
72.3
81.4
Xavier (Glorot Bengio, 2010)
72.1
82.8
Kaiming (He et al., 2015)
73.0
82.5
weight selection (uniform)
81.4
84.4
weight selection (consecutive)
81.6
84.0
weight selection (random w/ consistency)
81.7
83.9
weight selection (random w/o consistency)
77.4
82.8
Table 3: Comparison with classic initialization methods. We present CIFAR- test accuracy for weight selection's 's variants and classic initialization methods. Weight selection methods with consistency outperform classic initialization methods by large margin.
4.4 COMPATIBILITY WITH KNOWLEDGE DISTILLATION
Weight selection transfers knowledge from pretrained models via parameters. Another popular approach for knowledge transferring is knowledge distillation (Hinton et al., 2015), which utilizes outputs from pretrained models. Here explore the compatibility of these two techniques.
Settings. We evaluate the performance of combining weight selection with two different approaches in knowledge distillation logit-based distillation and feature-based distillation by using weight selection as initialization for student in knowledge distillation. Logit-based distillation uses KI.- divergence the loss function for matching student's is and teacher's logits. Denote student's output probabilities as P., and teacher's s output probabilities as Pt, the loss for logit-based distillation can be formulated
L = Lelase + a KL(pillps)
(1)
where Latass is supervised loss, and a the coefficient for distillation loss. Note that matching logits requires teacher to be trained on the same dataset student. For logit-based distillation, We train ViT-T on ImageNet- K while using the ImageNet- 1K pretrained ViT-S model from DeiT (Touvron al., a)21a) the teacher for both knowledge distillation and weight selection. is set 1.

---

Published as a conference paper at ICLR 2024
reature-based distillation steps in when classification head of the target dataset is not available. Denote teacher's 's output as O₁, and student's as Oz. Feature-base distillation can be formulated as Lelass + a (1 MLL(O.))
An MLP is used to project student's output to teacher's embedding dimension, and L1 loss used to match the projected student's output and teacher's ' output. For feature-based distillation, we perform CIFAR-100 training experiments on ViT-T, using ImageNet-21K pretrained ViT-S as the teacher for both knowledge distillation and weight selection. We tune a on distillation trials and use the same value for for trials that combine distillation and weight selection.
Results. Table provides results for knowledge distillation and weight selection when applied individually or together. These results show the compatibility between weight selection and different types of knowledge distillation. Without incurring additional inference costs, employing weight selec- tion alone produces a better result than vanilla logit-based distillation and feature-based distillation. More importantly, the combination of weight selection and knowledge distillation delivers the best results, boosting accuracies to 76.0% on ImageNet- -eeetttKK K and 83.9% on CIFAR- 1000 These results further confirm weight selection's 's usefulness between weight selection and knowledge distillation.
independent technique and the compatibility
setting
ImageNet-IK dlgittbassd distillation)
CIFAR-IO0 100 (feature fteature-based distillation)
test ace
change
test acc
change
baseline
73.9
72.3
distill
74.8
0.9
78.4
6.4
weight selection
75.5
1.6
81.4
9.1
distill weight selection
76.0
2
83.9
11.6
Table 4: Compatibility with knowledge distillation. Weight selection is useful as an independent technique, and can be combined with knowledge distillation to achieve the best performance.
ANALYSIS
In this section, we perform a comprehensive analysis of weight selection. Unless otherwise specified, the standard setting (for comparison) of weight selection is initializing ViT-T with uniform selection from the ImageNet-21K pretrained VIT-S, trained and evaluated on CIFAR-100.
Reduction in training time. We find weight selection can significantly reduce training time. We directly measure the reduction training time by training ViT-T with weight selection for different numbers of epochs and present the results in Figure 4a. The number of warmup epochs is modified to maintain its ratio with the total epochs for each trial. With weight selection, the same performance on CIFAR-100 100 can be obtained with only 1/3 epochs compared to training from random initialization.
90
90
80.3
81.4
X
80.0
81.5
82.5
80
71.2
80
76.3
77.2
73.0
72.3 x
X
70
70
train with weight selection train from random init
pretraining finetuning
62.7
X
X
weight selection
60
60
100
150
250
300
300
training opochs
pretraining opochs
(a) Comparison with random initialization
(b) Comparison with pretraining finetuning
Figure 4: Faster training. Compared to random initialization, ViT-T can reach the same performance on CIFAR-100 with only 1/3 epochs compared to training from random initialization. When compared to pretraining (on ImageNet-lK) + finetuning, weight selection is able to match the accuracy at 60 epochs of pretraining, saving 6. 12x training time.
Comparison with transfer learning. We conduct experiments to find the training budget needed for pretraining to match the accuracy of weight selection. For this experiment, train ViT-T
7

---

Published as a conference paper at ICLR 2024
ImageNet-1K with different numbers of epochs and then fine-tune CIFAR-100 for 300 epochs. As shown in Figure 4b, it takes 60 epochs of pretraining on ImageNet- IK to achieve the same performance CIFAR-100. Under this setting, weight selection is 6. 12 faster compared reaching the same performance with pretraining, without need to access the dataset used for pretraining.
Pretrained models. We study the effect of using different pretrained models weight selection teacher. Models with supervised pretraining turn out to be the best teacher. We evaluate the performance of ViT-B teacher under different pretraining regimes: CLIP (Radford al., 2021), MAE (He et al., 2022), and DINO (Caron et al., 2021). Table 5 presents the results. Initializing with pretrained weights consistently outperforms random initialization. ImageNet-214 pretrained teacher provides the most effective initialization. Note that for this experiment, we use ViT-B as teacher for weight selection, since is the smallest model that MAF and CLIP provide.
Pretrained models
CIFAR-10 CIFAR-I00
STL-10 10
supervised (ImageNet-2IK)
95.1
77.6
73.1
CLIP (Radforrd et al., 2021)
94.9
77.3
66.0
MAE (He et al., 2022)
95.9
77.2
71.0
DINO (Caron et al., 2021)
95.0
75.7
69.4
Table 5: Different prefrained models. Supervised pretraining makes the best teacher.
Layer selection. Shleifer Rush (2020) select evenly-spaced layers from BERT to initialize small models. We compare two layer selection methods: first-N layer selection and uniform layer selection. To evaluate different layer selection methods, we create ViT-A of 6 layers (half of the ViT-T depth), with other configurations identical to VTT-T. In this experiment, we use VIT-A and ConvNeXt-F as student, and ImageNet-21 pretrained ViT-S and ConyNeXI-T as their weight selection teacher. From the results presented in Table 6, we find first-N layer selection performs consistently better than uniform layer selection. Presumably, since layers initialized by first-N selection are naturally contiguous and closer to input processing, they offer more effective initialization for smaller models.
setting random init
ViT-A
ConvNeXt-F
69.6
81.3
first-N layer selection 77.6
84.4
uniform layer selection
76.7
83.2
Table 6: Layer selection. First-N layer selection performs better than uniform layer selection.
Comparison with pruning. We test the existing structured and unstructured pruning methods reducing pretrained ViT-S to ViT-T. An important thing to note is that our setting is different from neural network pruning. Structured pruning (Li et al., 2017a) typically only prunes within residual blocks for networks with residual connections, and unstructured pruning (Han et al., 2015) prune weights by setting weights to zero instead of removing it. Despite that these pruning methods are not designed for our setting, we can extend structured and unstructured pruning methods to be applied here. Specifically, we can adopt L, pruning and magnitude pruning for element selection. For L pruning.
L1 norm to prune the embedding dimension. For magnitude pruning, we squeeze the selected parameters into the required shape of student.
We present results in Table 7. L1 pruning yields better results compared to random initialization baseline. However, since L1 norm inevitably breaks consistency, it could not reach the same performance with weight selection. Magnitude praning only produces marginally better results over random initialization, presumably due to the squeezing operation which breaks the original structure.
setting random init
VIT-T
ConvNeXi-F
teacher
params
test acc
72.3
81.4
VIT-S
22M
81.4
weight selection
81.4
84.4
ViT-B
86M
77.6
Li pruning 79.5
82.8
ViT-L
307M
76.9
magnitude pruning
73.8
81.9
Table 8: Teacher's size. Smaller teacher provides better initialization.
Table 7: Comparison with pruning. L1 and magnitude pruning performs worse than weight selection.
Teacher's size. We show that initializing from a teacher of closer size produces better results. A larger teacher means higher percentage of parameters will be discarded, which translates more
8

---

Published as a conference paper at ICLR 2024
information loss during weight selection. We present results for using ViT-S, VIT-B, and ViT-L weight selection teacher to initialize ViT-T in Table Interestingly, even a 5M-55---pm-- 3M-parameter subset from 301M parameters in VIT-L is effective initialization, increasing accuracy by 4.5%.
Linear probing. We use linear probing to directly measure the raw model's ability as feature extractor, which be good indicator of the initialization quality. Linear probing technique used to assess a pretrained model's ss representations by training a linear classifier on top of the fixed features extracted from the model.
Following the recipe in He et al. (2022), we apply linear probing on CIFAR-100 to evaluate ViT-T and ConyNeXt-F initialized with weight selection from their ImageNNet221K pretrained teachers, VIT-S and ConvNeXt-T respectively. We compare between random initialization and weight selection as shown in Table 9. Without any training, weight selection performs significantly better than random initialization in producing features.
setting
VIT-T
ConvNeXi-F
random init
13.5
7.1
weight selection
28.2
23.6
Table Linear probing on CIFAR-100. Weight selection produces a non-trivial feature extractor.
Mimetic initialization. Mimetic initialization (Trockman Kolter, 2023) uses the diagonal properties observed in the pretrained self-attention layer's weights to initialize ViTs. We present results for mimetic initialization in Table 10. By directly utilizing pretrained parameters, weight selection outperforms mimetic initialization by large margin. In addition, and VW proj the first head in the first attention block of ViT-T with random initialization, pretrained ViT-S, and ViT-T with weight selection. As shown in Figure 5, weight selection enables small models to inherit the desirable diagonal properties in their self-attention layers, which typically only exist in pretrained models.
visualize the product of W, W
setting random init
CIFAR-10
CIFAR-100
STL-10
92.4
72.3
61.5
mimetic init
93.3
74.7
67.5
weight selection
97.0
81.4
83.4
Table 10: Comparison with mimetic initialization. Weight selection significantly outperforms mimetic initialization by directly utilizing pretrained parameters.
(a) random init
(b) pretrained VIT-S
(c) weight selection
VW Figure 5: Visualization of self-attention layers. Visualization of W. WT (left) and VW proj (right) for ViT-T with random initialization, pretrained VIT-S, and VIT-T with weight selection. Weight selection inherit the diagonal property of self-attention layers that only exists in pretrained ViTs.
CONCLUSION
We propose weight selection, novel initialization method that utilizes large pretrained models. With no extra cost, is effective for improving the accuracy of a smaller model and reducing its training time needed to reach certain accuracy level. We extensively analyze its properties and compare with alternative methods. We hope our research can inspire further exploration into utilizing large pretrained models create smaller models.
Acknowledgement. We would like to thank Haodi Zou for creating illustrative figures, and Mingjie Sun, Boya Zeng, Chen Wang, Jiatao Gu, Maolin Mao for valuable discussions and feedback.

---

Published as a conference paper at ICLR 2024
REFERENCES
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEÎT: BERT pre-training of image transformers. In ICLR, 2022.
Lucas Beyer, Xiaohua Zhai, Amélic Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In CVPR, 2022.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative compo- nents with random forests. In ECCV, 2014.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2bert: Towards reusable pretrained language models. arXiv preprint 11X0v:2110.07143, 2021.
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. In ICLR, 2016.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In CVPR, 2014.
Kevin Clark, Minh-Thame Luong, Quoe V Le, and Christopher D) Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. ICLR, 2020.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, 2011.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
Maciej Czyzewski, Daniel Nowak, and Kamil Piechowiak. Breaking the architecture barrier: method for efficient knowledge transfer across networks. arXiv preprint arXiv:2212. 133700, 2022.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. hierarchical image database. In CVPR, 2009.
Imagenet: A large-scale
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.044 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Hanan Gani, Muzammal Naseer, and Mohammad Yaqub How to train vision transformer on small-scale datasets? In BMVC, 2022.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NeurIPS, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing luvan-Level performance on imagenet classification. In ICCV, 2015.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, 2022.
10

---

Published as a conference paper at ICLR 2024
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IGARSS, 2018.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint urXiv:1503.02533 2015.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. ECCV, 2016.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV, 2020.
Philipp Krähenbühl, Carl Doersch, Jeff Donahue, and Trevor Darrell. Data-dependent initializations of convolutional neural networks. In ICLR, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Tech Report, 2009.
Hugo Larochelle, Yoshua Bengio, Jérôme Louradour, and Pascal Lamblin. Exploring strategies for training deep neural networks. Journal of machine learning research, 2009.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. NeurlPS, 1990.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ICLR, 2017a.
Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, and Yichen Wei. Fully convolutional instance-aware semantic segmentation. In CVPR, 2017b.
Ye Lin. Yanyang Li, Ziyang Wang, Bei Li, Quan Du, Tong Xiao, and Jingbo Zhu. Weight distillation: Transferrigg the knowledge in neural network parameters. arXiv preprint arXiv:2009.00 0et.:00099.91..,. 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. 2021.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In ICLR, 2019.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. In CVPR, 2022.
Zhuang Liu, Zhiqiu Xu, Joseph Jin, Zhiqiang Shen, and Trevor Darrell. Dropout reduces underfitting. In ICML, 2023.
Jonathan Masci, Ueli Meier, Dan Cireşan, and Jürgen Schmidhuber. Stacked convolutional auto- encoders for hierarchical feature extraction. In Artificial Neural Networks and Machine Learning- ICANN. 2011.
Dmytro Mishkin and Jiri Matas. All you need is a good init. In ICLR, 2016.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. NeurIPS, 2011.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008
11

---

Published as a conference paper at ICLR 2024
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. PyTorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamies of learning in deep linear neural networks. In ICLR, 2014.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
Sam Shleifer and Alexander M Rush. Pre-trained summarization distillation. arXiv preprint orXiv:2010.13000 2020.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In NeurIPS, 2014.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.102700 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR, 2016.
JingLei Tang, Dong Wang, ZhiGuang Zhang, LiJun He, Jing Xin, and Yang Xu. Weed identification based k-means feature learning combined with convolutional neural network. Computers and electronics in agriculture, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint wwX.v:1910.106900 2019.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. NeurIPS, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021a
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, 2021b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023
Tricu Trinh, Minh- Minh-Thang Luong, and Quoe Le. Selfie: Self-supervised pretraining for image embedding, arXiv preprint arXiv:1906 176.49090..02940, 02940, 2019.
Asher Trockman and J Zico Kolter. Mimetic initialization of self-attention layers. In ICML, 2023.
Asher Trockman, Devin Willmott, and Zico Kolter. Understanding the covariance structure of convolutional filters. In ICLR, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
12

---

Published as a conference paper at ICLR 2024
Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learning recurrent networks with long term dependencies. In ICML, 2017.
Ross Wightman. GitHub repository: Pytorch image models.
https://github..
twightman/pytorch -mwgemmodelleeeeeeeeemme models, 1s, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXi::2310.0664) 2023.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Ming Zhong, Chenxin An, Weizhu Chen, Jiawei Han, and Pengcheng He. Seeking neural nuggets: Knowledge transfer in large language models from parametric perspective. In ICLR, 2024.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion. In AAAI, 2020.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey transfer learning. Proceedings of the IEEE, 2020.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A TRAINING SETTINGS
Training recipe. We provide our training recipe with configurations in Table 11. The recipe is adapted from ConvNeXt (Liu et al., 2022).
Training Setting
Configuration
optimizer
Adam W
base learning rate
4e-3
weight decay
0.05
optimizer momentum
B1, =0,9, 0.999
batch size
4096
training epochs
300
learning rate schedule
cosine decay 50
warmup epochs
warmup schedule
linear
layer-wise decay (Clark al., 2020; Bao et al., 2022)
None
randaugment (Cubuk et al., 2020)
(9, 0.5)
mixup (Zhang et al., 2018)
0.8
cutmix (Yun al., 2019)
1.0
random erasing (Zhong al., 2020)
0.25
label smoothing (Szegedy et al., 2016)
0.1
layer scale (Touvron ct al., 2021b)
lc-6
head init scale (Touvron et al., 2021b)
None
gradient clip
None
Table 11: Our basic recipe.
Table 12 and Table 13 record batch size, warmup epochs, and training epochs of ConvNeXt-F and Vit-T, respectively, for each dataset. The batch size of each dataset is chosen proportional to its total size. The warmup epochs are set as around one-fifth of the total training epochs. Base learning rates for ConvNeXt-F and ViT-T are 4e-3 and 2e-3 respectively.
C-10
C-100
Pets
Flowers
STL-10
Food101
DTD
SVHIN
EuroSAT
INlk
batch size
1024
1024
128
128
128
1024
128
1024
512
4096
warmup epachs
50
50
100
100
50
50
100
10
50
50
training epochs
300
300
600
600
300
300
600
50
300
300
drop path rate
0.1
0.1
0.1
0.1
0.1
0.2
0.1
0.1
Table 12: Hyper-parameter setting on ConyNeXt-E.
C-10
C-100
Pets
Flowers
STL-10
Food101
DTD
SVHN
EuroSAT
IN1k
batch size
512
512
512
512
512
512
512
512
512
4096
warmup epachs
50
50
100
100
50
50
100
10
50
50
training epochs
300
300
600
600
300
300
600
50
300
300
Table 13: Hyper-parameter setting on VIT-T.
14

---

Published as a conference paper at ICLR 2024
B ADDITIONAL ANALYSIS
Layer selection. To further confirm the first-N layer selection's superiority, we conduct experiments that rule the effect of clement selection. Specifically, we evaluate the results of directly fine-tuning different subsets of 6 layers out of 12 layers of pretrained ViT-T model. We present the results in Table 15. There
clear trend favoring shallow layers, with first-N layers better than mid-N layers, and both of them have better performance than last-N layers. Uniform layer selection is slightly worse than first-N layer selection. Based on this empirical result, we find first-N layer selection suitable method for weight selection.
setting
CIFAR- test acc
first-N layer selection mid-N layer selection
81.6
68.3
last-N layer selection
62.0
uniform layer selection
76.3
Table 14: Layer selection. First-N layer selection performs significantly better than uniform layer selection when ruling out the effect of element selection.
We also find that empirical experiments from ViT-L (which has 24 layers) to VIT-S has different preference over layer selection. Under this setting, uniform layer selection achieves slightly better performance than first-N layer selection. Our main approach would still adopt first-N layer selection the default method, for the reason that the case of smaller teacher (which is favored in weight selection), first-N layer selection consistently outperforms its alternatives.
setting
CIFAR- test acc
first-N layer selection
76.9
mid-N layer selection
75.9
last-N layer selection
77.1
uniform layer selection
77.5
Table 15: Layer selection (VTT-L as teacher). Uniform layer selection yields slightly better results than first-N layer selection when student's 's ratio to teacher small.
Weight components. We conduct ablation studies on VIT-T understand the influence of distinet model components performance. In particular, evaluate the performance of weight selection without one of the following particular types of layers: patch embedding. position embedding, attention block, normalization layer, or MLP layer. As illustrated in Table 16, excluding component from initialization leads to substantial drops in accuracy for all datasets. The results confirm that initializing with all components from pretrained models is necessary.
Setting random init
CIFAR-10 CIFAR-100 100
STL-
92.4
72.3
61.5
weight selection
97.0
81.4
83.4
w/o patch embed
96.8
79.5
77.1
w/o pos embed w/o attention w/o normalization
95.6
78.4
80.2
96.2
77.3
80.5
96.2
79.0
79.8
w/o mlp
95.6
78.8
74.2
Table 16: VIT component ablation. Using all components from pretrained models is the best.
Longer training on ImageNet-1K. To assess if weight selection remains beneficial for extended training durations, we use the improved training recipe from Liu al. (2023). Specifically, the total epochs are extended to 600, and mixup/ / cutmix are reduced to 0.3. The results, as displayed in Table 17, affirm that our method continues to provide an advantage even under extended training durations. Both ViT-T and ConvNeXt-F, when initialized using weight selection, consistently surpass models with random initialization. This confirms that weight selection does not compromise the model's capacity to benefit from longer training.
15

---

Published as a conference paper at ICLR 2024
setting
VIT-T
ConyNeXt-F
test acc
change
test acc
change
random init
73.9
76.1
weight selection
75.5
1.6
76.4
0.3
random init (longer training)
76.3
77.5
weight selection (longer training)
77.4
111
+.22
Table 17: Longer training. Weight selection's improvement is robust under stronger recipe.
C RESULTS ON MORE ARCHITECTURES
We present results on additional architectures, namely ResNet (He al., 2016) and MLP-mixer (Tolstikhin et al., 2021). Model configurations are provided in Table 18 and the detailed results on image classification datasets are presented in 19. Notably, MLP-mixer enjoys more benefits from weight selection. We speculate that it is due to different inductive bias among different architectures.
configuration
student
teacher
model
Mixer-bl6t bl6-half
ResNet-18
Mixer-b16
ResNet-34
depth
12
2 2/2/2/1
12
3/4/6/3
emhodding dimension
384
64 / 128 / 256/512
768
64 / 128 256 512
number of parameters
15.8M
11M
59.9M
63.5M
Table 18: Model Configurations, We perform experiments on Mixer-MLP and ResNet, and use student that halve the embedding dimensions of their corresponding teacher.
dataset (scale 4) random init
weight selection
change
random init weight selection
change
SVHN
95.9
96.2
0.3
87.9
93.3
1.4
Food-101
83.8
85.9
12.1
77.5
83.7
6.2
EuroSAT
97.5
98.6
f 1.1
98.4
98.8
0.4
CIFAR-10
96.4
97.1
0.7
93.5
96.1
22.6
CIFAR-100
82.5
1.2
73.4
79.8
1.4
STL-10
84.3
94.1
19.8
73.3
82.0
1.77
Flowers
78.1
117.7
69.1
83.2
14.1
Pets
78.2
87.8
19.6
50.4
69.6
19.2
DTD
49.5
64.6
115.1
42.9
49.3
1.4
(a) ResNet
(b) MLP-mixer
Table 19: Test accuracy on image classification datasets. Weight selection shows consistent improvements on other architectures across all 9 image classification datasets.
16


Reviews:
[{"review_id": "RHAfQiwycU", "reviewer": "Reviewer_A6vA", "summary": "The paper introduces a novel approach to initialize smaller models given large pretrained models. The approach is simple, efficient and shows good results compared to random init and other numerous baselines and many image classification tasks.", "questions": "1. In Net2net [Chen2015], for initialization the copied weights are rescaled to take into account the width difference (i.e. for wider/thinner networks one needs smaller/larger weight values according to standard weight initialization such as K.He or X.Glorot). Have the authors tried to rescale copied weight values?\n\n2. For \"distill + weight selection\" is the target model first initialized with weight selection and then the distillation is run or it's different?\n\n3. It seems that the results of baselines (random init and mimetic init) are different from previous papers like the mimetic init paper even though the setup looks the same. For example, in the mimetic init paper for CIFAR-10 ViT-T the accuracy is 87.39 for rand init and 91.38 for mimetic init, while in this submission it is 92.4 and 93.3 respectively. Can the authors elaborate on the reasons for that?\n\n4. In L1 pruning experiments, is the final number of selected weights the same as in the proposed weight selection? It's a little bit surprising to see that L1 underperforms uniform weight selection implying that the latter is actually a stronger pruning method. Can the authors elaborate on that?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The problem of obtaining strong weights for small models given large pretrained models is important and solving it efficiently is challenging.\n\n2. The paper explores a very interesting idea of simple weight copying from the source to target network as an efficient and training-free way to initialize smaller model. \n\n3. The uniform selection approach is reasonable and takes into account multi-head structure and group convolution of source networks.\n\n4. The paper has many nicely outlined and run experiments that overall look interesting and convincing enough.\n\n5. The approach outperforms random init and works well on different vision tasks. \n\n6. The paper is well presented and enjoyable to read.", "weaknesses": "1. The paper misses some important related works:\n- [Czyzewski2022] Breaking the Architecture Barrier: A Method for Efficient Knowledge Transfer Across Networks\n- [Chen2021] bert2BERT: Towards Reusable Pretrained Language Models\n- [Chen2015] Net2net: Accelerating learning via knowledge transfer\n\nSpecifically, [Czyzewski2022] introduced a very similar weight transfer technique that seems more general than in the submitted paper since it allows for the source and target architectures to be different. They copied the center slices from source weights, which could be an interesting ablation in this submission. [Chen2015] proposed a simple approach to replicate weights to initialize models. While [Chen2015,Chen2021] focus on initializing a larger model, the idea of copying the weights from the source network to a target one is very similar and should be discussed in detail. This makes the novelty of the paper limited.\n\n2. The paper also should report results of student models pretrained and transferred to smaller datasets. While this would not be a fair baseline to the proposed method since it requires expensive pretraining, it could give important clues about what is the best performance of the model on this task. In addition, some papers (e.g. [Knyazev2021]) reported that very little pretraining is required to achieve competitive transfer learning performance. Therefore, it's not clear if the proposed weight selection approach would be competitive if someone has resources to pretrain a small target net on some large data first (at least for a little bit).\n\n- [Knyazev2021] Parameter Prediction for Unseen Deep Architectures\n\n3. For structured pruning, there are many stronger methods than L1 pruning, e.g. [Kwon2022, Halabi2022] and the baselines therein. The results in Table 6 are quite close and L1 pruning is also very simple, therefore stronger pruning methods could potentially match the performance.\n\n- [Kwon2022] A Fast Post-Training Pruning Framework for Transformers\n- [Halabi2022] Data-Efficient Structured Pruning via Submodular Optimization\n\n4. Most of the experiments are performed on transferring weights from ViT-S to ViT-T which have the same number of layers (12). It would be more interesting to see results when the source model is also deeper (which is usually the case in practice). Related to that, the teacher size results show that ViT-S is the best teacher and a potential reason for that could be that the layer selection approach (first-N) is not the best (perhaps a uniform selection of layers similarly how it is done for weights could be better than first N?). In Table 7, it would be very interesting to see the respective results for distillation which usually works better if the source model is larger/stronger (see the DeiT paper).\n\nDespite all the weaknesses, I'm looking forward to the the authors' response and will be willing to update the rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fm1r3HIyon", "reviewer": "Reviewer_rXy8", "summary": "This work introduces a method aimed at accelerating the convergence of training by transferring pre-trained knowledge from a different network. The approach involves extracting well-trained weights, which exhibit characteristics similar to high-pass filtering (HPF) or low-pass filtering (LPF), from a pretrained network and transplanting these weights onto a novel network, as opposed to starting with random initialization. This technique facilitates quicker training of the new architecture. The authors conduct comparisons between this method and random initialization, demonstrating its superior performance.\n\n\nHowever, I have some concerns as follows:\n\n1)\tFirstly, it's important to note that the comparisons made in this work may not be entirely adequate. From my understanding, this approach falls somewhere between traditional initialization and transfer learning. Given that the method involves adopting a set of pre-trained filters or weights, it can be argued that this work is closer in nature to transfer learning rather than a simple initialization process. Therefore, it might be more appropriate to compare it with transfer learning methods rather than just random initialization. If the proposed method can approach or match the convergence levels of ImageNet pre-training, it would further support its practicality and utility in various applications, reinforcing its value in the field of deep learning.\n\n2)\tComparing the proposed method directly to random initialization may not provide a fair assessment, given the distinct differences in the initialization processes. Unlike the proposed method, a network initialized randomly must construct its structural filters (HPF or LPF) from scratch, which typically requires some epochs and computational resources. In contrast, the proposed method can bypass this initial construction step and is expected to be faster.\n\n3)\tI strongly recommend reconsidering the categorization of this work from merely weight initialization to a more encompassing concept of knowledge transfer to a novel architecture. This work goes beyond traditional weight initialization methods by borrowing well-structured filters from a pre-trained network with a different architecture. This transfer of knowledge to a distinctly different architecture represents a significant departure from conventional weight initialization and indeed underscores the novelty of the approach. Categorizing it as a knowledge transfer method would better capture its unique contribution in the field.\n\n4)\tBuilding upon the suggested change in categorization to knowledge transfer, it raises an interesting question about the potential applicability of the proposed method to highly dissimilar architectures. For instance, exploring the feasibility of transferring knowledge from a pre-trained ResNet to a Vision Transformer (ViT) is a compelling idea like Model Evolution. Training ViT models often necessitates access to large-scale datasets, but they can offer superior performance in various tasks. If the proposed method could indeed facilitate the transfer of knowledge from a ResNet to a ViT, it would hold significant promise. This potential application could lead to advancements in leveraging the strengths of different architectures and addressing challenges related to dataset size, thus expanding the scope and impact of the proposed method.\n\nI appreciate your insights, and it's clear that you find the work promising. However, I think that the manuscript doesn’t well categorize the proposed method and the experiments are not adequately conducted. I believe this work can be much improved.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "See above", "weaknesses": "See above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pW7FResSyM", "reviewer": "Reviewer_HkYt", "summary": "The paper introduces 'weight selection', a simple but effective method that uses the weight of pre-trained large model to initialize the small model. The authors have conducted extensive experiments, showcasing the method's superiority compared to initializing models from scratch.", "questions": "1. Could the authors give further explanation on the difference between 'consecutive selection' and 'uniform selection' in section 3.2?\n2. For comparison with pruning, could the authors elaborate on how L1 pruning was implemented? I am asking the question since the embedding dimension is changed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The experiments are comprehensive, covering both vision transformers and CNNs. Additionally, they have delved into various aspects such as knowledge distillation, comparison with pruning methods, and linear probing, which offer valuable insights\n2. The paper is easy to read and follow.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3juOyKJ3qW", "reviewer": "Reviewer_FWr5", "summary": "The paper suggests that initializing the weight parameters of a smaller neural network from a pre-trained, larger model can significantly improve the smaller model's performance.", "questions": "Please see Section \"Weaknesses\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- This paper is easy to follow;\n- The proposed method can improve the performance.", "weaknesses": "- The proposed method of selecting layers or weights for initialization is somewhat trivial. A more technical approach is encouraged to fully leverage the capabilities of larger models.\n\n- The study is limited in its scope by only employing ViT-T and ConvNeXt-F architectures. Inclusion of additional architectures such as Swin, PVT, and EfficientViT would enrich the evaluation.\n\n- There is an inconsistency in the reported accuracies for the same ImageNet-1k on the same model across different tables. For instance, ConvNeXt-F yields an accuracy of 76.1% in Table 1 but achieves around 82.0% in Table 2. Clarification is needed.\n\n- A comparative analysis involving ConvNeXt is crucial in the \"COMPATIBILITY WITH KNOWLEDGE DISTILLATION\" section, given its architectural differences with ViT.\n\n- The paper lacks sufficient motivation and theoretical grounding for the proposed selection strategy. More insights are needed to justify why, for example, the first N layers are selected, and how this methodology generalizes to other architectures.\n\n- For better comprehension, it would be beneficial to present the customized structures, such as ConvNeXt-F, in tabulated form.\n\n- The potential for leveraging the weights of larger models (which are more powerful) is high. However, Table 7 indicates that the proposed method struggles to effectively learn from these larger models.\n\n- Some grammar errors: \n\t- \"Glorot & Bengio (2010a) maintains \" -> \"Glorot & Bengio (2010a) maintain\";\n\t- \"Lin et al. (2020) transforms\" -> \"Lin et al. (2020) transform\";\n\t- etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)