Re-parameterizing Your Optimizers rather than Architectures
Authors: Xiaohan Ding
Honghao Chen
Xiangyu Zhang
Kaiqi Huang
Jungong Han
Guiguang Ding
Venue: ICLR 2023 poster (2023)
Keywords: Deep Learning
Model Architecture
Optimizer
Re-parameterization

Abstract: The well-designed structures in neural networks reflect the prior knowledge incorporated into the models. However, though different models have various priors, we are used to training them with model-agnostic optimizers such as SGD. In this paper, we propose to incorporate model-specific prior knowledge into optimizers by modifying the gradients according to a set of model-specific hyper-parameters. Such a methodology is referred to as Gradient Re-parameterization, and the optimizers are named RepOptimizers. For the extreme simplicity of model structure, we focus on a VGG-style plain model and showcase that such a simple model trained with a RepOptimizer, which is referred to as RepOpt-VGG, performs on par with or better than the recent well-designed models. From a practical perspective, RepOpt-VGG is a favorable base model because of its simple structure, high inference speed and training efficiency. Compared to Structural Re-parameterization, which adds priors into models via constructing extra training-time structures, RepOptimizers require no extra forward/backward computations and solve the problem of quantization. We hope to spark further research beyond the realms of model structure design. Code and models https://github.com/DingXiaoH/RepOptimizers.

Published as a conference paper at ICLR 2023
RE-PARAMETERIZING YOUR OPTIMIZERS RATHER THAN ARCHITECTURES
Xiaohan Ding" Honghao Chen Xiangyu Zhang451 Kaiqi Huang Jungong Han Guiguang Ding"!
'eencent AI Lab
*CRIS,, Institute of Automation, Chinese Academy of Sciences
Sccooo of Artificial Intelligence, University of Chinese Academy of Sciences
*MEGVII Technology
'Beijing Academy of Artificial Intelligence *Department of Computer Science, the University of Sheffield School of Software, BNRist, Tsinghua University
xi xiachdi xiachding@@ee ng@@ 11. com chenhonghao2021 a aca..aaccco... cn
zhangxi iongxiangee angyu@megvii ccom
kaiqi .eg..huangnnnnn pr. an
2osgoonghan77gmmm com tngggetsinghhng edu.cn
ABSTRACT
The well-designed structures in neural networks reflect the prior knowledge incor- porated into the models. However, though different models have various priors, we are used to training them with model-agnostic optimizers such as SGD. In this paper, we propose incorporate model-specific prior knowledge into optimizers by mod- ifying the gradients according to a set of methodology referred Gradient Re- Ke-paraaeteerization, and the optimizers are named RepOptimizers. For the extreme simplicity of model structure, we focus on a VGG-style plain model and showcase that such simple model trained with a RepOptimizer, which is referred to as RepOpt- VGG, performs on par with or better than the recent well-designed models. From a practical perspective, RepOpt-VGG is favorable base model because of its simple structure, high inference speed and training efficiency. Compared to Structural Re- ae-parameterization which adds priors into models via constructing extra training-ting time structures, RepOptimizers require no extra forward/backward computations and solve the problem of quantization. We hope spark further research beyond the realms of model structure design. Code and models ht htttt::::// /tttt::::/iiii hub. www.:/iiggaaoo///oooooooo hww.:///tppppppiiiiiiieeeei imizers
hyper-parameters Such a
INTRODUCTION
The structural designs of neural networks are prior knowledge incorporated into the model. For example, modeling the feature transformation as residual-addition (y = x (xx)) outperforms the plain form (y f(()) (He al. 2016), and ResNet incorporated such prior knowledge into models via shortcut structures. The recent advancements structures have demonstrated the high-quality structural priors are vital to neural networks, e.g., EfficientNet (Tan & Le (019) obtained set of structural byper-parameet knowledge for constructing the model. Naturally, better structural priors result higher performance.
via architecture search and compound scaling, which served as the prior
Except for structural designs, the optimization methods are also important, which include 1) the first order methods such as SGD (Robbins & Monro 195T) and its variants (Kingma & Bal 20144) Duchi et al.] 2011 Loshchilov & Hutter 201 heavily used with ConvNet, Transformer (Dosovitskiy et al.
2020) and MLP (Tolstikhin et al. 2021 Ding et al. 2022), 2) the high-order methods (Shanno 1970
Hu ct al. 2019
Pajarinen et al. 2019) which calculate approximate the Hessian matrix (Dennis
& Moré 1977:
Roorta-Khorasami & Mahoney 2019). and 3) the derivative- free methods (Rios &
Sahinidis 2013
Berahas al. (019)) for cases that the derivatives may not exist (Sun et al. 2019).
We note that 1) though the advanced optimizers improve the training process in different ways, they have no prior knowledge specific to the model being optimized; 2) though keep incorporating
Equual contributions. This work was partly done during their internships at MEGVII Technology.
I Project leader. *Corresponding author.
Priorr knowledge refers to all information about the problem and the training data Krupka & Tishby (2007 Since we have not encountered any data sample while designing the model, the structural designs can be regarded inductive biases Mitchell (1980), which reflect our prior knowledge.

---

Published as a conference paper at ICLR 2023
our up-to-date understandings into the models by designing advanced structures, we train them with optimizers like SGD (Robbins & Monro 1951) and AdamW (Loshchilov & Hutter 2017). which are model-agnostic. To explore another approach, make the following two contributions.
1) A methodology of incorporating the prior knowledge into a model-specific optimizer. We focus on non-convex models like deep neural networks, so we only consider first-order gradient- based optimizers such as SGD and Adam W. We propose to incorporate the prior knowledge via modifying the gradients according to a set of model-specific hyper-parameters before updating the parameters. We refer to this methodology as Gradient Re-parameterization (GR) and the optimizers as RepOptimizers. This methodology differs from the other methods that introduce some extra parameters (c.g.. adaptive learning rate (Kingma & Bal 2014; Loshchilov & Hutter 2017)) into the training process in that we re- re-parameterize the training dynamies according to some hyper-parameters derived from the model structure, but not the statistics obtained during training (e.g., the moving averages recorded by Momentum SGD and AdamW).
2) A favorable base model. To demonstrate the effectiveness of incorporating the prior knowledge into the optimizer, we naturally use a model without careful structural designs. We choose a VGG-style plain architecture with only a stack of 3x3 conv layers. It is even simpler than the original VGGNet (Simonyan & Zisserman 2014) (which has max max-pooling layers), and has long been considered inferior to well-designed models like EfficientNets, since the latter have more abundant structural priors. Impressively, such a simple model trained with RepOptimizers, which is referred to as KepOpt-VGG, can perform on par with or better than the well-designed models (Table B)
We highlight the novelty of our work through a comparison to RepVGG (Ding et al., (021]. We adopt RepVGG as a baseline because it also produces powerful VGG-style models but with different methodology. Specifically, targeting a plain inference-lime architecture, which is referred to as the target structure, RepVGG constructs extra training-timm -tme structures and converts them afterwards into the target structure for deployment. The differences are summarized as follows (Fig. D. 1) Similar to the regular models like ResNet, RepVGG also adds priors into models with well-designed structures and uses generic optimizer, but PepOpt-VGGC adds priors into the optimizer. 2) Compared to a RepOpt- though the converted Rep VGG has the same inference-time structure, the training- tramning-time RepVGG is much more complicated and consumes more time and memory to train. In other words, RepOpt-VGG VGG is a real plain model during training, but a RepVGG is not. 3) We extend and deepen Structural parrametterization (Ding et al. 2021), which improves the performance of model by changing the training dynamics via extra structures. We show that changing the training dynamics with optimizer has similar effect but is more efficient.
Of note is that we design the behavior of the RepOptimizer following RepVGG simply for a fair comparison and other designs may work as well or better; from broader perspective, we present VGG-style model and an SGD-based RepOptimizer as an example, but the idea may generalize to other optimization methods or models, e.g., RepGhostNet (Chen al., 2022) (Appendix D).
From a practical standpoint, RepOpt-VGG VGG is also a favorable base model, which features both efficient inference and efficient training. 1) As extremely simple architecture, it features low memory consumption, high degree of parallelism (one big operator is more efficient than several small operators with the same FLOPs (Ma et al.) 2018)). and greatly benefits from the highly optimized 3x3 conv (e.g., Winograd Algorithm (Lavin & Gray 2016)). Better still, as the model only comprises one type of operator, we may integrate many 3x3 conv units onto a customized chip for even higher efficiency (Ding et al.) 2021). 2) Efficient training is of vital importance to the application scenarios where the computing resources are limited or we desire a fast delivery or rapid iteration of models, e.g., we may need to re-train the models every several days with the data recently collected. Table 2 shows the training speed of RepOpt- VGG is around 1.8x as Rep VGG. Similar to the inference, such simple models may be trained more efficiently with customized high-throughhut training chips than complicated model trained with general- purpose devices like GPU. 3) Besides the training efficiency, RepOptimizers overcome major weakness of Structural Re-parameterization: the problem of quantization. The inference-ting time RepVGG is difficult to quantize via Post- Training Quantization (PTQ). With simple INTS PTQ, the accuracy of RepVGG on ImageNet (Deng et al. 2209)) reduces to 54.55%. We will show RepOpt- VGG is friendly to quantization and reveal the problem of quantizing VGG results from the structural transformation of the trained model. We naturally solve this problem with RepOptimizers as RepOpr- VGG undergoes no structural transformations at all.
2

---

Published as a conference paper at ICLR 2023
training structure (ResNet)
training structure rogular Optimizer (RepVGG)
training structure (real VGG)
Rep0ptümizer
panmnelens
RN jocsintaas
93
3-3
3
I=1
3
forward
forward
dveeage strlisdiins
update
1x1
updatte
hackward
gradients
3-3
3+3
1=1
3x3
BN grabents
Grad Mult
BV
replace structures
convert parameters
directly deploy
layee
Structural
N layer
Re-paramelerication
inference structure (a) ResNet
inference structure
or orgradient
inference structure
RepV GG + rgullr optimizer
(c) real VGGNet RepOptimizer
Figure 1: The differences among (a) a regular model, (b) structurally- reparameterized RepVGG and (c) real plain model trained with RepOptimizer. In this example, assume the conv layers have two input/output channels so that a 3 layer has 2x2x3x3 parameters and a 1 xx layer has a parameter matrix of 2.2. The parameters of batch normalization (BN) (loffe & Szegedy) (015) denoted by (scaling factor), 3 (bias), (mean) and a (standard deviation). We use rectangle to denote trainable parameter well as its gradient, or accumulated BN parameter together with its batch-wise statistics, Please note that (a) has the same inference-time structure as its training-time structure and does (c), but (b) does not. Though (b) and (c) have the same simple inference-time structure, all the extra structures and parameters of (b) requires forward/backwwrd computations and memory during training, while (c) is simple even during training. The only computational overhead of (c) is an element-wise multiplication of the gradients by pre-computed multipliers Grad MulI.
RELATED WORK
Structural re-paraamttrrzation (SR). RepVGG adopts Structural Re-parameterizatioo (Ding et 202T), which converts structures via transforming the parameters. Targeting VGG-style inference- time architecture, constructs extra identity mappings and 1 conv layers for training and converts the whole block into a single 3x3 conv. In brief, the BN layers are fused into the preceding cony layers (note the identity mapping can also be viewed as x conv layer whose kernel is an identity matrix), and the xx1 conv kernels are added onto the central points of the 3x3 kernels (Fig. [b). A significant drawback of general SR is that the extra training costs cannot be spared. Compared to RepOpt-VGG, RGp has the same target structure but complicated training structure.
Classic re- repaaaanttrizaaton. GR extends and deepens SR just like SR extended the classic meaning of re- paramerization, e.g., DiracNet (Zagoruyko & Komodakis 2017). which encodes a kernel a different form for the ease of optimization. Specifically, a 3 conv kernel (viewed as a matrix) re- parameterized as W = diag(a) (a)l T diag(b) W norm: where a, are learned vectors, and W nonn is the normalized trainable kernel. At each iteration, conv layer first normalizes its kernel, compute W with a. and W norm- then use the resultant kernel for convolution. Then the optimizer must derive the gradients of all the parameters and update them. In contrast, RepOptimizers do not change the form of trainable parameters and introduces no extra forward/hackward computations.
Other optimizer designs. Recent advancements in optimizers include low-rank parameteriza- tion (Shazeer & Stern 2018), better second-order approximation (Gupta et al. (2018), etc., which improve have no prior knowledge specific to the model structure. In contrast, we propose novel perspective that some prior knowledge about architectural design may be translated into the design space of optimizer. Designing the behavior of optimizer is related to meta-learning methods that learn the update function (Vicol et al.) 202T; Alber et al. 2018), which may be combined with our method because adding prior knowledge in way may facilitate the learning process.
3

---

Published as a conference paper at ICLR 2023
train target dataset
target structure
prior knowledge
(imaginary) CSLA structure
Hyper Search striduure
constant
staling
trainable
scaling
constant valuo
dataset
search
construct
Grad Mult RepOptimizer
layer
layer
searched conslants
Figure 2: The pipeline of using RepOptimizers with RepOpt-VGG as an example.
REPOPTIMIZERS
RepOptimizers change the original training dynamics, but it seems difficult to imagine how the training dynamics should be changed to improve a given model. To design and describe the behavior of a RepOptimizer, we adopt a three-step pipeline: 1) we define the prior knowledge and imagine complicated structure to reflect the knowledge; 2) figure out how to implement equivalent training dynamics with a simpler target structure whose gradients are modified according to some syper-parameters: 3) we obtain the hyper-parameters to build up the RepOptimizer, Note the design of RepOptimizer depends on the given model and prior knowledge, and we showcase with SepOpt-VGGr
3.1 INCORPORATE K KNOWLEDGE INTO STRUCTURE
The core of RepOptimizer is the prior knowledge we desire utilize. In the case of RepOpt-VGG the knowledge is that model's s performance may be improved by adding up the inputs and outputs of several branches weighted by diverse scales. Such simple knowledge has sparked multiple structural designs. For examples, ResNet (He et al.) (0166) can be viewed as a simple but successful application of such knowledge, which simply combines (adds up) the inputs and outputs residual blocks; VGG (Ding et al., 2021) shares similar idea but uses different implementation. With such knowledge, we design structures to meet our demands. this case, we desire to improve VGG-style models, choose the structural design of RepVGG (3x3 conv 1x1 conv identity mapping).
3.2 SHIFT THE STRUCTURAL PRIORS INTO AN EQUIVALENT REPOPTIMIZER
Then we describe how to shift the structural prior into a RepOptimizer. We note an interesting phenomenon regarding the above-mentioned prior knowledge: special case where each branch only contains one linear trainable operator with optional constant scales, the model's performance still improves if the constant scale values are set properly. We refer to such a linear block as Constant- Scale Linear Addition (CSLA). We note that we can replace a CSLA block by single operator and realize equivalent training dynamics (i.e., they always produce identical outputs after any number of training iterations given the same training data) via multiplying the gradients by some constant multipliers derived from the constant scales. We refer to such multipliers as Grad Mult (Fig. []. Modifying the gradients with Grad Mult can be viewed as concrete implementation of GR. In brief,
Proposition:
CSLA block regular optimizer single operator + optimizer with GR.
(1)
We present the proof of such equivalency, which is referred to as CSLA = GR, in Appendix A. For the simplicity, here we only showcase the conclusion with two convolutions and two constant scalars as the scaling factors. Let O ag be two constant scalars, W(A), (B)) be two conv kernels of the same shape, X and Y be the input and output, * denote convolution, the computation flow of the CSLA block is formulated as YCSLA = a a (X * W(4)) + ax * W(B)). For the GR counterpart, directly train the target structure parameterized by W", that YGs X W'. Let i be the number of training iterations, we can ensure (i) CSLA = (()))
GR Vi 2 0 as long as we follow two rules.
Rule of Initialization: W should be initialized as (*()) + a ((4))) + ag (B)(0) In other words, the GR counterpart should be initialized with the equivalent parameters (which can be easily obtained by the linearity) as the CSLA counterpart in order to make their initial outputs identical.
while the CSLA counterpart is updated with regular SGD (optionally with momentum) update rule, the gradients of the GR counterpart should be multiplied by (a² + ag). Formally, let I. be the objective function, À be the learning rate, we should update W by
Rule of Iteration:
4

---

Published as a conference paper at ICLR 2023
ØL
W(++ t W - -Ma's + azz) aW!))
(2)
With CSLA = GR, we design and describe the behavior of RepOptimizer by first designing the CSLA structure. For RepOpt- VGG, the CSLA structure is instantiated by replacing the Batch Normalization (BN) following the 3x3 and x 1 layers in RepVGG block by constant channel-wise scaling and the BN in the identity branch by a trainable channel- wise scaling (Fig. 2 since a CSLA branch has no more than one linear trainable operator. In such slightly more complicated case with convolutions of different kernel sizes channel-wise constant scales, the Grad Mult is a tensor and the entries should be respectively computed with the scales the corresponding positions. We present the formulation of the Grad Mult for training a single 3 xx3 conv corresponding to such a CSLA block. Let be the number of channels, s, t RCC be the constant channel-wiss scales after the 3x3 and 1 layers, respectively, the Grad Mult MCxCx3x3 is constructed by
1
+ tt
if = d. p= and = 2,
Me.d.p.-
if d- 2 and 2,
(3)
elsewise
Intuitively. p 2 and q q= 2 means the central point of the 3x3 kernel is related to the I x branch (just like a RepVVGG block that merges the 1x1 conv into the central point of the 3x3 kernel). Since the trainable channel-wise scaling can be viewed as a "depthwise x conv" followed by a constant scaling factor 1. we add 1 to the Grad Mult at the "diagonal" positions (if the output shape does not match the input shape, the CSLA block will not have such a shortcut, so we simply ignore this term).
Remark Compared the common SR forms like RepVGG, CSLA block has no training-timi nonlinearity like BN nor sequential trainable operators, and it can also be converted via common SR techniques into an equivalent structure that produce identical inference results. However, inference- time equivalency does not imply training-time equivalency, as the converted structure will have different training dynamics, breaking the equivalency after an update. Moreover, we must highlight that the CSLA structure is imaginary, which is merely an intermediate tool for describing and visualizing a RepOptimizer but we never actually train it, because the results would be mathematically identical directly training the target structure with GR.
There remains only one question: how to obtain the constant scaling vectors s and t?
3.3 OBTAIN THE HYPER- HYPER-PARAMETRRE PARAMETERS OF REPÖPTIMIZER VIA HYPER--SARRCE
Naturally, the the dyper-parameee of RepOptimizer, Grad Mults affect the performance. Due to the non-convexity of optimization of deep neural network, the research community still lacks a thorough understanding of the black box, so we do not expect provably optimal solution. We propose a novel method to associate the uyper-parameters of an optimizer with the trainable parameters of an auxiliary model and search, which is therefore referred to as Hyper-Search (HS). Given RepOptimizer, we construct the auxiliary Hyper-Search model by replacing the constants in the RepOptimizer's corresponding CSLA model with trainable scales and train it on a small search dataset (e.g., CIFAR-100 (Krizhessky & Hinton] (009)). Intuitively, HS is inspired by DARTS (Liu et al.) 2018) that the final values of trainable parameters are the values that the model expects them to become, so the final values of the trainable scales are the expected constant values in the imaginary CSLA model. By CSLA = GR, the expected constants in the CSLA models are exactly what we need to construct the expected Grad Mults of the RepOptimizer. Note the high-level idea of HS is related to DARTS but we do not adopt the concrete designs of DARTS (e.g., optimizing the trainable and architectural parameters alternatively) since HS is merely an end-to-end training, not a NAS method.
3.4 TRAIN WITH REPOPTIMIZER
After Heper-Search, we use the searched constants to construct the Grad Mults for each operator (each 3x3 layer in RepOpt-VGGl and store them in memory. During the training of the target model on the target dataset, the RepOptimizer element-wiser wiieey multiplies the Grad Mults onto the gradients of the corresponding operators after each regular forward/backwart computation (Rule of Iteration).
We should also follow the Rule of Initialization. To start training with RepOptimizer, we re-initialize the model's 's parameters according to the searched parameerrseeeers In the case of SepOpt-VGGr just
5

---

Published as a conference paper at ICLR 2023
Table 1: Architectural settings of RepOpl/RepVVG, including the number of 3x3 conv layers and
channels of the four stages, and the theoretical inference-time FLOPs and number of parameters.
Model
3x3 layers
Channels
FLOPs (B)
Param (M)
B1
4, 6, 16, 1
128, 256, 512, 2048
11.9
51.8
B2
4, 6, 16, 1
160, 320, 640, 2560
18.4
80.3
L1
14, 24,
128, 256, 512, 2048
21.0
76.0
L2
8, 14, 24,
1
160, 320, 640, 2560
32.8
118.1
like the simplest example above (W < a.A W(4)(0) + ag W(B)())), we should also initialize each 3x3 kernel with the equivalent parameters of a corresponding CSLA block at initialization. More precisely, we suppose to initialize the xx3 and xx kernels in the imaginary CSLA block naturally by MSRA-inittalization (He al. 2015) common practice and the trainable channel-wise scaling layer 1. Let ((*) E C x3x3 and W(0()) E RCxCxIxI be the randomly initialized 3 3 and 1 1x1 kernels, the equivalent kernel is given by (note the channel-wise scaling is a x cony initialized an identity matrix, we should add 1 to the diagonal central positions like Eq. 3
(())(0) 1 1+ so c.d.p.g + ttWW
e,d,1,1
if d. p 2 and
Wo e.d.p.g
Sc W. + tt
w((10)
if + d. = 2 and 2, elsewise
(4)
c,d,1,1
(s))
8. c.d.p.g
We ( to initialize the 3x3 layer in RepOpt- VGG VGG the training dynamics will be equivalent to CSLA block initialized in the same way. Without either of the two rules, the equivalency would break, so the performance would degrade (Table
EXPERIMENTS
4.1 REPÖPTIMERS FOR IMAGENET CLASSIFICATION
We adopt RepVGG as the baseline since it also realizes simple yet powerful ConvNets. We validate RepOptimizers by showing RepOpt- VGG closely matches the accuracy of RepVGG with much faster training speed and lower memory consumption, and performs par with or better than EfficientNets, which are the state-of-the-ard well-designed models. For a simpler example, we design RepOptimizer for RepGhostNer Chen Chen et al. (2022) (a lightweight model with Structural (e-paaaameterzztion)) to simplify the training model, eliminating structural transformations (Appendix D).
Architectural setup. For the fair comparison, RepOpt- VGG adopts the same simple architectural designs as Rep apart from the first stride-2 3x3 conv, we divide multiple 3x3 layers into four stages, and the first layer of each stage has stride of 2. Global average pooling and FC layer follow the last stage. The number of layers and channels of each stage are shown in Table [ Training setup. For training RepOpt- VGG and RepVGG on ImageNet, training settings. Specifically, we use 8 GPUs, a batch size of 32 per GPU, input resolution of 224, and learning rate schedule with 5-epoch warm-up, initial value of 0. and cosine annealing for 120 epochs. For the data augmentation, we use pipeline of random cropping, left-right flipping and RandAAgment (Cubuk et al., 2020). We also use label smoothing coefficient of 0. The regular SGD optimizers for the baseline models and the RepOptimizers for RepOpt- VGG use momentum of 0.9 and weight decay of 4 x 10-5, We report the accuracy on the validation set. Hyper-Search (HS) setup. We use CIFAR- for searching the srper-parameters of RepOptimizers with the same configurations as ImageNet except 240 epochs and simple data augmentation (only left-right flipping and cropping). CIFAR-100 has only 50K images and an input resolution of 32, so the computational cost of HS compared to the ImageNet training is roughly only
adopt the identical
50 x 120 x
240
1281
32 )2 x 1x3+1x1 = 0.18%. Note that the trainable scales of the HS model are initialized according 221
3x3
to the layer's 's depth. Specifically, let be the layer's 's depth (i.e., the first layer in each stage has = 1,
the next layer has = 2), the trainable scales are initialized as / 7- The intuition behind this common trick is to let the deeper layers behave like identity mapping at initialization to facilitate training (Goaa et al. 2017 Shao et al. 2020). which is discussed and analyzed in Appendix B.
Comparisons with VGG. First, we test the maximum batch size (MaxBS) per GPU to measure the training memory consumption and compare the training speed. That is, a batch size of MaxBS+1 would cause OOM (Out Of Memory) error the 2080Ti GPU which has 1 GB of memory. For the

---

Published as a conference paper at ICLR 2023
Table 2: ImageNet accuracy and training speed (2080Ti, measured in samples//eccon//GPPP)
Train MaxBS
Top-1
accuracy
Train speed
Params (M)
Model
@BS=32
@ MaxBS
@ @BS=32
MaxBS
train
inference
RepVGG-BI
198
78.41110.01
78.65
213
243
57.4
51.8
RepOpt-VGG-BB
260
78.47=00.3
78.62:#0.03
380
445
51.8
51.8
RepVGG-B2
79.53-10.08
79.78:10.14
142
163
89.0
80.3
RepOpt-VGG-B2
200
79.36-10.09
79.66:10.12
246
264
80.3
80.3
RepVGG-LI
106
79.39100.07
79.84:10.02
124
136
84.3
76.0
RepOpt-VGG-L1
140
79.46++0.02
79.83+10.03
221
252
76.0
76.0
RepVGG-L2
77
80.52+0.005
80,56+0.09
82
86
131.0
118.1
RepOpt-VGG-LL
103
80.29:10.02
80.53+00009
138
149
118.1
118.1
Table 3: Comparisons among RepOpt-VGGi the original VGGNet-16 1 Simonyan & Zisserman (2014)
and EfficientNets (Tan & Le 2019). The inference speed tested with FP32 on the same 2080Ti
GPU. indicates extra SE Blocks (Hu al., 2018). We also report the official EfficientNets trained with much more consuming scheme (Tan & Le (2199) in parenthesis.
Train
Train
Test
Test batch size
Throughpul (samples/second)
Model
Top-1 acc
epochs
resolution
resolution
RepOpt-VGG-BB VGG-BI
120
224
224
78.47
128
970
EfficientNetB22
120
260
260
76.29 (79.8)
128
912
RepOpt- VGG-B2
120
224
224
79.39
128
645
VGGNet-10 16
120
224
224
74.47
128
621
RepOpt- VGG-L2
120
224
224
80.31
128
372
EfficientNet-B.
120
300
300
79.12 (81.1)
128
301
RepOpt-VGG-L2- VGG-L2
200
224
320
83.13
64
180
EfficientNetB44
300
380
380
82.02 (82.6)
64
183
EfficientNet-B5
300
456
456
82.96 (83.3)
64
fair comparison, the training costs of all the models are tested with the same training script on the same machine with eight 2080Ti GPUs. The training speed of each model is tested with both the same batch size (32 per GPU) and the respective MaxBS. We train the models for 120 epochs both with batch size of 32 and with their respective MaxBS. We linearly scale up the initial learning rate when enlarging the batch size. We repeat training each model three times and report the mean and standard deviation of accuacy. We have the following observations (Table 2h.
1) RepOpt- VGG consumes less memory and trains faster: RepOpt-
trains 1.8 x as fast as Rep VGG-B1 with the respective MaxBS. 2) With larger batch size, the performance every model raises by a clear margin, which may be explained by BN's better stability. This highlights the memory efficiency of RepOptimizers, which allows larger batch sizes. 3) The accuracy of RepOpt- VGG closely matches RepVGG, showing clearly better trade-off between training efficiency and accuracy.
VVG
Comparisons with VGGNet-16 and EfficientNets. As shown in Table 3 the superiority of RepOpt- VGG over the original VG trained with the identical settings can be explained by the depth (VGGNet-16 has 13 conv layers while RepOpt-VGG-B3 VGG-B2 has 27) and better architectural design (e.g.. replacing max- max-pooling with stride-2 conv). Then we compare RepOpt- VGG VGG with EfficientNets, which have abundant high-quality structural priors. As EffiiientNet (Tan Le 2019) did not report the number of epochs, we first train with the identical settings (BS=32) described before. The input resolution is larger for EfficientNets because it is considered as an architectural hyper-pane parameter of EfficientNets. Table 3 shows RepOpt-VGG delivers a f'aooralle speed-accuracy trade-off. For the -fficiennNNe-B44B55 models, we extend the training epochs to 300 to narrow the gap between our implementation and the reported results (Tan & Le 2019). Since EfficientNets use SE Blocks (Hu) et al. 2018) and higher test resolution, we also mcrease the test resolution of RepOpt-VGG-L2 VGG-L2 to 320 and insert SE Blocks after each 3x3 conv. Due to the limited GPU memory, we reduce the batch size while testing the throughput. To highlight the training efficiency of our model, we still use training resolution of 224 and only train for 200 epochs. Impressively, RepOpt-VGG VGG outperforms the official EfficientNet-Be. even though the latter consumed much more training resources.
4.2 ABLATION STUDIES
With VGG-pttVGG-BB first present baseline results with advanced optimizers Shazeer & Stern 2018 Loshchilov Hutter ( Kingma & Ba] (0144; Tieleman & Hinton 2012), which can be viewed re-parameterizing the training dynamics but without model-specific prior knowledge. Then
7

---

Published as a conference paper at ICLR 2023
Table 4: Accuracy of RepOpt-VGG-BI with different behaviors of optimizers.
Optimizer
Source of constants
Change initialization
Modify gradients
Top-1 acc
SGD
N/A
76.91
RMSprop
N/A
76.72
Adam
N/A
76.14
AdamW
N/A
75.03
Adafactor
N/A
74.69
RepOptimizer
Hyper-Search
78,47
RepOptimizer
Hyper-Search
77.42
RepOptimizer
Hyper-Search
77.07
RepOptimizer
All 1
77.59
RepOptimizer Same as HS initialization
77.71
RepOptimizer Average across channels
77.40
Table 5: Transfer experiments with RepOpt- VGG-B1 on different search/target datasets.
Target dataest
Search datasct
Top- target dataset
CIFAR-100
78.47
ImageNet
ImageNett
78.43
Caltech256
78.19
CIFAR-100
67.46
Caltech256
ImageNet
67.55
Caltech256
67.42
we let the RepOptimizer not change the initialization or not modify the gradients. Table 4 shows either ablation degrades the performance because the training dynamics become no longer equivalent change the constant scales to 1) all ones, 2) same the
to training CSLA model. Furthermore,
initial values (vP) in the HS model or 3) the value obtained by averaging the scale values across all the channels for each layer. all the cases, the performance degrades, suggesting that the diverse constant scales, which instruct the RepOptimizer to change the training dynamics differently for each channel, encode the vital model-specific knowledge. Note that our methodology generalizes beyond SGD, which applies to the other update rules like AdamW (please refer to the released code).
4.3 REPÖPTIMIZERS ARE MODEL-SPECIFIC BUT DATASET-AGNOSTE
The typer-paramettrn searched on CIFAR- 100 are transferrable to ImageNet, suggesting the RepOpti- mizers may be model-specific but dataset-agnostic. We further investigate by also Hyper- Searching on ImageNet and Caltech256 (Grifnn et al. (007) and using Caltech256 as another target dataset. The training and search settings on Caltech256 are the same as CIFAR- 100, and the ImageNet search settings are the same as the ImageNet training settings. We have the following observations (Table 5). RepOptimiizers searched on the target datasets deliver no better results
1) The byper-parameters than those obtained on a different dataset: by searching and training both ImageNet, the final accuracy is 78.43%, which closely matches the outcome of the searcheed RepOptimizer. Please note that searching ImageNet does not mean training the same model twice ImageNet because the second training only inherits the HS model's 's trained scales into the RepOptimizer but 2) With different hyper- syper--prrmmttr sources, RepOptimizers deliver similar results on the target datasets, suggesting that the RepOptimizers are dataset-agnostic.
not the other trained parameters.
Interestingly, the accuracy on the search dataset does not reflect the performance on the target dataset (Table (). As RepOpt-VGG is designed for ImageNet, it has five stride-2 conv layers for downsampling. Training its corresponding HS model on CIFAR-100 seems unreasonable because CIFAR-100 100 has resolution of x2232, meaning the 3x3 conv layers in the last two stages operate on 2x2 and Xx] feature maps. As expected, the accuracy of the HS model low on CIFAR (54.53%), but the searched RepOptimizer works well on ImageNet. As we reduce the downsampling ratio of HS model on CIFAR- 100 by re-configuring the five original downsampling layers stride-1, 1, the accuracy of HS model improves, but the corresponding RepOpt-Voo VGG degrades. Such observation serves as another evidence that the searched constants are model-speciffcc, hence the RepOptimizers model-specific: the original model has total downsampling ratio of 32x, but setting a layer's stride to makes a different HS model with 16x downsampling, so the constants searched by such an HS model predictably work poorly with the original 32 downsampling model.
On downstream tasks including COCO detection (Lin et al. 2014) and Cityscapes (Cordts et al. 2016) semantic segmentation, RepOpt-VGG also performs on par with RepVGG (Table 7). For
8

---

Published as a conference paper at ICLR 2023
Table 6: Results of changing the stride of the original five stride-2 layers suggesting the CIFAR-100 accuracy of the Hyper-Search model does not reflect the ImageNet accuracy the target model. Stride
CIFAR-100 acc of Hyper-Search model ImageNet acc of target model
2, 2, 2,
54.53
78.47
2, 1,
70.66
77.81
2, 1,1
74.37
77.88
Table 7: Results COCO detection and Cityscapes segmentation.
Backbone
COCO mAP Cityscapes mloU
RepOpt-VGG-BI VGG-BI 43.50
79.36
RepVGG-BI
43.60
79.15
0.50
0.03
0.00 Kenull Kenuel-Vales
Kenuel Kemal-Values
(a) RepOpt-VGG.
(b) RepVGG.
Figure 3: Parameter distribution of the 3 kernels from RepOpt-VGG VGG and inference-time Rep
COCO, we use the implementation of Cascade Mask R-CNN (He et al. 2019) in MMDetection (Chen et al. 2019) and the Ix training schedule. For Cityscapes. we use UperNet (Xiao et al. 2018) and the 40K-iteration schedule in MMSegmentation (Contributors 2020).
2017 Cai & Vasconcelos
4.4 REPÖPTIMIZERS FOR EASY QUANTIZATION
Quantizing a structurally re- -a-ametterizeed model (i.e., model converted from training-time model with extra structures) may result in significant accuracy drop. For example, quantizing a RepVGG-B to INT8 with simple PTQ -Post-Training Quantization) method reduces the top-1 accuracy around 54.55%. Meanwhile, directly quantizing RepOpt-VGG only results in 2.5% accuracy drop. We investigate the parameter distribution of the kernel of a RepOpt-VGG-BI and inference-time RepVGG-BI. Specifically, we first analyze the 8th 3 x layer in the 16-layer stage of RepOpt- VGG- B1 and then its counterpart from RepVGG-BL. Fig. 3 and Table shows the distributions of the kernel parameters from Rep as the RepOpt-VCG VGG kernel. In Appendix C, we reveal that the structural conversion (i.e., BN fusion and branch addition) cause the coantzzationuuffrinddle unfriendly parameter distribution, and RepOptimizer naturally solves the problem because there is no structural conversion at all.
are obviously different and the standard deviation is almost 4x
CONCLUSIONS AND LIMITATIONS
This paper proposes to shift model's 's priors into optimizer, which may spark further research beyond the realms of well-designed model structures. Besides empirical verification, we believe further investigations (e.g., mathematically provable bound) will be useful, which may require a deeper understanding of the black box deep neural networks. Another limitation is that our implementation presented in this paper relies the linearity of operations training dynamics of a single operator equivalent to a complicated block, but the methodology of shifting the structural priors into the training process may generalize beyond equivalently mimicking the training dynamics of another structure. For examples, we may use some information derived from the structure guide the training, but do not have ensure the equivalency to another model.
desire to make the
Table 8: Quantized accuracy and standard deviation of parameters from RepOpt- VGG RepVGG.
Model
Quantized accuracy
Standard deviation of parameters
RepOpt- VGG-BI
75,89
0.0066
RepVGG-BB B1
54.55
0. 0.0234

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (Nos. 61925107, U1936202, 62021002), Beijing Natural Science Foundation (No. L223023), and the Beijing Academy of Artificial Intelligence (BAAI).
REFERENCES
Maximilian Alber, Irwan Bello, Barret Zoph, Pieter-Jan Kindermans, Prajit Ramachandran, and Quoe Le. Backprop evolution. arXiv preprint arXiv:/80M 02822, 2018.
Albert S Berahas, Richard H Byrd, and Jorge Nocedal. Derivative-free optimization of noisy functions via quasi-newtor methods. SIAM Journal on Optimization. 29(2):965 993, 2019.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cmn: High quality object detection and instance segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
Chengpeng Chen, Zichao Guo, Haien Zeng, Pengfei Xiong, and Jian Dong. Repghost: A hardware- efficient ghost module via re-parameterization. arXiv preprint arXiv:22 1.06888, 2022.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijic Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang. Jiamping Shi, Wanli Ouyang. Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: /906::99000000 09009990.7111 055.,, 2019.
MMSegmentation Contributors. MMSegmentation Openmmlab semantic segmentation toolbox and benchmark. https:///i www. cww/open//epoooooooo mm ab/ wat.aaammmnegeettttioooo 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In 2016 IEEE Conference on Computer Vision and Patlern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 3213-3223. IEEE Computer Society, 2016. doi: 10.1109/CVPR.200 20116.3300
|R..ttts:: doi wwg.oo/g///.... 1109///VRRR...00... CVt.. 201 350
Ekin D Cubuk, Barret Zoph. Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248-255. IEEE, 2009.
John E Dennis, Jr and Jorge J Moré. Quasi-newton methods, motivation and theory. SIAM review, 19 (1):46-89 89, 1977.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13733-13742, 2021.
Xiaohan Ding, Honghao Chen, Xiangyu Zhang, Jungong Han, and Guiguang Ding, Repmlpnet: Hier- archical vision mlp with re- parameterized locality. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 578-587, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11992 202..
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
10

---

Published as a conference paper at ICLR 2023
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv preprint
2017.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimiza- tion. In International Conference on Machine Learning, pp. 1842-1850. PMLR, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-29666 2017.
Jiang Hu, Bo Jiang, Lin Lin, Zaiwen Wen, and Ya-xiang Yuan. Structured quasi-newton methods for optimization with orthogonality constraints. SIAM Journal on Scientific Computing, 41(4): A2239-A2269, 2019.
Jie Hu, Shen, and Gang Sun. Squeere-and-excitatio networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning, pp. 448-456, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11Xiv:1412.6990, 2014.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Eyal Krupka and Naftali Tishby. Incorporating prior knowledge on features into learning. In Artificial Intelligence and Statistics, pp. 227-234. PMLR, 2007.
Andrew Lavin and Scott Gray. Fast algorithms for convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4014-40222 2016.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755 Springer, 2014.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv
preprint arXiv: 8805.0005, 2018
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1711.05 101, 2017.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018.
Tom M Mitchell. The need biases in learning generalizations, Department of Computer Science, Laboratory for Computer Science Research. 1980.
Joni Pajarinen, Hong Linh Thai, Riad Akrour, Jan Peters, and Gerhard Neumann. Compatible natural gradient policy search. Machine Learning, 108(8): 1443-146,, 2019.
Luis Miguel Rios and Nikolaos V Sahinidis. Serivative-free optimization: review of algorithms and comparison of software implementations. Journal of Global Optimization, 56(3):1247-1299 2013.
11

---

Published as a conference paper at ICLR 2023
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
Farbod Roosta-Khorasani and Michael W Mahoney. Sub-sampled newton methods. Mathematical Programming, 174(1):293-326, 2019.
David F Shanno. Conditioning of quasi-newton methods for function minimization, Mathematics of computation, 24(111):647-656, 1970.
Jie Shao, Kai Hu, Changhu Wang, Xiangyang Xue, and Bhiksha Raj. Is normalization indispensable for training deep neural network? Advances in Neural Information Processing Systems, 33, 2020.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604, PMLR, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint
2014.
Shiliang Sun, Zehui Cao, Han Zhu, and Jing Zhao. A survey of optimization methods from machine learning perspective. IEEE transactions on cybernetics, 50(8):3668-366 -3610 2019.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 1946, 2019.
Tieleman and G. Hinton. Divide the gradient by running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4, 2012, 2012.
Ilya Tolstikhin. Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Un- terthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucie, al. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.0160,, 2021.
Paul Vicol, Luke Metz, and Jascha Sohl-Dickstein. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. In International Conference on Machine Learning, pp. 10553-10563. PMLR, 2021.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 418-434, 2018
Sergey Zagoruyko and Nikos Komodakis. Diracnets: Training very deep neural networks without skip-connections. arXiv preprint 18X6:1706.003888 2017.
Matthew D) Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
12

---

Published as a conference paper at ICLR 2023
APPENDIX A: PROOF OF CSLA = GR
CSLA means each branch only comprises one differentiable linear operator with trainable parameters (e.g., conv, FC, scaling layer) and no training-time nonlinearity like BN or dropout. We note that training a CSLA block with regular SGD is equivalent to training a single operator with modified gradients.
We begin with a simple case where the CSLA block has two parallel conv kernels of the same shape each scaled by a constant sealar. Let O the two conv kernels. Let X and Y be the input and output, the computation flow of the CSLA counterpart is formulated as YCSLA a (X W + ax W(B),, where denotes convolution. For the GR counterpart, we directly train the target structure parameterized by W", so that YoR = X W Assume the objective function is L, the number of training iterations is i, the gradient of specific kernel W is OL and F ((aw)) denotes an arbitrary transformation on the gradient of the GR counterpart.
the two constant sealars, W(B), W(B) be
SW
Proposition: there exists a transformation F' determined by only a and ag so that updating W with F ow)) will ensure
Y-
(t) GR
Vi 1 0.
(5)
CSLA
Proof: By the additivity and homogeneity of conv, need to ensure 0.A ((A(0) + ag ((B)
w
Vi 220. 0.
(6)
At iteration O, correct initialization ensures the equivalency: assume W(AXO) W(BX0) the arbitrarily initialized values, the initial condition is
W = aA (A)(O)) an (B)))
(7)
so that
(0) CSLA
Y (0) GR
(8)
Then we use mathematical induction to show that with proper transformations on the gradients of W the equivalency always holds. Let be the learning rate, the update rule is
ØL
w(+1) = w A
Vi 2 0.
OW()
After updating the CSLA counterpart, we have
aL
0L
a w(1)((+)) + an ((B((+1) = a. w(1) + ta W(B) M(as (W(A)
a
(W))
(10)
We use F( F( OL (ow)) to update W which means
aL
W(++1 = w(0 XF
(11)
Assume the equivalency holds at iteration i(i 0). By Eq.
6 10
we must ensure
aL awna
ØL
0L
F( ) = a (W())))
+ aB @W(B))
(12)
Taking the partial derivatives on Eq. we have (w (W(A)) CA,
aw aW(B)) QB
(13)
Then we arrive at
OL F( @w
OL
OW
AL OW())
= a awn)) (W(A)))
+ &B awnl (W(B)))
(14)
ØL
= -a² + a aW!))
13

---

Published as a conference paper at ICLR 2023
which is exactly the formula of the transformation.
With Eq. 14 we ensure a W(4(++1) + og ((B))+)) = WH+1,, assuming a W(16)
oB (B)6
W By the initial condition (Eq. 8) and the mathematical induction, the equivalency
for any i is proved.
Q.E.D.
In conclusion, the gradients use for updating the operator of the GR counterpart should be simply scaled by constant factor, which is (a) + ag) in this case. By our definition, this is exactly the formulation we desire to construct the Grad Mult with the constant scales.
More precisely, with constants OA, on and two independently initialized (e.g.. by MSRA- initialization He et al. 20155) kernels W(4)()) wi the following two training processes always give mathematically identical outputs, given the same sequences of inputs.
CSLA: construct two conv layers respectively initialized as W(A)()) and (( and train the CSLA block (Yes = a Wa) + ax((x * W WB)).
GR: construct a single conv parameterized by W initialize WK()- W(A)()) + W(B)0), and modify the gradients by Eq. 14 before every update.
In this proof, we assume the constant scales are scalars and the two conv kernels are of the same shape for the simplicity, so the Grad Mult is a scalar. Otherwise, the Grad Mult will be a tensor and the entries should be respectively computed with the scales on the corresponding positions, just like the formulation to construct the Grad Mults for RepOpt- SepOpt-VGG, VGG, which is presented in the paper. Except for conv, this proof generalizes to any linear operator including FC and scaling.
APPENDIX B: DISCUSSIONS OF THE INITIALIZATION
INITIALIZATION OF THE HS MODEL
For Hyper-Search (HS), the trainable scales of the HS model are initialized according to the layer's depth. Let / be the layer's depth (i.e., the first block in each stage that has an identity branch is indexed 1 = 1, the next has = 2), the trainable scales (denoted by s, t in the paper) after the conv layers are initialized as / --- The trainable scales in the identity branches are all initialized as 1.
Such an initialization strategy is motivated by our observation: in a deep model with identity shortcuts (c.g., ResNet, RepVGG and an HS model of RepOpt- significant at deeper layer at initialization. In other words, the deeper transformations should be more like an identity mapping to facilitate training (Shao al. (2020). Intuitively, in the early stage of training, the shallow layers learn most and the deep layers do not learn much Zeiler & Fergus 2014)) because the deep layers cannot learn well without good features from shallow layers.
the shortcut's 's effect should be more
For a quantitative analysis, we construct a ResNet with a comparable architecture to RepOpt-VGG-BI, which means the ResNet's first three stages have 4, 6, 16 blocks, respectively. For the blocks within the 16-block stage (except the first one because it has no identity path), we record the variance of the identity path together with the variance of the sum, and compute the ratio of the former to the latter. Fig. [ shows that the identity variance ratio is larger at deeper layers. Such a phenomenon is expected as the output of every residual block has a mean of and variance of I (because it goes through a batch normalization layer). After every shorteut-additiinn the variance of the identity path increases, so that it becomes more and more significant compared to the parallel residual block. This discovery is consistent with a prior discovery Goyal et al. (2017) that initializing the scaling factor of the aforementioned batch normalization (BN) layer as facilitates training.
We did not observe similar pattern in the HS model. Predictably, as there is no BN before the addition, the variances of the three branches all become larger as the depth increases so that the identity variance ratio maintains. Therefore, in order to produce a similar pattern, we initialize the trainable scales after the conv layers in the HS model according to the depth. Specifically, by
initializing the trainable scales as / P. we make the identity branch more significant at deeper layers (Fig. 4 By doing so, the HS model and the RepOpt-VGG VGG model are both improved (Table 9).
14

---

Published as a conference paper at ICLR 2023
Table The accuracy of the HS model with different initialization of the trainable scales after cony layers and the corresponding RepOpt-VGG-a on ImageNet.
Initialization
CIFAR-100 acc
ImageNet
All
47.70
77.59
54.53
78.47
idensity variamter ralo
0.5
Res
HS model, init
0.4
HS model, init by depth
of stage3
Figure 4: The identity variance ratio (the ratio of the variance of the identity path to the variance of the sum). By initializing the trainable scales according to the depth, produce pattern similar to ResNet.
Of note that such an initialization scheme may not be optimal and we do not seek for a better initialization since the performance the target dataset not closely related to the accuracy of the HS model, as shown in Sec. 4.3.
INITIALIZATION OF THE TARGET MODEL
Of note is that the target model on the target dataset initialized independent of the initial values of the conv kernels in the HS model. In other words, the only knowledge inherited from the HS model is the trained scales and we do not need to record any other initial information of the HS model. More precisely, let the initial state (including the values of every randomly initialized conv kernel) of the HS model and the target model be Ô and O', respectively,
note that though the RepOptimizer's hyper-parameters are obtained via training process beginning with è on the search dataset, it works well with O' on the target dataset. As shown in the paper, we simply initialize a 3x3 kernel in the RepOpt- -GG as the equivalent kernel the imaginary CSLA block computed with the trained scales inherited from the HS model and two independently initialized conv kernels. This discovery, again, shows that the RepOptimizers are specific model structures, not datasets nor a certain initial state.
APPENDIX C: DISCUSSIONS OF THE QUANTIZATION
QUANTIZATION RESULTS
Quantizing a structurally re-parameterized network may result in a significant drop in accuracy, which limits its deployment. For example, directly quantizing a RepVGG-B1 to INT8 with simple Post- Sea--Trannng Quantization (PTQ) method reduces the top-1 accuracy to around 54.55%, making the model completely unusable. Even worse, anymore (and is tricky to add BN layers into it again without harming the performance). finetuning with Quantization- Aware Training (QAT) method is not straightforward to implement. Currently,
p--armmtterizeed RepVGG has no BN layers
15

---

Published as a conference paper at ICLR 2023
Table 10:
Quantization results on RepOpt-VGG and RepVGG via both fost-Training Quantization (PTQ) and Quantization-Awan Training (QAT).
Model
PTQ
QAT
RepOpt- VGG-BI
75.89
78.24
RepVGG-1
54.55
N/A
Table 11: The standard deviation of parameters from RepVGG (both before and after the structural transformations) and RepOpt-VGG:
Model
Kernel
Overall
Central
Surrounding
kernel before fusing BN
0.0131
0.0128
0.0131
RepVGG-BI
after fusing BN after branch addition
0.0176
0.0175
0.0177
0.0236
0,0500
0.0177
RepOpt-VGG-B
3x3 kernel
0.0066
0.0151
0.0045
125
Knand bilces
Kmar Videe
Knal Vrice
(a) RepOpt-VGG, overall.
(b) RepOpt-VGG, central positions. (c) RepOpt-VGGr surrounding posi-
tions.
791
390
290
Koral-Vaa
(d) RepVGG overall.
(c) RepVGG, central positions.
(f) RGG surrounding positions.
Figure Parameter distribution of the 3x3 kernel from RepVGG-BI and RepOpt-VGG-BI different locations.
the quantization of structurally re-parameterized models remains an open problem, which may be addressed by custom quantization strategy.
Compared RepVGG, a RepOpt- VGG has the same inference-time structure. contrast, naturally solve the problem of quantization by completely eliminating the need for any structural transformations. For examples, simple PTQ method provided by the PyTTrrhh official library (torch. )ucc.qquantization) can quantize RepOpt-VGG-BB VGG-BI to INT8 with only 2.58% drop in the accu- racy (78. 47% + 75.89%). And with a simple QAT method we narrow the gap to only 0.23% (78.47%-478 24%) after only 10-epoch finetuning.
INVESTIGATION INTO PARAMETER DISTRIBUTION
To understand why VGG is difficult quantize, first investigate the parameter distribution of conv kernels of a RepOOpt-VG-BB VGG-BI and an inference-time RepVGG-BI. Specificallyy, we sample the 8th 3x3 3 layer in the 16-layer stage of RepOpt-VGGG VGG-B1 together with its counterpart from Rep VGG-BI. Fig. 3 and Table in the paper show that the distribution of the kernel parameters of Rep VGG-B1 is obviously different from the RepOpt-VGG kernel.
Following the
official example
htcps://pytorcch.crrgbboogg
rocuctt to- -oe--uunnttto zat on- on on--on-yyorch/
16

---

Published as a conference paper at ICLR 2023
Table 12: ImageNet accuracy of RepGhostNet and RepOpt-ChosiNet.
Model
Optimizer SGD
Extra parallel fusion layers
Top-1 I accuracy
RepCihostNet 0.5x Repoptt-Chosttet 0.5x
66.49
RepOptimizer
66.51
We continue to investigate the sources of such a high- variance parameter distribution of RepVGG. Since a training- -ime RepVGG block undergoes two transformations to finally become a single 3 3 conv, i.e., 1) fusing BN and 2) adding up branches, we evaluate the effects of the two transformations separately. Moreover, since the branch addition involves the identity branch, 1 x I conv and only the central points of the 3x3 kernels, we show the standard deviation of the central and the surrounding
points (i.c., the eight non non-central points in a 3 x matrix) separately (Fig. 5 Table [T].
We make the following observations. 1) In the 3 kernel sampled from RepVGG, the weights at different locations have similar standard deviation, and fusing BN enlarges the standard deviation. After branch addition, the standard deviation of the central points increases significantly, which is expected. 2) The central points of the kernel form RepOpt- than the surrounding points (but lower than the converted kernel from RepVGG), which is expected because we use a Grad Mult with larger values at the central positions. 3) The RepOpt- kernel has much lower standard deviation than the converted RepVGG kernel, which expected since undergoes no BN fusion nor branch addition. 4) Interestingly, even before BN fusion, the surrounding points of the xx3 kernel in RepVGG has a larger variance than the counterparts in the RepOpt- VGG kernel. This can be explained that placing a BN after a conv can be viewed as multiplying a scaling factor to the conv's ' kernel and the scaling factor varies significuntly from batch to batch (because the batch-wise mean and variance may differ significantlyy, leading to high variance of the conv kernel.
have a larger standard deviation
In summary, compared to Structural Re-parameterization, significant strength of Gradient Re- parameterization is the capability to train a model without transforming the structures, which is not only efficient but also beneficial to the quantization performance.
APPENDIX D: GENERALIZING TO REPGHOSTNET
As another example showing the potential generality of the proposed method, RepOptimizer to train a lightweight model named RepGhostNet Chen et al. (2022) without the original parallel fusion layers (implemented by Batch Normalization layers). More precisely, RepGhostNet contains multiple RepGhost Modules, where we use fusion layers parallel to xx convolutional layers (which are followed by Batch Normalization) enrich the feature space. While designing the corresponding RepOptimizer, the CSLA structure is constructed by simply replacing the fusion layer with a trainable scaling layer and the Batch Normalization following X convolution by a constant scaling layer. Similar to RepOpt- VGG, we add a Batch Normalization layer after the addition of the two branches.
We experiment with RepGhostNet 0.5 x. We search the scales of the two imaginary scaling layers for each target operator (i.e., 1x conv). Then we construct the Grad Mults, which can be viewed as and from three branches to two branches).
degraded case of RepOpt-VGe VGG (from 3x3
For training the target model on ImageNet, we follow Chen et al. (2022). extending the training schedule to 300 epochs, enlarging the learning rates and using simpler data augmentation. The RepGhostNet and corresponding
ChoOpttthostNetttt are trained with identical configurations. With RepOptimizer, We obtain an accuracy of 66.51, which closely matches that of our reproduced RepGhostNet (66.49), just like RepOpt- VGG closely matches RepVGG.
We have released the reproducible code and training scripts at httts:://githb.. com/ Setgxiiaai//eeOptiing zers
APPENDIX E: RUN- TIME ANALYSIS OF SEARCHED SCALES
1) We depict the mean value of the scales obtained on different Hyper-Search datasets after each epoch. Specifically, sample the scales from the last block of the third stage, including the scales
17

---

Published as a conference paper at ICLR 2023
Mean of scales searched CIFAR-100
Mcan of scales searched Caltech2566
1.0
1xl
1.0
1xl
3x3 conv
3xd copv
idanlity
identity
0.8
0.7
0.6
0.4
0.3
Epoch
Epoch
(a) Mean values on CIFAR-100.
(b) Mean values on Caltech256.
Figure 6: Mean values of scales from the last block of the third stage during Search on different datasets after each epoch. The scales searched on CIFAR-106 100 and Caltech256 show similar trends.
Mcan scales searched CIFAR-100
Mean of scales searched on Caltech256
1xl
1xl cucv
1.50
3x3 conv
identity
idenrity
1.25
1.0
1.00
0.75
0.3
0.50
0.25
10
15
20
25
0
10
15
20
25
Block Index
Black Index
(a) Mean values on CIFAR-100.
(b) Mean values on Caltech256.
Figure 7: Mean values of the trained scales from different blocks after Hyper-Search The 1st, 2nd, 6th, 12th and last blocks have no identity branches (because they are the output does not match the shape of input) so that the curves are split, and continuous curve represents a stage with more than one block. The scales searched on CIFAR-100 and Caltech256 show similar trends.
of the x 1, 3x3 and identity branches. As shown in Fig. 6 the scales searched on two significantly different datasets show similar trends, which further support that the RepOptimizer is model-specific but dataset-agnostic.
2) In order to show how the searched scales vary with the depth of layers, we show the mean value of the searched scales after the last epoch on different Hyper-Search datasets. As shown in Fig. 7 the hyper-parameters of RepOptimizer searched on the two different datasets show similar patterns: the scales of x and 3x3 branches have similar mean values, which are higher at the beginning of stage; as the depth increases, the mean values of scales at the identity branch vary insignificantlyy
18


Reviews:
[{"review_id": "ZVKRtykqKk_", "reviewer": "Reviewer_96Hr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes incorporating model-specific prior knowledge into optimizers by modifying the gradients according to a set of model-specific hyper-parameters. The proposed methods are employed on VGG architectures using SGD and AdamW optimizers for image classification and segmentation datasets.", "summary_of_the_review": "The paper introduces a new approach to reparameterize gradients and parameters according to the prior knowledge “adding up the inputs and outputs of several branches weighted by diverse scales”.\n\nHowever, there are various major and minor issues with the paper. Therefore, the paper is not ready for publication without fixing these issues.", "strength_and_weaknesses": "The paper introduces a new approach to reparameterize gradients and parameters according to the prior knowledge “adding up the inputs and outputs of several branches weighted by diverse scales”. \n\nAlthough this proposal is interesting, it is more like a general idea instead of a particular method. Therefore, the main weakness of the paper is the limitation of generalization of the proposal to different models with different architectures, optimizers and tasks. The proposed method should be also compared with other other reparameterization and optimizer design methods as well.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NdanWa7To-w", "reviewer": "Reviewer_Hu4S", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to incorporate model-specific prior knowledge into optimizers by modifying the gradients according to a set of model-specific hyper-parameters. They name the method as Gradient Re-parameterization, and the optimizers are named RepOptimizers. They show that a VGG-style plain network can be trained with the proposed optimizer to perform on par with or better than the recent well-designed models. \n", "summary_of_the_review": "The authors explore the space of optimizer design that is equivalent to architecture prior. This paper performs initial exploration and demonstration replacing the residual connections, which could inspire future works in this direction.", "strength_and_weaknesses": "Strength\n- the idea of integrating prior knowledge of architectural design into the design of optimizer is interesting and worth exploring.\n- the authors provide a concrete example of implementing the idea with a plain CNN network, specifically demonstrating the effectiveness of RepOptimizers by showing RepOpt-VGG closely matches the accuracy of RepVGG. \n\nWeakness\n- the model prior is not well defined.\n- the advantage of the proposed method in comparison with traditional architecture design is not quite clear.\n- It is not clear whether all architecture design can be transferred to equivalent optimization design. The authors only provide examples such as converting residual connections, but it is not clear for other architectures whether the RepOptimizers can perform the same. If not, it makes the application of the proposed approach quite limited.\n- The hyper-parameters searched on CIFAR-100 are transferable to ImageNet may not be enough to support the claim that “the RepOptimizers may be model-specific but dataset-agnostic”.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CLlJrC1lphV", "reviewer": "Reviewer_yhDx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a way to impose model-specific prior knowledge into optimizers for training neural networks. Given CSLA structure inspired by the prior, authors seek for parameters of Grad Mult tensor using Hyper-Search. The Grad Mult tensor changes back-propagation gradients directly. It allows us to use the same model architecture for training and testing.", "summary_of_the_review": "See above. Overall the paper is novel, well-written and claims supported.", "strength_and_weaknesses": "Strength:\n\nThe approach of this paper is quite novel. Authors pointed out that existing works mainly focus on structural re-parameterization (SR). Compared to SR, the clear advantage of the proposed RepOptimizer is that the model architecture stays the same during train and test. It makes training faster and lighter without decreasing accuracy (Table 2 and Table 3). The efficacy of Hyper-Search is validated in Table 4 and Table 5. It is impressive to me that the found constants from different search datasets using Hyper-Search ensures similar accuracies on target datasets. The paper showcased two priors, adding up branches (RepOpt-VGG) and locality (RepOpt-MLP), which can be handled by the proposed algorithm on different tasks such as image classification, detection, and segmentation (Table 7). In addition, the parameter distribution in Figure 3 shows that the RepOptimizer is robust to parameter quantization error. Overall, the paper is well-written and presents valuable insights on optimizers with supporting experiments.\n\nWeakness:\n\nAs authors mentioned in the conclusion section, the proposed method relies on the linearity of operations. Hence, the prior should be able to be represented using linear operations in CSLA structure. It might limit the applicable domains of this approach. It would be great if authors show that the non-linear operations can be approximated using RepOptimizer without sacrificing accuracy too much. The Hyper-Search requires another training on an additional dataset. It might not be concretely fair to make comparisons to other approaches that use a single target dataset. I am also curious if the proposed method is scalable to large-scale distributed training scenarios.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "98DppeCWQYO", "reviewer": "Reviewer_FagT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method they call Gradient Reparamerization which allows them to efficiently add priors to the model without additional training structures other than adding a Gradient Multiplier generated based on tuned hyperparameters. They are able to train models that perform on par or better than recent models with a network with a simple structure, and high training and inference efficiency. They achieve comparable results to RepVGG with significantly faster and more memory efficient training with the same inference model. They demonstrate that their models are naturally much easier to quantize compared to structurally re-parameterized models. They derive their method by proposing that based on prior work performance improvements can be made with parallel convolutions multiplied by a single channelwise scaler which are then summed. They derive that the training dynamics are exactly equivalent to by channelwise rescaling the gradients for a single convolution and propose Gradient Reparamerization which involves this rescaling as well as a scale-aware initialization. This method does require hyperparameter tuning of the multipliers, but they find that they are dataset agnostic even to different sized inputs and need only be specialized for each model architecture. They tune the hyperparameter setting the channelwise scalers to be trainable instead of fixed and optimizing it along with the weight parameters. They demonstrate that they can achieve good results on Imagenet by first hyperparameter tuning on cifar10 which theoretically should only require approximately 0.18% of training on full Imagenet.", "summary_of_the_review": "Overall this paper makes a strong contribution. Their novel method to shift architecture priors into the optimizer opens up a new direction to consider for researchers. They demonstrate strong empirical results in improving training efficiency. The main downside is that to achieve best results, their method requires a hyperparameter tuning step and the work would benefit significantly from more analysis of this area. The presentation is clear and they intend to make the code publicly available.", "strength_and_weaknesses": "Strength:\n- Proposes a novel method for gradient reparameterization which allows incorporation of certain model priors equivalent to changes in the training architecture more efficiently.\n- Achieves strongs results on improving training speed while achieving results comparable to prior work while also being easier to quantize.\n- Demonstrated that the hyperparameters seem to be dataset agnostic even with different input sizes.\n\n\nWeakness:\n- The method requires relatively expensive hyperparameter tuning on a per-model basis.\n- Does not conduct significant analysis on the actual values of these important hyperparameters.\n\n\nComments:\nThe paper may benefit from more runtime analysis of the hyperparameter search method in terms of real time measurements as the major empirical improvement of this method is on training speed.\n\nThe paper may benefit from stronger analysis of the search process for hyperparameters. While the reinitialization is justified with the connection to NAS and it could be understandable how it alters the training dynamics, it would benefit from further analysis. Is my understanding correct that you conduct the hyper-parameter search similar to the first order approximation of DARTS no the second order search method? It would be interesting to how the model performs if you reinitialize the network weight, but still allow those parameters to be tuned. Mainly it would greatly improve the understanding of this hyperparameter which is integral to the method if you analyzed how the values and distribution of those hyperparameters changed during the search process as well as how they differ for different layers at different depths and channel sizes.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster