Efficient ConvBN Blocks for Transfer Learning and Beyond
Authors: Kaichao You
Guo Qin
Anchang Bao
Meng Cao
Ping Huang
Jiulong Shan
Mingsheng Long
Venue: ICLR 2024 spotlight (2024)
Keywords: transfer learning
batch normalization
efficient training
Primary area: general machine learning (i.e., none of the above)
TL;DR: Propose Tune Mode, bridging the gap between Eval and Deploy Modes in ConvBN blocks, enhancing computational efficiency while retaining transfer learning performance. Integrated into PyTorch and MMCV.

Abstract: Convolution-BatchNorm (ConvBN) blocks are integral components in various computer vision tasks and other domains. A ConvBN block can operate in three modes: Train, Eval, and Deploy. While the Train mode is indispensable for training models from scratch, the Eval mode is suitable for transfer learning and beyond, and the Deploy mode is designed for the deployment of models. This paper focuses on the trade-off between stability and efficiency in ConvBN blocks: Deploy mode is efficient but suffers from training instability; Eval mode is widely used in transfer learning but lacks efficiency. To solve the dilemma, we theoretically reveal the reason behind the diminished training stability observed in the Deploy mode. Subsequently, we propose a novel Tune mode to bridge the gap between Eval mode and Deploy mode. The proposed Tune mode is as stable as Eval mode for transfer learning, and its computational efficiency closely matches that of the Deploy mode. Through extensive experiments in object detection, classification, and adversarial example generation across $5$ datasets and $12$ model architectures, we demonstrate that the proposed Tune mode retains the performance while significantly reducing GPU memory footprint and training time, thereby contributing efficient ConvBN blocks for transfer learning and beyond. Our method has been integrated into both PyTorch (general machine learning framework) and MMCV/MMEngine (computer vision framework). Practitioners just need one line of code to enjoy our efficient ConvBN blocks thanks to PyTorch's builtin machine learning compilers.

Published as a conference paper at ICLR 2024
EFFICIENT CONVBN BLOCKS FOR TRANSFER LEARNING AND BEYOND
Kaichao You' Guo Qin Anchang Bao Meng Cao Ping Huang Jiulong Shan Mingsheng Long i
t School of Software, BNRist, Tsinghua University, China Apple
(ykc20, bac20, ging20) wtw.ss...sngua.................. edu. cn (mengcac, Huang_ping, jlshan) @apple. com mingsheng@tssinguuae edu. cn
ABSTRACT
iooolution-BacchNooee (ConvBN) blocks are integral components in various computer vision tasks and other domains. A ConvBN block operate in three modes: Train, Eval, and Deploy. While the Train mode is indispensable for training models from seratch, the Eval mode is suitable for transfer learning and beyond, and the Deploy mode is designed for the deployment of models. This paper focuses on the trade-off between stability and efficiency in ConvBN blocks: Deploy mode is efficient but suffers from training instability; Eval mode is widely used in transfer learning but lacks efficiency. To solve the dilemma, we theoretically reveal the reason behind the diminished training stability observed in the Deploy mode. Subsequently, we propose a novel Tune mode bridge the gap between Eval mode and Deploy mode. The proposed Tune mode is stable as Eval mode for transfer learning, and its computational efficiency closely matches that of the Deploy mode. Through extensive experiments in object detection, classification, and adversarial example generation across 5 datasets and 12 model architectures, we demonstrate that the proposed Tune mode retains the performance while significantly reducing GPU memory footprint and training time, thereby contributing efficient ConvBN blocks for transfer learning and beyond. Our method has been integrated into both PyTorch (general machine learning framework) and MMCVMMMnninne (computer vision framework). Practitioners just need one line of code to enjoy our efficient ConvBN blocks thanks to PyTorch's 's builtin machine learning compilers.
INTRODUCTION
Feature normalization (Huang et al. 2023) is a critical component in deep convolutional neural networks to facilitate the training process by promoting stability, mitigating internal covariate shift, and enhancing network performance. BatchNorm (loffe & Szegedy, 2015) is a popular and widely adopted normalization module computer vision. A convolutional layer (LeCun et al., 1998) together with consecutive BatchNorm layer is often called a ConvBN block, which operates three modes:
Train mode. Mini-batch statisties (mean and standard deviation p,, o) are computed for feature normalization, and running statistics (fi, o) are tracked by exponential moving averages for testing individual examples when mini-batch statistics are unavailable.
Eval mode. Running statistics are directly used for feature normalization without update, which is more efficient than Train mode, but requires tracked statistics remain stable in training. It can also be used to validate models during development.
Deploy mode. When the model does not require further training, computation in Eval mode can be accelerated (Markuš 2018) by fusing convolution, normalization, and affine transformations into single convolutional operator with transformed parameters. This is called Deploy mode, which produces the same output as Eval mode with better efficiency.
This work is conducted during Kaichao You's internship at Apple. Mingsheng Long is the corresponding author.

---

Published as a conference paper at ICLR 2024
In Deploy mode, parameters for the convolution are computed once-for-all, removing batch normalization for faster inference during deployment.
The three modes of ConvBN blocks present trade-off between computational efficiency and training stability, as shown in Table | Train mode applicable for both train from scratch and transfer learning, while Deploy mode optimizes computational efficiency. Consequently, these modes traditionally align with three stages in deep models' lifecycle: Train mode for training, Eval mode for validation, and Deploy mode for deployment.
Table I: Trade-off among modes of ConvBN blocks.
Mode
Train
Eval
Tune (proposed)
Deploy
Train From Scratch
x
X
X
Transfer Learning
/
X
Training Efficiency
**
* *
* * *
With the rise of transfer learning (Jiang et al. 2022), practitioners usually start with pre-trained model, and instability of training from scratch less of a concern. For instance, an object detector typically has one pre-trained backbone to extract features, and head trained from scratch predict bounding boxes and categories. Therefore, practitioners have started to explore Eval mode for transfer learning, which is more efficient than Train mode. Figure | presents the distribution of the normalization layers used in MMDetection (Chen et al. 2019), popular object detection framework. In the context of transfer learning, majority of detectors (496 out of 634) are trained with ConvBN blocks in Eval mode. Interestingly, our experiments suggest that Eval mode not only improves computational efficiency but also enhances the final performance over Train mode in certain transfer learning scenarios. For example, Appendix A shows training Faster-RCNi (Ren et al. 2015) on COCO (Lin et al. 2014) with Eval mode achieves significantly better mAP than Train mode, with either pre- ire-trained ResNet101 backbone or pre-trained HRNet backbone.
GN, SyncBN,
BN Eval (496)
And Others
78.2%
21.8%
Configs with Transfer Learning (634)
Figure 1: Usage of normalization layers in all the 634 object detectors with pre-trained backbones in the MMDetection framework (Chen et al., (019). GN denotes GroupNorm, SyncBN represents synchronized BatchNorm across multiple GPUs and Eval indicates training ConvBN blocks Eval mode. A majority of detectors (over 78%) are trained with ConvBN blocks in Eval mode.
Since transfer learning of ConvBN blocks in Eval mode has been a common practice, and forward calculation results between Deploy mode and Eval mode are equivalent, it is natural to ask if we can use Deploy mode for more efficient training. Unfortunately. Section 3.3 shows direct training in Deploy mode can lead to instability, as it is not designed for training.
In quest of efficient ConvBN blocks for transfer learning with pre-trained models, we theoretically uncover the underlying causes of training instability in Deploy mode and subsequently propose novel Tune mode. It bridges the gap between Eval mode and Deploy mode, preserving functional equivalence with Eval mode in both forward and backward propagation while approaching the computational efficiency of Deploy mode. Our extensive experiments across transfer learning tasks (object detection and classification) and beyond (adversarial example generation) confirm the reduction in memory footprint and wall-clock training time without sacrificing performance.
Our contributions summarized follows:
We theoretically analyze why Deploy mode is unstable for training, and propose efficient ConvBN blocks with Tune mode take advantages from both Eval and Deploy modes.
2

---

Published as a conference paper at ICLR 2024
We present extensive experiments across 12 models and 5 datasets to confirm the gain of Tune mode in transfer learning and beyond.
Our method has been quickly integrated into open-source framework libraries like PyTorch and MMCV/MMEEnine because of its evident benefit, improving the efficiency of hundreds of models for everyone using these frameworks.
RELATED WORK
2.1 NORMALIZATION LAYERS
Feature normalization has long been established in machine learning (Bishop 2006 e.g., Z-score normalization to standardize input features for smooth and isotropic optimization landscapes (Boyd & Vandenberghe| 2004). With the emergence of deep learning, normalization methods specifically tailored to intermediate activations, or feature maps, have been developed and gained traction.
Batch normalization (BN), proposed by Loffe & Szegedy (2013), demonstrated that normalizing intermediate layer activations could expedite training and mitigate the effects of internal covariate shift. Since then, various normalization techniques have been proposed to address specific needs, such as group normalization (Wu & He 2018) for small batch sizes, and layer normalization (Ba) cet al. 2016) typically employed in sequence models such as recurrent networks and Transformers. We direct interested readers to the survey by Huang et al. (2023) for an in-depth exploration of normalization layers.
Among various types of normalization, BatchNorm is popular choice, partly due to its ability to be fused within convolution operations during deployment. Conversely, other normalization layers exhibit different behaviors compared to BatchNorm and often entail higher computational costs during deployment. The fusion allows ConvBN blocks to be efficiently deployed to massive edge and mobile devices where efficiency and power consumption control are critically important. This paper focuses on improving the efficiency of widely used ConvBN blocks for transfer learning and beyond.
2.2
VARIANTS OF BATCH NORMALIZATION
While batch normalization successfully improves training stability and convergence, it presents several limitations. These limitations originate from the different behavior during training and validation (Train mode and Eval mode), which is referred to train-inference mismatch (Gupta et al., 2019) in the literature. Loffe (017) proposed batch renormalization to address the normalization issues with small batch sizes. Wang et al. (2019) introduced TransNorm to tackle the normalization problem when adapting model to a new domain. Recently, researchers find that train-inference mismatch of BatchNorm plays an important role in test-time domain adaptation (Wang et al. [021] (022). In this paper, we focus on training ConvBN BN blocks in Eval mode, which is free of train-inference mismatch because its behavior is consistent in both training and inference.
Another challenge posed by BatchNorm is its memory intensiveness. While the computation of BatchNorm is relatively light compared with convolution, it occupies nearly the same memory as convolution because it records the feature map of convolutional output for back-propagation. To address this issue, |Bulo et al. (2018) proposed replace the activation function (ReLU (Nair & Hinton 2010)) following BatchNorm with an invertible activation (such as Leaky ReLU (Maas al., (013)), thereby eliminating the need store the output of convolution for backpropagation. However, this approach imposes an additional computational burden on backpropagation, as the input of activations must be recomputed by inverting the activation function. Their reduced memory footprint comes with the price of increased running time. In contrast, our proposed Tune mode effectively reduces both computation time and memory footprint for efficient transfer learning without any modification of network activations or any other architecture.
2.3 TRANSFER LEARNING
Training deep neural networks used to be difficult and time-consuming. Fortunately, with the advent of advanced network architectures like skip connections (He et al., 2016), and the availability of foundation models (Bommasani et al. 2022), practitioners can now start with pre-trained models and fine-tune them for various applications. rre-trained models offer general representations (Donahue) et al. 2014) that can accelerate the convergence of fine-tuning in downstream tasks. Consequently, the rule of thumb in computer vision tasks is to start with models pre-trained on large-scale datasets like ImageNet (Deng al. 2009), Places (Zhou al. 2018), OpenImages (Kuznetsova et al.
3

---

Published as a conference paper at ICLR 2024
2018). This transfer learning paradigm alleviates the data collection burden required to build deep model with satisfactory performance and can also expedite training, even the downstream task has abundant data (Mahajan et al. 2018).
Train mode is the only mode for training ConvBN blocks from scratch. However, it is possible to use Eval mode for transfer learning. as we can exploit pre-trained statistics without updating them. Moreover, Eval mode is more computationally efficient than Train mode. Consequently, researchers usually fine-tune pre-trained models in Eval mode (Chen et al. 2019) (Figure D. which maintains performance while offering improved computational efficiency. In this paper, we propose novel Tune mode that further reduces memory footprint and training time while maintaining functional equivalence with the Eval mode during both forward and backward propagation. The Tune mode is drop-in replacement of Eval mode while making ConvBN blocks more efficient.
2.4 MACHINE LEARNING COMPILERS
PyTorch (Paszke et al., 2019). widely adopted deep learning framework, uses dynamic computation graphs that are built on-the-fly during computation. This imperative style of computation user- friendly and leads to Py Torch'ss 's rapid rise in popularity. Nevertheless, the dynamic computation graphs complicate the speed optimization. Traditionally, operator analysis and fusion were only applied to models after training. The speed optimization usually involved separate language or framework such as TensorRT (Vanholder) 2016) or other domain-specific languages, distinct from the Python language commonly employed for training. PyTorch has explored several ways, including symbolic tracing with torch. fx (Reed et al., 2022) and just-in-time tracing with torch. jit, to introduce machine learning compilers into the framework, and all the efforts are consolidated into PyTorch 2.0 (Wul 2023). Leveraging PyTorch's pioneering compiler, our proposed Tune mode can automatically identify consecutive Convolution and BatchNorm layers without manual intervention.
3 METHOD
3.1 PROBLEM SETUP
In this paper, we study ConvBN blocks that are prevalent in various computer vision applications, especially in edge and mobile devices. A ConvBN block consists of two layers: (1) convolutional layer with weight w and bias b; (2) a BatchNorm layer with tracked mean f and standard deviation ô, and weight y and bias B. We focus on the computation within ConvBN blocks, which affected by activation functions or skip connections after ConvB BN blocks.
Given an input tensor X with dimensions [N, Cin Hin Wi.. where N represents the batch size, Cin the number of input channels, and Wnn in the spatial height/width of the imput, ConvBN block in Eval mode (the majority choice in transfer learning as shown in Figure [T] operates as follows. First, the convolutional layer computes intermediate output tensor Y convolution). resulting in dimensions [N, Cou, Han. Wou]. Subsequently, the BN layer normalizes and applies an affine transformation the intermediate output, producing the final output tensor Z Y + 3 with the same dimensions as Y
X + b (we
to denote
Usually, the training loss consists of two parts: J = J(Z) calculated on the network's 's output, and regularization loss calculated on the network's trainable parameters. Training dominated by the gradient from J(Z), especially at the beginning of training. The influence of R is rather straightforward to analyze, since it directly and independently applies to each parameter. We omit the analysis for simplicity, as it does not change the main conclusion of this paper. Therefore, our primary focus lies in understanding the gradient with respect to the output loss function J(Z) under different modes of ConyBN blocks. Note that (Z) can represent loss directly calculated on Z, as well as loss computed based on the output of subsequent layers operating on Z.
3.2 PRELIMINARY
3.2.1 BACKWARD PROPAGATION OF CONVOLUTION
To discuss the stability of training, we must examine the details of backward propagation to understand the behavior of the gradient for each parameter. For a convolution layer with forward computation
Y w @ X + b, if the gradient back propagaat Y is 0.
gy. then the gradients of each input of the
convolution layer, as explained in Bouvrie (2006), are: aw BY
8d as X; aJ = wan @ 81, 0J = gy.
ax
3y: 35
0J
The represents cross-correlation, and Whot 18 the rotated version of w. both are used to compute
4

---

Published as a conference paper at ICLR 2024
the gradient of convolution (Rabiner & Gold 1975). Note that these equations potentially contain broadcasting, a technique to allow element- arithmetic between two tensors with different shapes. Appendix B clarifies how broadcasting works in details.
3.2.2 ASSOCIATIVE LAW CONVOLUTION AND AFFINE TRANSFORM
Convolution can essentially be viewed as patch-wise matrix- multiplication, with the matrix (kernel weight) having a shape of [Cou, Cm and the vector having shape of [k Cin.. an affine transform is applied to the weight along the Cout dimension, then the affine transform is associative with the convolution operator. Formally speaking. 7 (w @ X) = (7 w) & X, where 7 is a Caur" dimensional vector multiplied to each row of the weight This association law lays the foundation of analyses for the Deploy mode and our proposed Tune mode. The associative law also applies to transposed convolution (Zeiler al. 2010) and linear layers, therefore the proposed Tume mode also works for TransposedConv-Be and Linear-BN blocks.
With the necessary background established, we directly present the forward, backward, and memory footprint details in Table Further analyses will be provided in subsequent sections.
Table 2: Computation graph of ConvBN blocks in different modes. Shape annotations for each tensor are available in Appendix [ We introduce Tune mode to improve the efficiency of ConvBN blocks, alleviating the dilemma between training stability and computational efficiency.
Computation Graph
Backward Propagation
Mode
0J
0, az
ax
V0a
+
Wha 8
X
0J
aJ +
C X
07
Eval
Y 1 x
Y- # Y VoT-r
0J
aJ
60a +
8J
w
3
87 aZ v +
8J
5Z
0J dn 8J
OX 07
8J 8J
X
Z x=
05 07
0J
AJ
v0a + Ow
Tune (Proposed)
w' b'
V Fc82
W,
?
OJ
8J b
87 Ow V0 Te
0Z V
DJ
03 OZ
X
0J dn
8.
aX 0Z
7 W= 8
8J 0J
Deploy
X
O aZ
w.,b'
aJ AJ
OZ
7
Convolution
w
=
w
8 = (b P)
+ 3 v0 +
0 +
Intermodiate Tensor Saved For Backward Propagation
Parnmeter
Intermediate Tensor
Buffer (Constant)
5

---

Published as a conference paper at ICLR 2024
3.3 ANALYZING EVAL MODE AND DEPLOY MODE
With the help of equations in Table 2 the comparison between Eval mode and Deploy mode on efficiency and training stability is straightforward as follows.
3.3.1 FORWARD COMPUTATION EFFICIENCY
We first observe that Eval mode and Deploy mode have equivalent results in forward computation, and Deploy mode is more efficient. The equivalence can be proved by the definitions w' -ooTz w and
= (b (t) vें+z + B, together with the associative law for convolution and affine transformations. However, Deploy mode pre-computes the weight w' and b', reducing the forward propagation to single convolution calculation. Conversely, Eval mode requires a convolution, supplemented by normalization and an affine transform on the convolutional output. This results in slower forward propagation process for Eval mode. Moreover, Eval mode requires storing X, Y for backward propagation, while Deploy mode only stores X. The memory footprint of Eval mode is nearly double of that Deploy mode. Therefore, Deploy mode emerges as the more efficient the two in terms of memory usage and computational time.
3.3.2 TRAINING STABILITY
Our analyses suggest that Deploy mode tends to exhibit less training stability than Eval mode, Focusing on the convolutional weight, which constitutes the primary parameters in ConvBN blocks, we observe from Table 2 that the relationship of values and gradients between Deploy mode and
Eval mode is w/ V and 8J vā=+e aj The scaling coefficients of the weight V5*+c
8w
are inverse of the scaling coefficients the gradient (V This can cause training instability in Deploy mode. For instance, if
v0"+e is small (say 0.1), the weight reduces to one-tenth of its original value, while the gradient increases tenfold. This is significant concern in real-world applications. As illustrated in Figure [(a) these scaling coefficients range from as low to high as 30, leading to unstable training. Figure 2(b) further substantiates this point through end-to-end experiments in both object detection and classification using Eval mode and Deploy mode. Training performance in Deploy mode is markedly inferior to that in Eval mode.
90
HRNet
RegNet
bbox mAP
0.35
Test Acturary
70
ResNet50
60
ResNet101
40
ResNex:101
0.25
0.3
Daploy:
coco
20
Deploy:
Dusity
2
6
10
12
14 16 18 20 Epoch
10
12
Epoch
90
90
80
80
Fe Accurocy
70
Thxt Accurryy
70
60
0.1
s0
40
30
CU8200 Deploy:
Deploy:
CU8200
20
0
20
2
4
14 Epoch
10 12
16
18
14 16 Epoch
10 12
18 20
(a)
(b)
Figure 2: (a): Distribution of scaling coefficients for weight (7/V0² + in different backbones. (b): Comparison between training with Eval mode and Deploy mode in both object detection and classification. Severe performance degradation is observed for training with Deploy mode.
In conclusion, Deploy mode and Eval mode share the same forward calculation results, but present a dilemma in computation efficiency and training stability.

---

Published as a conference paper at ICLR 2024
3.4 TUNE MODE DEPLOY MODE AND EVAL MODE
Table 2 describes the detailed computation of the proposed Tune mode specifically designed for efficient transfer learning. This mode leverages the associative law of convolution and affine transfor- mation, optimizing both memory and computation. The main point is to calculate the transformed parameters dynamically, on-the-fly. Next, we provide two critical analyses show how the proposed Tune mode addresses the dilemma between training stability and computational efficiency, and how it bridges the gap between Eval mode and Deploy mode.
3.4.1 TRAINING STABILITY
The associative law between convolution and affine transformation readily implies that the forward calculations between Eval mode and Tune mode are equivalent. The equivalence of backward calculations is less intuitive, particularly when considering the gradient of y. To validate this, we employ an alternative approach: let Z1, Zg represent the outputs of Eval mode and Tune mode, respectively. We define Zi = Zi (w, b, 7, 3), Z2 = (z b, 7, 3). Given that Z = Zz. and both are functions computed from the same set of parameters (w, b, 7, [3), we can assert that their Jacobian a2; matrices are the same: J[.. B..,b,7.3] This immediately suggests that both modes share the
az;
same backward propagation dynamics. Consequently, we can conclude that Tune mode is stable as Eval mode in transfer learning.
3.4.2 EFFICIENCY
According to Table 2 Eval mode requires saving the input feature map X and the convolutional output Y. with total memory footprint X + Y for each ConvBN block. In contrast, Tune mode stores X and the transformed weights w', with total memory footprint X + w' for each ConvBN block. Since feature maps Y are usually larger than convolutional weights w', this difference signifies that Tune mode requires less memory for training. 'he same applies to the analysis of computation: computation in Eval mode consists of convolution followed by an affine transformation on the convolutional feature map Y; Tune mode computation consists of an affine transformation on the original convolutional weights w succeeded by a convolution with the transformed weights w'. An affine transformation on convolutional weights executes faster than on feature maps. Therefore, Tune mode outperforms Eval mode both memory usage and computation speed. Please refer to Appendix M for formal analyses of efficiency using the O notation.
The above conclusion can be empirically validated using a standard ResNet-50 (He et al. 2016) model with variable batch sizes and input sizes. The results, displayed in Figure 3 clearly indicate that Tune mode is more efficient than Eval mode across all tested settings. The memory footprint of Tune mode consumed by pre-trained backbone in transfer learning can be reduced to one half of that in Eval mode, and the computation time is reduced by about 10%. The comparison between Tune and Deploy in efficiency be found in Appendix D they are nearly the same in terms of efficiency, but Deploy mode is less stable and incurs worse accuracy than Tune mode.
0.7
Eval Mod: -----
20.
Evall Mode
Eval Mode
Eval Made
o.c-
Turn Mode
-oc.
Tume Modi
Nussiing lime (Seecand)
0.30
Tume Mode
Mannnry Fcorpriet (Cas)
Tume Mode
625
Bunsing time
0.3
0020
02-
01.
255
25.
214 z5o 210 a52 418
z2o 320 aba 418 446
Figure 3: Memory footprint and running time comparison for Eval mode and Tune mode. The base setting is batchsize = 32 and input dimension = 224 x 224, and we vary batchsize and input dimension to test the efficiency.
The comparison among Eval/Tune/Deplog can be summarized as follows:
Deploy mode v Tune mode > Eval mode, in terms of efficiency.
Eval mode Tune mode Deploy mode, in terms of training stability.
Therefore, the proposed Tune mode successfully bridges the gap between Eval mode and Tune mode, improving the efficiency of ConyBN blocks with Eval mode while keeping the stability of training.
7

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
Our algorithm has been tested against 5 datasets and 12 model architectures, as summarized in Appendix B The total computation for results reported in this paper is about 3400 hours of V100 GPU (32GB) counted by our internal computing infrastructure. More details are given in Appendix F
Extensive experiments confirm the benefit of our method, and convince the community of Py PoToch and MMMCV/MMEnninn (Contributors) 2018; 2022) to quickly integrate our method. Consequently, anyone using either library can enjoy the benefit of our algorithm with as simple as one-line code change. We provide guidelines for each library to turn on the Tune mode in Appendix N
Since Tune mode improves efficiency over Eval mode, experiments focus on the efficiency and accuracy comparison between Eval mode and Tune mode. Train mode is often inferior in efficiency and Deploy mode is often inferior in accuracy, so don't 't include them in the main paper but leave the full comparison among four modes Appendix G and
4.1 OBJECT CLASSIFICATION IN TRANSFER LEARNING
We first show the benefit of Tune mode in object classification in transfer learning. We use popular open-source TLlib (Jiang et al., 2022), and the datasets include CUB-200 (Wah et al.] 201T) for fine-grained bird classification, Standford Cars (Krause et al. 2013) and Aircrafts (Maji et al. 2013). The network backbone is ResNet50 pre- see-trained on ImageNet. Each experiment is repeated three times with different random seeds report mean and standard deviation. Results are reported in 3] with further details in Appendix G Compared with Eval mode, the proposed Tune mode reduces than 7% computation time and 36% memory footprint.
Table 3: Results for Tune mode in classification using TLlib.
Dataset
mode
Accuracy
Memory (GB)
Time (second/iteration)
Eval
82.62 (# 0.14)
19.499
0.549
CUB-200
Tune
83.20 (#: 0.00)
12.323 (36.80%.)
0,501 ().74%.)
Eval
85.21 (#: 0.22)
19.497
0.548
Aircrafts
Tune
85.90 (+ 0.26)
12.321 (36.81 %))
0.505 (7.85%
Eval
90.11 (#: 0.03)
19.499
0.541
Stanford Cars
Tune
90.13 (#: 0.12)
12.321 (36.81%) %).)
0,491 (9.24%()
Table Object Detection results on different detectors and backbones.
Detector
Backbone
BatchSize
Precision
mode
mAP
Memory (GB)
Eval
0.3739
3.857
Faster RCNN
ResNe150
2
FP32
Tune
0.3728 (-0.0011)
3.003 (22.15%.)
Eval 0.3824
4.329
Mask RCNN
ResNe150
2
FP32
Tune
0.3825 (+0.0001)
3.470 (19.85%.)
Eval 0.3755
13.687
Mask RCNN
ResNet101
16
FP16
Tune
0.3756 (+0.0001)
9.980 (27.08 %
Eval 0.3675
3.631
Retina Net
ResNer50
2
FP32
Tune
0.3647 (-0.0028)
2.774 (23.59 %
Eval 0.3944
5,781
Faster RCNN
ResNet101
2
FP32
Tune
0.3921 (-0.0023)
4.183 (27.65%
Eval 0.4126
6.980
Faster RCNN
ResNext101
2
FP32
Tune
0.4131 (+0.0005)
4.773 (31.62% ()
Eval 0.3985
4.361
Faster RCNN
RegNet
2
FP32
Tune
0.3995
(+0.0010)
3.8 (28.06% L)
Eval 0.4017
8.504
Faster RCNN
HRNee
2
FP32
Tune
0.4031
(+0.0014)
5.463 (35.76% L)
Eval 0.3350
15.794
Faster RCNN
RepVGG
16
FP16
Tune
0.3350 (+0.0000)
8.996 (43.04%.l
8

---

Published as a conference paper at ICLR 2024
4.2 OBJECT DETECTION TRANSFER LEARNING
This section presents object detection results on the widely used COCO (Lin et al.
2014) dataset. The
MMDetection library uses Eval mode by default, and we compare the results by switching models to Tune mode. We test against various mainstream CNN backbones and detection algorithms (including
Faster RCNN (Ren al. 2015), Mask RCNN (He et al., 2017). and Retina Net (Lin al. 2017))
Results are displayed in Table 4 with additional results available in Appendix H Object detection experiments are costly, and therefore we do not repeat three times to calculate mean and standard
deviation. Appendix shows that the standard deviation of performance across different runs is as small as 0.0005. The change of mAP in Table falls into the range of random fluctuation across experiments.
With different architecture, batch size and training precision (Micikevicius et al. 2018). Tune mode has almost the same mAP as Eval mode, while remarkably reducing the memory footprint by about 20% ~ 40%. Note that detection models typically have a pre-trained backbone for extracting features, and head trained from scratch for producing bounding boxes and classification. The head consumes the major computation time, and the backhone consumes the major memory footprint. Because ConvBN blocks mainly lie in the backbone, our Tune mode mainly benefits the backbone, therefore reducing only the memory footprint. Computation speedup is not obvious in objection detection, and we only report the reduction of memory footprint here.
4.3 APPLICATION OF TUNE MODE BEYOND TRANSFER LEARNING
Our method is designed for transfer learning. However, find that its application can go beyond transfer learning. Any model using Eval mode can benefit from our Tune mode. Adversarial example generation (Szegedy al., 2013) is a representative application of method: when generating adversarial examples for adversarial training (Goodfellow et al., 2015). an important step is to calculate the gradient V,,(O0, x, y) with respect to inputs æ, given inputs, labels y, and parameters 0. Common techniques for producing adversarial samples, such as FGSM (Goodfellow et al. 2013). BIM (Kurakin et al.] 2016), and PGD (Madry et al., 2017). all perturb the inputs based on the gradients, where the model is in Eval mode. Turning on Tune mode can improve the efficiency of adversarial sample generation. Concretely, we perform forward and backward propagation of samples through the model to compute the gradient of input, and measure the time cost as well as GPU memory footprint. The experimental results can be found in Figure with detailed numbers available in Appendix | Across different models, Tune mode can achieve 5%-8% speedup and save 30%-45% of GPU memory. These experiments cover widely used network architectures, and also cover UNet that has transposed convolution layers, demonstrating the broad application of our method.
Time cost
Memory footprint
Select1
7.14%
Roal
Ture
SelecSLS
5.77
10.95
38.06%
Eval
Res2NeRt
0,256
6.25%
Res2Next:
8.22
14:85
43. 89.99%%
0,254
8,786
7.09%
43.41%
ResheSt
ReslieSt
15:37
44.03%
Net
0.548
0.577
5.03%
GhostNit
13.05
15:18
34.825
UNct
5.0
0.011%
13.4
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
10
20
Time/ second
Memory/ GB
Figure 4: Tune mode v.s. Eval mode in adversarial example generation.
CONCLUSION
This paper proposes efficient ConvBN blocks with a novel Tune mode for transfer learning and beyond. Tune Mode is equivalent to Eval mode in both forward and backward calculation while reducing memory footprint and computation time without hurting performance. Our experiments confirm the benefit across dozens of models and tasks, reducing most 44% memory footprint and 9% computation time. We further bring the proposed method into open-source frameworks the community uses everyday, reducing the cost of training networks with ConyBN blocks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS ACKNOWLEDGMENTTS
We would like to thank many open-source contributors for helping the adoption of this technique into PyTorch, MMDetection, and MMCV, including Jason Ansel from Meta, and Wenwei Zhang, Haochen Ye, Zaida Zhou from OpenMMLab.
This work was supported by the National Key Research and Development Plan 202IYFBI7152000 the National Natural Science Foundation of China (U2342217 and 62022050), the BNRist Innovation Fund (BNR2024RC01010, and the National Engineering Research Center for Big Data Software.
Kaichao You is partly supported by the Apple Scholar in AI/ML.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. In NeurlPS Deep Learning Symposium Workshop, 2016.
Christopher M. Bishop. Pattern recognition and machine learning. 2006.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Gcoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong. Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the Opportunities and Risks of Foundation Models. arXiv:2106 08.57258, 2022.
Jake Bouvrie. Notes on convolutional neural networks. 2006.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. 2004.
Samuel Rota Bulo, Lorenzo Porzi, and Peter Kontschieder. In-place activated batchnorm for memory- optimized training of dnns. In CVPR, 2018.
Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, and Jiarui Xu. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: / 196600755, 2019.
MMCV Contributors. MMCV: OLeb computer vision foundation. attps://qithuu:
com/ abb aalaabmmaa hammaabbmmcy///////m mmcv 2018.
MMEngine Contributors. MMEngine: Openmmlab foundational library for training deep learning models. 2022.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. In NeurIPS, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
10

---

Published as a conference paper at ICLR 2024
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In CVPR, 2021.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In ICML, 2014.
Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yi Yu Zhang, Ming Hauan Yang, and Philip Torr. Res2net: A new multi-scale backbone architecture. TPAMI, 2019.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Tanmay Gupta, Alexander Schwing, and Derek Hoiem. No-frills human-object interaction detection: Factorization, layout encodings, and training techniques. In CVPR, 2019.
Kai Han, Yunhe Wang. Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In CVPR, 2020.
K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask R-CNN. In ICCV, 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Normalization techniques in training dnns: Methodology, analysis and application. TPAMI, 2023.
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in batch-normalized models. In NeurIPS, 2017.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML, 2015.
Junguang Jiang, Yang Shu, Jianmin Wang, and Mingsheng Long. Transferability in deep learning: A survey. arXiv preprint arXiv:2201.0588 2022.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d Object Representations for Fine-Grained Categorization. In ICCV Workshop, 2013.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning scale. arXiv preprint 161 101236, 2016.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, and Tom Duerig. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv preprint 1111v:1811.00982, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 1998.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal loss for dense object detection. ICCV, 2017.
Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In JCML, 2013.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.066 2017.
11

---

Published as a conference paper at ICLR 2024
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining, In ECCV, 2018.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5155 2013.
Nenad Markuš.
Fusing batch normalization and convolution in runtime. http://ddnnruuuuccooooo/ussnnn/ sancchorrm-nnd-come 2018.
Dushyant Mchta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Mohamed Elgharib, Pascal Fua, Hans-Peter Seidel, Helge Rhodin, Gerard Pons-Moll, and Christian Theobalt. Xnect: Real-time multi-person 3d motion capture with a single rgb camera. TOG, 2020.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training. In ICLR, 2018.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In NeurIPS, 2019.
Lawrence R. Rabiner and Bernard Gold. Theory and application of digital signal processing. Englewood Cliffs: Prentice-Hall, 1975.
Iijaa Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, 2020.
James Reed, Zachary Vito, Horace He, Ansley Ussery, and Jason Ansel. torch. fx: Practical Program Capture and Transformation for Deep Learning Python. In MLSSys 2022.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Time Object Detection with Region Proposal Networks. In Neur/PS, 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, 2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Han Vanholder. Efficient inference with tensorrt. In GPU Technology Conference, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell, Tent: Fully Test-Time Adaptation by Entropy Minimization. In ICLR, 2021.
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual Test-Time Domain Adaptation. In CVPR, 2022.
Ximei Wang, Ying Jin Mingsheng Long, Jianmin Wang, and Michael I. Jordan. Transferable Normalization: Towards Improving Transferability of Deep Neural Networks. NeurlPS, 2019.
Peng Wu. PyTorch 2.0: The Journey to Bringing Compiler Technologies to the Core of PoTorh. In CGO, 2023.
12

---

Published as a conference paper at ICLR 2024
Yuxin Wu and Kaiming He. Group Normalization. In ECCV, 2018.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wen Lo, and Ross Girshick. Detectron2.
2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. CVPR, 2018.
Matthew D. Zeiler, Dilip Krishnan, Graham W. Taylor, and Rob Fergus. Deconvolutional networks. In CVPR, 2010.
Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong He, Jonas Mueller, R Manmatha, al. Resnest: Split-attention networks. In CVPR, 2022.
B. Zhou, A. Lapedriza, Khosla, A. Oliva, and A. Torralba. Places: 10 Million Image Database for Scene Recognition. TPAMI, 2018.
13

---

Published as a conference paper at ICLR 2024
A COMPARISON OF TRAIN AND EVAL FOR OBJECT DETECTION
We compared the performance of detection models trained in Train Mode and Eval Mode, using two backbones (Resnet101 and HRNet). Results are shown in Table and the training curves are shown in Figure
To ensure fair comparison, we use official training schemes from MMDetection: faster renn/faster renn r50 fpn Ix coco.py and hrnet/faster remn hrnetv2p mmnetv2p_332_1mmmm22 1x _x_cooppy The default choice MMDetection is training with Eval Mode, and we only change model. backbone. no rm_eva =F =Fal se to switch training to Train Mode.
These results indicate that Eval Mode sometimes outperforms Train Mode transfer learning of object detection.
Table 5: mAP of Faster RCNN trained under different ConvBN block Mode
Configuration File
Eval Mode
Train Mode
faster renn/faster remn r50 fpn 1x coco.py
0.3944
0.3708 0.3828
hetpaasseeeerrnnneeeeeeeepeppeee cx_coopppppp
0.4017
Faster RCNN ResNet101
Faster RCNN HRNet
0.430
0.315
0.315
0.322
mAP
6.325
bbox mvd
0.325
0.500
5.755
0.22
5.325
Train Mode
Train Mode
Epoch
Epech
Figure 5: Training curve Faster RCNN with ResNet101 and HRNNe backbone. Models trained in Train Mode shows noticeable performance deterioration compared to Eval Mode.
B BACKWARD PROPAGATION OF BROADCAST
Take the convolution operation as an example: the convolutional output Y has shape of [N, Cout: Hams W Woun], while the tracked mean P has shape of [Cou], and Y f implies first replicat- ing ji to have shape of [N, Comt, Hou. Wou], then performing element-wise subtraction. This can be explained by introducing an additional broadcast operator By, where By (î²) broadcasts ja to match the shape of Y. The underlying calculation is actually Y (ÿ)). The backward calculation for the broadcast operator aBy [A] is the reverse of replication, i.c., summing over a large tensor with the shape of [N, Caus, Hom: Wou] into a smaller tensor with the shape of [Coa]- This helps understand 85y (b) aJ
the backward equation for the bias term LJ 2J 0J summing BY to be compatible with the shape of Due to the prevalence of broadcast in neural
26 7y. which actually means aJ
Ø6 =- 2b BY i.e.,
networks, we omit them to simplify equations.
14

---

Published as a conference paper at ICLR 2024
C CODE DETAILS OF TRAIN/EVAL/DEPLOY MODE
Cmputation details of ConvBN blocks in different modes, with shape annotations for each tensor available in the following code snippet.
input: A faeture map X. The shape of X is [N, Co., Hen. Won]
input: convolutional layer "cony"
output channel number Cour? has wei paraneter W with the shape of [Couts Cin k, and blas parometer with the shape of [Coue]-
kernel-size
input: BatchNorn. layer "ba" with output channel number Couer it has weight parametez with the shape of [Coutl, and a weight parameter B with the shape of [Cout] The momencum DatchNorm is constant number 1).
update
cutput: & bn (conv (X)
code explained in pytorch style inport torch
"Train Mode
output convolution
W X +b & for convolution. The shape of Y is [N, Cout, Hout, Wouel calculate meen for normalization
11
torot.mean (Y, dim- 2, 311 has shape of [Cout]
13
calculate the variation for normalization
a² torch.var dim=(0,
shape [Coud]
update
o keep track moving meen moving variance. They initialized to 0 and 1 respectived LE the model is trained fron sezatch, or are inherited from pre-trained velues. A a(p-a()))
statistics,
6² c- 8² tola" - -
18
normalize the output
20
is small positive numbr to avoid zero division
Y=p P. and o² are broadcast to match the shape
Va?+,
apply the affine transform
==yaY + and 8 are broadcast to match the shape of
24
"Eval Mode"
26
$ calculate the output of convolution
W x
# normalize the output with tracked statistics = Y-A
29
30
Ve24e
apply the affine tranaform
31
= 7
32.
"Deploy Moda"
14
update the weight and blas of the convolution once for
35
W W
8 (b
37
Vo24e
convolution with updated parameters is equivalent to consecutive convolution and batch normal ion
39
W
Listing Computation details for consecutive Convolution and BatchNorm layers different modes
15

---

Published as a conference paper at ICLR 2024
D EFFICIENCY COMPARISON BETWEEN TUNE AND DEPLOY
Table 6: Efficiency comparison between Eval, Tune and Deploy. Time is measured by second/iteration and the memory is the peak memory footprint (GB) during forward and backward. We can observe that Deploy Tune > Eval terms of efficiency.
Batch Size
Input Size
Eval Mode
Tune Mode
DeployMode Time
Time
Memory
Time
Memory
Memory
32
224
0.0945
2.8237
0.0849
1.5973
0.0830
1.5619
256
0.1110
3,5965
0.1032
1.9732
0.1011
1.9416
32
288
0.1488
4.5130
0.1382
2.4325
0.1356
2.3728
32
320
0.1761
5.5216
0.1630
2.9207
0.1609
2.8363
32
352
0.2153
6.6120
0.1991
3.4546
0.1969
3.3682
32
384
0.2503
7,9005
0.2304
4.0995
0.2281
4.0219
416
0.2983
9.2317
0.2738
4.7429
0.2721
4.6640
32
448
0.3567
10.6448
0.3104
5,4306
0.3077
5.3421
16
224
0.0505
1.5671
0.0467
0.9727
0.0448
0,9397
32
224
0.0948
2.8237
0.0849
1.5973
0.0831
1.5631
224
0.1837
5,4125
0.1613
2.8617
0.1590
2.7808
128
224
0.3577
10.6001
0.3081
5.4088
0.3060
5.3284
256
224
0.7035
21.0107
0.6001
10.5011
0.5966
10.4234
E
MODELS AND DATASETS TESTED
We conduct extensive experiments in object detection, classification, and adversarial example genera- tion.
Our experiments cover datasets:
CUB-200 (Wah et al. 2011: CUB-200 is the most widely-used dataset for fine- tine-granned
visual categorization task. It contains 11,788 images of 200 subcategories belonging to birds.
Standford Cars (Kausse al. 2013): Standford Cars consists of 196 classes of cars with total of 16,185 images.
Aircrafts (Maji et al. 2013): Aircrafts contains 10,200 images of aircraft, with 100 images for each of 102 different aircraft model variants.
COCO (Lin al. 4): COCO a large-scale object detection, segmentation, key-point detection, and captioning dataset released by Microsoft. The dataset consists of 328K images.
ImageNet (Deng al. 2009): according to WordNet hierarchy and is instrumental in advancing computer vision and deep learning research.
ImageNet dataset contains 14,197,122 annotated images
Our experiments cover 12 model architectures:
ResNet50, ResNet101 (He al. 2016): ResNet introduced the residual structure, making it possible to train models with hundreds or thousands of layers, which was a significant breakthrough in deep learning.
ResNeXt101 (Xie et al. 2017): ResNeXt is constructed by repeating building block that aggregates set of transformations with the same topology and achieved second place in ILSVRC 2016.
RegNet (Radosavovic et al. 2020): RegNet is self-regulated network for image classifica- tion and can be easily implementer and appended to any ResNet architecture.
HRNet (Sun al. 2019): HRNet is general purpose convolutional neural network for tasks like semantic segmentation, object detection and image classification and able to maintain high resolution representations through the whole process.
16

---

Published as a conference paper at ICLR 2024
RepVGG (Ding et al., 2021): I: RepVGG is simple but powerful architecture of convolutional
neural network, which has a VGG-like inference-ting body composed nothing but a stack of 3x3 convolution and ReLU.
SelecSLS (Mehta et al. 2020): SelecSLS uses novel selective long and short range skip connections to improve the information flow allowing for drastically faster network without compromising accuracy.
Res2NeXt (Gao et al. 2019): Res2NeXt represents multi-scale features granular level and increases the range of receptive fields for each network layer.
DLA eu al. 2018): Extending "shallow" skip connections, DLA incorporates depth and sharing. It contains iterative deep aggregation (IDA) and hierarchical deep aggregation (HDA).
ResNeSt (Zhang et al., 2022): ResNeSt applies the channel-wise attention on different network branches leverage their success in capturing cross-feature interactions and learning diverse representations.
GhostNet is type of convolutional neural network that is built using Ghost modules, which aim to generate more features by using fewer parameters.
GhostNet (Han al. 2020):
UNet (Ronneberger et al. 2013): UNet consists of contracting path and an expansive path and is widely employed across various facets of semantic segmentation.
F ESTIMATION OF TOTAL COMPUTATION USED IN THIS PAPER
Each trial of classification experiments in Section requires about 2 hours of V100 GPU training. The numbers reported in this paper requires 18 trials, which cost about 36 GPU hours.
Each trial of detection experiments in Section 4.2 requires about 12 hours of 8 V100 GPU training, which is 96 GPU hours. The numbers reported this paper requires trials, which cost about 2400 GPU hours.
Each trial of pre-training experiments Section requires about 24 hours V100 GPU training, which is 192 GPU hours. The numbers reported in this paper requires about full trials, which cost about 320 GPU hours.
Some experiments for the purpose of analyses also cost computation. Figure 2 requires two trials of object detection and trials of object classification, with about 204 GPU hours. Figure requires four trials of object detection, with about 384 GPU hours.
Summing the above numbers up, and considering all the fractional computation for the rest analyses experiments, this paper costs about 3400 GPU hours.
Considering the cost of prototyping and previous experiments that do not get into the paper, the total cost of this project is about 5000 GPU hours.
Note that these numbers are rough estimation the cost, and do not include the additional cost for storing data/system maintenance etc.
G COMPARISON OF FOUR MODES FOR OBJECT CLASSIFICATION
The below settings are taken from the default values the TLlib library: ResNet50 the backbone network and all parameters are optimized by Stochastic Gradient Descent with 0.9 momentum and 0.0005 weight decay. Each training process consisted of 20 epochs, with 500 iterations per epoch. We set the initial learning rates to 0.001 and 0.01 for the feature extractor and linear projection head respectively, and scheduled the learning rates of all layers to decay by 0. at epochs 8 and 12. The input images were all resized and cropped to 448 x 448, and the batch size was fixed at 48. Since the backbone network takes the major computation, the memory and time in three different dataset are very similar.
17

---

Published as a conference paper at ICLR 2024
Table 7: Comparison of four modes in classification using TL.lbb.
Dataset
mode
Accuracy
Memory (GB)
Time (second/iteration)
Train
83.07
19.967
0.571
Eval
82.62
19.499
0,549
CUB-200
Deploy
62.96
12.002
0,511
Tune
83.20
12.323
0.501
Train
85.40
19.965
0.564
Eval
85.21
19.497
0.548
Aircrafts
Deploy
41.22
12.000
0.506
Tune
85.90
12.321
0.505
Train
89.87
19.967
0.571
Eval
90.11
19.499
0.541
Stanford Cars
Deploy
47.42
12.002
0.507
Tune
90.13
12.321
0,491
H DETAILED OBJECT DETECTION EXPERIMENTAL RESULTS
For object detection, more detailed comparison of Eval mode and Tune mode is presented in Table while a comparison of the four modes can be found in Table 9
Table 8: Detailed Object Detection experimental results.
Detector
Backhone
BatchSize
Precision
mode
mAP
Memory(GB)
Fval
0.3739
3.857
Faster RCNN
ResNet50
FP32
Tune
0.3728 (-0.0011)
3.003 (22.15%.J
Eval
0.3824
4.329
Mask RCNN
ResNet50
FP32
Tune
0.3825
(+0.0001)
3.470 (19.85%
Eval 0.3755
13.687
Mask RCNN
ResNet101
16
FP16
Tune
0.3756
(+0.0001)
9,980 (27.08% (
Eval 0.3675
3,631
Retina Net
ResNet50
FP32
Tune
0.3647 (-0.0028)
2.774 (23.59% ()
Eval
0.3944
5,781
Faster RCNN
ResNet101
FP32
Tune
0.3921 (-0.0023)
4.183 (27.65%) %.L)
Eval 0.3944
3,849
Faster RCNN
ResNet101
FP16
Tune
0.3925 (-0.0019)
3. 3.138 (18.47 %.)
Eval 0.3922
10.411
Faster RCNN
ResNet101
FP16
Tune
0.3917 (-0.0005)
7,036 (32.41 %..)
Eval 0.3902
19.799
Faster RCNN
ResNet101
16
FP16
Tunc
0.3899 (-0.0003)
12.901
Eval
0.4126
6,980
Faster RCNN
ResNext101
FP32
Tune
0.4131 (+0.0005)
4.773 (31.62%) %
Eval 0.3985
4.361
Faster RCNN
RegNet
FP32
Tune
0.3995 (+0.0010)
3.138 (28.06 (.
Eval 0.4017
8,504
Faster RCNN
HRNet
FP32
Tune
0.4031 (+0.0014)
5,463 (35.76%
Eval 0.3350
15.80
Faster RCNN
RepVGG
16
FP16
Tune
0.3350 (+0.0000)
9,00 (43.04%
18

---

Published as a conference paper at ICLR 2024
Table 9: Comparison of four modes in detection.
Detector
Backbone
Batchsize
Precision
mode
mAP
Memory(GB)
Time(sec/iter)
Train
0.3708
5.782
0.3116
Eval
0.3944
5.781
0.3060
Faster RCNN
10s
2
FP32
Deploy
0,3690
4.02
0.3060
Tune
0.3921
4.18
0.3085
I SMALL RANDOMNESS IN OBJECT DETECTION
The high computational cost limited us to repeating the experiment only once for validating small randomness in object detection. We conducted three trials of the/ RCNN ResNet50) con- figuration and obtained an average best mAP of 0.3748, 0.3735, and 0.3739, with a standard deviation of 0.000543. These results demonstrate that object detection tasks have very little randomness.
ADVERSARIAL EXAMPLE GENERATION
Table 10 presents more detailed model information and batch size information for the adversarial example generation experiments.
Table 10: Adversarial example generation result.
Arch
ResNeSt1Olc
DLA102
SelecSLS42b
Re<2NeXt50
GhostNet_
UNet
Batch
256
512
Time
Eval
0.239
0,286
0.294
0.256
0.577
0.359
(s)
Tume
0,22 (7.95%))
0.264(7.69%L
0,273 (7.14%()
0.240 (6.25%%)))
0.548 (5.03%.
0.341(5.01%..
Memory (GB)
Eval
15.17
16.13
10.93
14.65
18.18
19.11
Tune
(44.03%1)
9.45
(41.41%
6.77 (38.06%))
8.22 (43.89%)
11.85 (34.82%))
13.40 (29.88%
K
TUNE MODE CONvBN FOR PRE-TRAINING
Tune mode is designed for transfer learning because it requires tracked statistics to normalize features. Here we show that Tune mode can also be used in late stages of pre-training. We use the prevalent ImageNet pre-training as baseline, which has three stages with decaying learning rate. We tried to turn on Tune mode at the third stage, the accuracy slightly dropped. Nevertheless, due to our implementation with torch. fx, we can dynamically switch the mode during training. Therefore, we also tried to alternate between Train mode and Tune mode at the third stage, which retained the accuracy with less computation time.
70-
Train
Stage2 Train Mode
Stage3
Altemate
Top1 Accuracy
231
Mw
Mode
Tone
Alternate
Stagel
Top-1 Accurney
75.684
74.548
75.634
Train Mode
Memory (GB)
5.258
4.021
5.258
Time (s/iter)
0.887
0.820
0.853
20
60
80
Epoch
Figure 6: ImageNet Pre-training Results
L COMPARING WITH ALTERNATIVES TO REDUCING MEMORY FOOTPRINT
L.1 FROZENBATCHNORM
FrozenBatchNorm,, as used in Detectron2 (u et al. 2019). freezes the weight and bias of Batch- Norm layers, reducing memory footprint at the cost of less trainable parameters, which limits the
19

---

Published as a conference paper at ICLR 2024
network's expressive power and hinders the accuracy of trained models. As shown in Table [1 FrozenBatchNorm's mAP in detection is lower than baseline, while our Tune mode doesn't t hurt mAP.
Table 11:
Comparision between proposed detection baseline and FrozenBatchNorm. FrozenBatch- Norm incurs performance loss.
Faster RCNN ResNet 50 2X
Mask RCNN ResNet 50 2X
Baseline FrozenBatchNorm
0,3832
0.3911 0,3904
0.3790
L.2 INPLACE-ABN
We compare our proposed Tune Mode with the Inplace-ABN (Bulo et al. 2018), memory reduction method for ConvBN blocks by invertible activations. We show transfer learning experiments on TLlib (Jang al. 2022) with the same settings of section 4.1
To apply Inplace-ABN blocks, we find BN-ReL.U patterns in the pretrained network using torch. fx and replace them with the Inplace-ABN blocks provided by Bulo et al. (2018). Results are summarized in Table 12 Inplace-ABM ABN block saves approximately 16% memory at the cost of 8% additional computation time, and also hurts the accuracy significantly as it modifies the network architecture. Compared to Inplace-ABN, computation time, while retaining the accuracy.
proposed Tune mode more memory footprint and requires less
Table 12: Comparision between proposed Tune Mode and the Inplace-ABN (Bulo et al. 2018). We can observe that Inplace-ABN saves memory at the cost of speed and accuracy, while Tune mode saves memory at no cost of computation time or accuracy.
Dataset
Method
Accuracy
Memory (GB)
Time (second/iteration)
Baseline
83.07
19.967
0.571
CUB-200
Inplace-ABN
74.80 (8.27.)
16.739 (16.17%..
0.623 (9.10%f)
Tune Mode (ours)
83.20 (0.131)
12.323 (38.28%.)
0.501 (12.26%.)
Baseline
85.40
19.965
0.564
Aircraft
Inplace-ABN
78.23 (7.17.)
16.737 (16.17%.)
0.620 (8.58%
Tune Mode (ours)
85.90 (0.501)
12.323 (38.28%))
0.505 (10.51%4)
Baseline
89.87 06
19.967
0.571
Stanford Car
Inplace-ABI ABN
86.30 2s (3.57
16.739 (16.17%.)
0.614 (7.53%))
Tune Mode (ours)
90.13 (0.26*)
12.321 (38.28%.)
0.491 (14.00%.)
M
THEORETICAL ANALYSES OF BENEFIT IN MEMORY/TIME COST
M.
MEMORY ANALYSIS
Memory cost for Eval mode: O(X + Y) O(NC + Nouu Wout).
Memory cost for Tune mode: O(X + w') = ON Hi Wi, + Aanin
For each ConvBN block, the memory cost reduction of Tune mode is O(N W kk= Cin Cout).
conclude that networks with larger feature maps (larger Hout W out such as HRNet that features high resolutions), smaller kernel sizes (smaller k. such as RepVGG with many k 1 conv kernels), and larger batch sizes (larger N) will benefit more from the proposed Tune mode. The conclusion can be empirically validated from Table 8 We can observe that:
From the analysis,
The memory cost reduction ratio grows from 18. 18.47% to 34.83% when batch size grows from 2 to 16, for the same Faster RCNN detector with ResNet101 backbone.
20

---

Published as a conference paper at ICLR 2024
The memory cost reduction ratio grows from 18.47% to 35. when changing the network backbone from ResNet101 to HRNet while keeping the rest the same.
The memory cost reduction ratio grows from 34.83% to 43.04% when changing the network backbone from ResNet101 to RepVGG while keeping the rest the same.
M.2 TIME ANALYSIS
As pointed out by the FlashAttention paper (Dao et al. 2022). the computation time of modern GPU hurdware often scales with memory access (the number of bytes the program reads from and writes to memory). The cost of memory access can be effectively estimated by summing all node sizes from computation graphs in Table 2 Therefore, can analyze the time cost of Eval mode and Tune mode as follows:
Eval mode time cost:
+
-a Wont + h2 Cou
Tune mode time cost:
O(X (1x+++btw'a +wwttttttttttttttttttuttt (uto::u+++oooo+++++0
Wout + 2k 222 +
For each ConvBN block, the time cost reduction of Tune mode is:
O2 Hom W our Cn Cont).
Since feature maps (size NCou Houl are typically much larger than convolutional kernels (size Com),, the proposed Tune mode can reduce time cost.
N
INTEGRATION WITH DL COMPILERS AND COMMON LIBRARIES
Our algorithm has been integrated into PyTorch core, MMCV, and MMEngine. We also support standalone usage.
N.1 PYTORCH
Our method has been integrated into PyTorch core since version 2.2. People using PyTorch can turn on Tune mode the following code
torch forch._industor.c. htw.:_iiinnttrr.oooooooo config www.ffiggefiiiiettccoom Sn__ean____a evai_ fx_passes
Frue
N.2 MMCV/MMENGINE
Our method has been integrated into popular computer vision libraries. MMCV holds operators, while MMEngine is the training framework. People using MMCV/MMFngine can turn on the Tune mode via adding command line argument:
--afg-options lerffcieet__oo______ = = (backbore)
N.3 STANDALONE USAGE
For people using old versions of PyTorch (we require PyTorch larger than 1.8), they can turn on Tune mode via the online code
madel MyMadel() () init model
import tatee__od__coovbss.
cur.coo_effoiiiiinnttttt......t. bo__val_ [aal___o____ee sarr_singll_mdae (model) # now this model oon benefit from tune mode, if in trained with "Eval" mode.
21


Reviews:
[{"review_id": "JL6XXobaGD", "reviewer": "Reviewer_E7re", "summary": "This paper primarily investigates the forward computation and backpropagation process in model training or testing of three modes (i.e., Train, Eval, Deploy) of the ConvBN module (Convolution layer plus BN layer), with a focus on the instability of the Deploy mode during transfer learning. Thus, a new mode called Tune mode is proposed, which boasts the advantages of training stability and low computational time and space costs. \n\nTo verify the effectiveness of Tune mode, the authors selected 5 datasets and 12 models, and conducted extensive experiments in classification and detection tasks. The experiments show that using Tune mode for transfer learning results in a slight improvement in model performance, and a significant reduction in time and space costs. Furthermore, the authors tested the effectiveness of Tune mode in the generation of adversarial samples, thereby demonstrating the general applicability of Tune mode as a replacement for Eval mode in tasks that originally used Eval mode.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The article is clearly articulated, with detailed experimental settings, and the proposed Tune mode is simple to implement and easy to reproduce.\n2. The theoretical analysis and experimental verification of the instability in Deploy mode training presented in the article seem plausible.\n3. The author conducted a large number of experiments to validate the advantages of Tune mode in terms of training stability and low time-space cost, with the latter being significantly beneficial.", "weaknesses": "1. The method proposed in this article is only applicable to methods that originally used Eval mode, its application scenarios are not very broad, and the impact is relatively small.\n2. This article only provides experimental verification for the time and space cost advantages of Tune mode. Is it possible to make theoretical estimates and give a result similar to O(N)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X0SAJi1Hyf", "reviewer": "Reviewer_nwrB", "summary": "The paper proposes a novel calculation strategy for ConvBN by changing the order of calculation. The proposed Tune mode have exact same forward/back propagation expression with the eval mode yet are faster and has lesser memory footprint, demonstrated both theoretically and empirically. This makes it a useful component in transfer learning. The authors also attributed the instability of the training using deploy mode to the scaled weight and gradient, leading to the necessity of saving part of the parameters.", "questions": "Is it possible to merge $b$ and $\\beta$?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The writing is pretty clear with solid proof and extensive result.\n2. The proposed method can be used widely in computer vision tasks.", "weaknesses": "Not much.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "79NPObxHup", "reviewer": "Reviewer_1pnn", "summary": "In this paper, the authors introduce the Tune mode of ConvBN blocks, which can save memory and time costs during transfer learning. Experiments show that when transferring the backbone with ConvBN to downstream tasks, Tune mode can save 20%-40% of memory costs without loss of accuracy.", "questions": "1. The conclusion that \"Eval mode gets significantly better mAP than Train mode\" is too strong for me. From my personal experience, I sometimes get better accuracy using BN's training mode than using eval mode for transfer learning. Therefore, the authors should provide more experiments than the two selected models to verify this strong conclusion.\n2. As SyncBN's eval mode has similar behavior to BN, I am not sure whether the Tune mode can be directly applied to the backbone trained with SyncBN. The author can explain this in the response.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The writing and presentation of the paper are clear and easy to understand. \n2. The method (Tune mode for ConvBN blocks) is reasonable and novel.\n3. The experiments on five datasets and 12 model architectures show the method's effectiveness: Tune mode can save 20%-40% of memory costs without losing accuracy.", "weaknesses": "1. Although the method to reduce the cost of Eval mode is very clever, the contribution of the paper to academic research is limited. This paper is more like introducing a useful technical trick to reduce the overhead of ConvBN's eval mode in transfer learning. There are no foreseeable follow-up research directions here. This method technically belongs to Gradient Checkpointing (Bulo et al. (2018)), but the authors cleverly use the affine transformation of BN to escape the time cost. On the other hand, this trick can only apply to BN with eval mode.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EvXEulkoV2", "reviewer": "Reviewer_iJzH", "summary": "This paper explored the trade-off between stability and efficiency in Convolution-BatchNorm blocks which are popular convolution neural networks：Deploy mode is efficient but suffers from training instability; Eval mode is widely used in transfer learning but lacks efficiency. \nBased on detailed analysis, the paper proposed a Tune mode, which is stable and efficient, to bridge the gap between Eval mode and Deploy mode. Numerous experiments conducted in various tasks have verified the effectiveness of the proposed Tune mode.", "questions": "see weaknesses above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper provided detailed  analysis of proposed Tune mode from both theoretical and experimental perspectives.\n2. The author's writing is very good, and the entire paper is relatively easy to understand.\n3. Simple algorithm, easy to follow. (codes are available for the public according to the abstract: \"Our method has been integrated into both PyTorch (general machine learning framework) and MMCV/MMEngine (computer vision framework).\")\n4. The experimental results are reliable and sufficient to verify the effectiveness of this method.", "weaknesses": "1. the proposed Tune mode looks like an engineering technique. (this method might be more suitable for patent applications) \n2. The novelty of this method is somewhat weak for top-tie conferences focused on theoretical research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)