Network Augmentation for Tiny Deep Learning
Authors: Han Cai
Chuang Gan
Ji Lin
Song Han
Venue: ICLR 2022 Poster (2022)
Keywords: Tiny Deep Learning

Abstract: We introduce Network Augmentation (NetAug), a new training method for improving the performance of tiny neural networks. Existing regularization techniques (e.g., data augmentation, dropout) have shown much success on large neural networks by adding noise to overcome over-fitting. However, we found these techniques hurt the performance of tiny neural networks. We argue that training tiny models are different from large models: rather than augmenting the data, we should augment the model, since tiny models tend to suffer from under-fitting rather than over-fitting due to limited capacity. To alleviate this issue, NetAug augments the network (reverse dropout) instead of inserting noise into the dataset or the network. It puts the tiny model into larger models and encourages it to work as a sub-model of larger models to get extra supervision, in addition to functioning as an independent model. At test time, only the tiny model is used for inference, incurring zero inference overhead. We demonstrate the effectiveness of NetAug on image classification and object detection. NetAug consistently improves the performance of tiny models, achieving up to 2.2% accuracy improvement on ImageNet. On object detection, achieving the same level of performance, NetAug requires 41% fewer MACs on Pascal VOC and 38% fewer MACs on COCO than the baseline.

Published as a conference paper at ICLR 2022
NETWORK AUGMENTATION FOR TINY DEEP LEARNING
Han Cai', Chuang Gan Ji Lin't Song Han' Massachusetts Institute of Technology. 2MIT-IBM Watson AI Lab https: / //t /tinyyl....... mit edu
ABSTRACT
We introduce Network Augmentation (NetAug), a new training method for improv- ing the performance of tiny neural networks. Existing regularization techniques (e.g., data augmentation, dropout) have shown much success on large neural networks by adding noise to overeome over-fitting. However, we found these techniques hurt the performance of tiny neural networks. We argue that training tiny models are different from large models: rather than augmenting the data, we should augment the model, since tiny models tend to suffer from under-fitting rather than over-fitting due to limited capacity. To alleviate this issue, NetAug augments the network (reverse dropout) instead of inserting noise into the dataset or the network. It puts the tiny model into larger models and encourages to work sub-model of larger models get extra supervision, in addition to functioning as an independent model. At test time, only the tiny model is used for inference, incurring zero inference overhead. We demonstrate the effectiveness of NetAug on image classification and object detection. NetAug consistently improves the performance of tiny models, achieving up 2.2% accuracy improvement on Im- ageNet. On object detection, achieving the same level of performance, NetAug requires 41% fewer MACs on Pascal VOC and 38% fewer MACs on COCO than the baseline.
INTRODUCTION
Tiny IoT devices are witnessing rapid growth, reaching 75.44 billion by 2025 (iot). Deploying deep neural networks directly on these tiny edge devices without the need for a connection to the cloud brings better privacy and lowers the cost. However, tiny edge devices are highly resource-constrained compared to cloud devices (e.g., GPU). For example, microcontroller unit (e.g., STM32F746) typically only has 320KB of memory (Lin et al., 2020), which is 50,000x smaller than the memory of a GPU. Given such strict constraints, neural networks must be extremely small to run efficiently on these tiny edge devices. Thus, improving the performance of tiny neural networks (e.g., MCUNet (Lin et al., 2020)) has become a fundamental challenge for tiny deep learning.
Conventional approaches improve the performance of deep neural networks rely regularization techniques to alleviate over-fitting, including data augmentation methods (e.g., AutoAugment (Cubuk al., 2019), Mixup (Zhang et al., 201 8a)), dropout methods (e.g.. Dropout (Srivastava et al., 2014), DropBlock (Ghiasi et al., 2018)). and so on. Unfortunately, this common approach does not apply to tiny neural networks. Figure (left) shows the ImageNet (Deng al., 2009) accuracy of state-of- the-art regularization techniques on ResNet50 (He et al., 2016) and MobileNetV2-Ting (Lin et al., 2020). These regularization techniques significantly improve the ImageNet accuracy of ResNet50, but unfortunately, they hurt the ImageNet accuracy for MobileNetV2 Tiny, which is 174x smaller. We argue that training tiny neural networks is fundamentally different from training large neural networks. Rather than augmenting the dataset, we should augment the network. Large neural networks tend to over-fit fit the training data, and data augmentation techniques can alleviate the over- wert-fttingg issue. However, tiny neural networks tend to under-fit the training data due to limited capacity (174x smaller); applying regularization techniques to tiny neural networks will worsen the under-fitting issue and degrade the performance.

---

Published as a conference paper at ICLR 2022
Baseline
Mixup NetAug
AutoAugment
Large Neural Network
Dropblock
79
ImageNet
Ace (%6)
78
Over-fitiing
77
Data Aug
76
Dropout
ResNet50 (4.1B MACs)
174x smaller
ImageNet
Tiny Neural Network
Ace (%)
Under-fitting
50
48
Net Aug
MobileNetV2- Tiny (23.5M MACs)
Figure 1: Left: ResNer50 (large neural network) benefits from regularization techniques, while MobileNetV2-Ting (tiny neural network) losses accuracy by these regularizations. Right: Large neural networks suffer from over-fitting, thus require regularization such as data augmentation and dropout. In contrast, tiny neural networks tend to under-fit the dataset, thus requires more capacity during training. NetAug augments the network (reverse dropout) during training to provide more supervision for tiny neural networks. Contrary regularization techniques, improves the accuracy of tiny neural networks and as expected, hurts the accuracy of non-tiny neural networks.
In this paper, we propose Network Augmentation (NetAug), a new training technique for tiny deep learning. Our intuition is that tiny neural networks need more capacity rather than noise at the training time. Thus, instead of adding noise to the dataset via data augmentation or adding noise to the model via dropout (Figure upper right), NetAug augments the tiny model by inserting into larger models, sharing the weights and gradients; the tiny model becomes a sub-model of the larger models apart from working independently (Figure lower right). It can be viewed as we enlarge the target model instead of shrinking it. At the training time, NetAug adds the gradients from larger models as extra training supervision for the tiny model. At test time, only the tiny model is used for inference, causing zero overhead.
reversed form of dropout,
Extensive experiments on ImageNet (ImageNet, ImageNet-2Ik-PP and five fine-graine fine-grained image classi- fication datasets (Food101, Flowers Cars, Cub200, and Pets) show that NetAug is much more effective than regularization techniques for tiny neural networks. Applying NetAug to MobileNetV2- Tiny improves the ImageNet accuracy by 1.6% while adding only 16.7% training cost overhead and zero inference overhead. On object detection datasets, NetAug improves the AP50 of YoloV3 (Redmon & Farhadi, 2018) with Mbv3 w0.35 as the backbone by 3.36% on Pascal VOC and by 1.8% on COCO.
2 RELATED WORK
Knowledge Distillation. Knowledge distillation (KD) (Hinton et al., 2015; Furlanello et al., 2018; Yuan et al., 2020; Beyer et al., 2021; Shen et al., 2021; Yun et al., 2021) is proposed to transfer the "dark knowledge" learned in large teacher model small student model. It trains the student model to match the teacher model's output logits (Hinton et al., 2015) or intermediate activations (Romero et al., 2015; Zagoruyko & Komodakis, 2017) for better performances. Apart from being used alone, KD can be combined with other methods to improve the performance, such as (Zhou al., 2020) that combines layer-wise KD and network pruning.
Unlike KD, our method aims to improve the performances of neural networks from different perspective, i.c., tackling the under-fitting issue of tiny neural networks. Technically, our method does not require the target model to mimic teacher model. Instead, we train the target model to work as a sub-model of a set larger models, built by augmenting the width of the target model, to get extra training supervision. Since the underlying mechanism of our method is fundamentally different from KD's s. Our method is complementary to the use KD and can be combined to boost performance (Table 2).
2

---

Published as a conference paper at ICLR 2022
Regularization Methods. Regularization methods typically can be categorized into data augmen- tation families and dropout families. Data augmentation families add noise to the dataset by applying specially- -eiiiggd transformations on the imput, such as Cutout (DeVries & Taylor, 2017) and Mixup (Zhang et al., 2018a) 8a). Additionally, AutoMI. has been employed to search for a combination of transformations for data augmentation, such as AutoAugment (Cubuk et al., 2019) and RandAugment (Cubuk et al., 2020).
Instead of injecting noise into the dataset, dropout families add noise the network overcome over- fitting. A typical example is Dropout (Srivastava et al., 2014) that randomly drops connections of the neural network. Inspired by Dropout, many follow-up extensions propose structured forms of dropout for better performance. such as StochasticDepth (Huang et al., 2016), SpatialDropout (Tompson et al., 2015), and DropBlock (Ghiasi et al., 2018). addition, some regularization techniques combine dropout with other methods to improve the performance, such as Self-distillation (Zhang et al., 2019a) that combines knowledge distillation and depth dropping, and GradAug (Yang al., 2020) that combines data augmentation and channel dropping.
Unlike these regularization methods, our method targets improving the performance of tiny neural networks that suffer from under-fitting by augmenting the width of the neural network instead of shrinking it via random dropping. is reversed form of dropout. Our experiments show that NetAug more effective than regularization methods on tiny neural networks (Table 3).
Tiny Deep Learning. Improving the inference efficiency of neural networks is very important in tiny deep learning. One commonly used approach is to compress existing neural networks by pruning (Han et al., 2015; He et al., 2017; Liu et al., 2017) and quantization (Han et al., 2016; Zhu al., 2017; Rastegari et al., 2016). Another widely adopted approach is to design efficient neural network architectures (landola et al., 2016; Sandler et al., 2018; Zhang et al., 2018b). In addition to manually designed compression strategies and neural network architectures, AutoML techniques recently gain popularity in tiny deep learning, including auto model compression (Cai et al., 2019a; Yu & Huang, 2019) and auto neural network architecture design (Tan et al., 2019; Cai et al., 2019b; Wu et al., 2019). Unlike these techniques, our method focuses improving the accuracy of tiny neural networks without changing the model architecture. Combining these techniques with our method leads better performances in our experiments (Table 1).
3 NETWORK AUGMENTATION
In this section, we first describe the formulation of NetAug. Then we introduce practical implementa- tions. Lastly, we discuss the overhead of NetAug during training (16.7%) and test (zero).
3.1 FORMULATION
We denote the weights of the tiny neural network as W, and the loss function as During training, 3C(W(") We is optimized to minimize L with gradient updates: W = WT - 77 aw? where is the learning rate, and assume using standard stochastic gradient descent for simplicity. Since the capacity of the tiny neural network is limited, it is more likely to get stuck in local minimums than large neural networks, leading worse training and test performances.
We aim to tackle this challenge by introducing additional supervision to assist the training of the tiny neural network. Contrary dropout methods that encourage subsets of the neural network to produce predictions, NetAug encourages the tiny neural network to work as a sub-model of a set of larger models constructed by augmenting the width of the tiny model (Figure 2 left). The augmented loss function Caug is:
Laug
C(W))
CWW, Wil) + ... a; C(W,, Wi) + +...,
base supervision
muxiliary supervision, working as sub model of augmented models
where [W,, W.J represents an augmented model that contains the tiny neural network W, and new weights W. og is the scaling hyper-parameter for combining loss from different augmented models.
3

---

Published as a conference paper at ICLR 2022
Input
Step1
Step2
Block
1x1 Conv
8 Shuse
Baux
Depthwise Conv
Block ne1
Smnc
Song
1x1 Conv
flow
supervision Auxilarry supervision
Output
Auxiiaary forward flow
Augment Width Multiplier
Augment Expand Ratio
Figure 2: Left: We augment a tiny network by putting it into larger neural networks. They share the weights. The tiny neural network is supervised to produce useful representations for larger neural networks beyond functioning independently. At each training step, we sample one augmented network provide auxiliary supervision that is added to the base supervision. At test time, only the tiny network is used for inference, which has zero overhead. Right: NetAug is implemented by augmenting the width multiplier and expand ratio of the tiny network.
3.2 IMPLEMENTATION
Constructing Augmented Models. Keeping the weights of each augmented model (e.g., [W, W:) independent resource- proouiiieppiiiiiivee, as the model size grows linearly the number of augmented models increases. Therefore, we share the weights of different augmented models, and only maintain the largest augmented model. We construct other augmented models by selecting the sub-networks from the largest augmented model (Figure left).
This weight-sharing strategy is also used in one-shot neural architecture search (NAS) (Guo et al., 2020; Cai et al., 2020a) and multi-task learning (Ruder, 2017). Our objective and training process completely different from theirs: i) one-shot NAS trains a weight-sharing super-net that supports all possible sub-networks, Its goal is provide efficient performance estimation in NAS. In contrast, NetAug focuses on improving the performance of tiny neural network by utilizing auxiliary supervision from augmented models. In addition, NetAug can be applied to NAS-designed neural networks for better performances (Table 1). ii) Multi-task learning aims to transfer knowledge across different tasks via weight sharing. In contrast, NetAug transmits auxiliary supervision on a single task, from augmented models to the tiny model.
Specifically, construct the largest augmented model by augmenting the width (Figure 2 right), which incurs smaller training time overhead on GPUs than augmenting the depth (Radosavovic et al., 2020). For example, assume the width of a convolution operation is w, we augment its width by an angmentation factor r. Then the width of the largest augmented convolution operation is r x w. For simplicity, we use single hyper-parameter to control the augmentution factor for all operators the network.
After building the largest augmented model, we construct other augmented models by selecting a subset of channels from the largest augmented model. We use syper-parameter s, named diversity factor, to control the number of augmented model configurations. We set the augmented widths to be linearly spaced between w and x For instance, with 3 and 2, the possible widths are [w, 2w, 3w). Different layers can use different augmentation ratios. In this way, we get diverse augmented models from the largest augmented model, each containing the target neural network.
Training Process. As shown in Eq. I, getting supervision from one augmented network requires an additional forward and backward process. is computationally expensive involve all augmented networks in one training step. To address this challenge, we only sample one augmented network at each step. The tiny neural network is updated by merging the base supervision (i.e., #C(W?)) aW.
and the
auxiliary supervision from this sampled augmented network (Figure left):
W W n OC(WT) OC(Wr. WF])
(2)
OW?
OW?
4

---

Published as a conference paper at ICLR 2022
where [W7', WT"| represents the sampled augmented network this training step. For simplicity, use the same scaling In addition, W; is also updated via gradient descent in this training step. It is possible to sample more augmented networks in one training step. However, in our experiments, we found it not only increases the training cost but also hurts the performance. Thus, only sample one augmented network in each training step.
a (a = 1.0 in our experiments) for all augmented networks.
3.3 TRAINING AND INFERENCE OVERHEAD
NetAug is only applied at the training time. At inference time, we only keep the tiny neural network. Therefore, the inference overhead of NetAug is zero. In addition, as NetAug does not change the network architecture, it does not require special support from the software system or hardware. making it easier to deploy in practice.
Regarding the training overhead, applying NetAug adds an extra forward and backward process in each training step, which seems to double the training cost. However, in our experiments, the training time is only 16.7% longer (245 GPU hours v.s. 210 GPU hours, shown Table 3). It because the total training cost of a tiny neural network is dominated by data loading and communication cost, not the forward and backward computation, since the model is very small. Therefore, the overall training time overhead of NetAug is only 16.7%. Apart from the training cost, applying NetAug will increase the peak training memory footprint. Since we focus on training tiny neural networks whose peak training memory footprint is much smaller than large neural networks', in practice, the slightly increased training memory footprint can still fit in GPUs.
EXPERIMENTS
4.1 SETUP
Datasets. We conducted experiments seven image classification datasets, including ImageNet (Deng et al., 2009), ImageNet-21K-2 (winter21 version) (Ridnik et al., 2021), Food101 (Bossard al., 2014), Flowers (Nilsback & Zisserman, 2008), Cars (Krause et al., 2013), Cub200 (Wah et al., 2011) I). and Pets (Parkhi et al., 2012). In addition to image classification, we also evaluated our method on Pascal VOC object detection (Everingham al., 2010) and COCO object detection (Lin et al., 2014)".
Training Details. For ImageNet experiments, we train models with batch size 2048 using 16 GPUs. We use the SGD optimizer with Nesterov momentum 0.9 and weight decay 4e-5, By default, the models are trained for 150 epochs on ImageNet and 20 epochs on K-P except stated explicitly. The initial learning rate is 0.4 and gradually decreases following the cosine schedule. Label smoothing is used with a factor of 0.1 on ImageNet.
For experiments on fine- ane-grained image classification datasets (Food101, Flowers 102, Cars, Cub200, and Pets), we train models with batch size 256 using 4 GPUs. We use ImageNet-pretrained weights to initialize the models and finetune the models for 50 epochs.
For Pascal VOC object detection, we train models for 200 epochs with batch size 64 using 8 GPUs. The training set consists of Pascal VOC 2007 trainval set and Pascal VOC 2012 trainval set, while Pascal VOC 2007 test set is used for testing. For COCO object detection, we train models for 120 epochs with batch size 128 using 16 GPUs. COCO2017 train is used for training while COC02017 val is used for testing.
We use the YoloV3 (Redmon & Farhadi, 2018) detection framework and replace the backbone with tiny neural networks. We also replace normal convolution operations with depthwise convolution operations in the head of Y3.. while the detection head is initialized randomly.
weights are used to initialize the backbone
4.2 RESULTS ON IMAGENET
Main Results We apply NetAug to commonly used tiny neural network architectures in TLL (Lin et al., 2020; Saha et al., 2020; Banbury et al., 2021), including MobileNerV2-Ting (Lin et al.,
Code and pre-trained weights: https: /ttp://///tt////// com/mi tahhn-hl- 1ab/tinbbttta
5

---

Published as a conference paper at ICLR 2022
MobileNetV2 -Tny,, r144
MCUNet r176
MobileNetV3 r160, w0.35
ProxylesSNAS r160
MobileNerV2 r160
ResNet50
Model
w0.35
w1.0
w0.35
w1.0
r224
Params
0.75M
0.74M
2.2M
1.8M
4.1M
1.7M
3.5M
25,5M
MACs
23.5M
81.8M
19.6M
35.7M
164.1M
30.9M
154.
4.1G
Baseline 51.7%
61.5%
58.1%
59.1%
71.2%
56.3%
69.7%
76.8%
NetAug 53.3%
62.7%
60.3%
60.8%
71.9%
57.8%
70.6%
76.5%
AAcc +1.6%
+1.2%
+2.2%
+1.7%
+0.7%
+1.5%
+0.9%
-0.3%
Table I: NetAug consistently improves the ImageNet accuracy for popular tiny neural networks. The smaller the model, the larger the improvement. 'w" represents the width multiplier and 'r' represents the input image size.
55
26
ImageNet Top1 (%)
54
25
75% epochs
Top1 (%)
83% epochs
NetAug Baseline
NetAug
23
Baseline
50
22
600 # Training Epochs
300
750
40
120
# Training Epochs
Figure 3: NetAug outperforms the baseline under different numbers of training epochs on ImageNet and ImageNet-21 M- for MobileNetV2-Tinng With similar accuracy, NetAug requires 75% fewer training epochs on ImageNet and 83% fewer training epochs on ImageNet-21K-P.
2020), ProxylesNNAS-Mohins (Cai et al., 2019b), MCUNet (Lin et al., 2020), MobileNetV3 (Howard al., 2019), and MobileNetV2 (Sandler al., 2018). As shown in Table 1, NetAug provides consistent accuracy improvements over the baselines on different neural architectures. Specifically, for TroxylessNAS-Moblle MCUNet and MobileNetV3, whose architectures NAS, NetAug still provides significant accuracy improvements (1.7% for TroyllesNAAS-Mohhe 1.2% for MCUNet, and 2.2% for MobileNetV3).. In addition, we find NetAug tends to provide higher accuracy improvement on smaller neural networks (+0.9% on MobileNetV2 w1.0 + +1.5% on MobileNetV2 w0.35). We conjecture that smaller neural networks have lower capacity, thus suffer more from the under-fitting issue and benefits more from NetAug. Unsurprisingly NetAug hurts the accuracy of non-tiny neural network (ResNet50), which already has enough model capacity on ImageNet.
optimized using
Figure summarizes the results of MobileNerV22-Tnn on ImageNet and ImageNet-21I-4 under different numbers of training epochs. NetAug provides consistent accuracy improvements the baseline under all settings. fewer training epochs than the baseline (75% fewer epochs on ImageNet, 83% fewer epochs Mnag which can save the training cost and reduce the CO2 emissions.
addition, to achieve similar accuracy, NetAug requires much
Comparison with KD. We compare the ImageNet performances of NetAug and knowledge distilla- tion (KD) on MobileNetV2- (L.in et al., 2020), MobileNetV2, MobileNetV3, ProxylessNAS. All models use the same teacher model Mesemble--ResNt550 (Lee et al., 2020)) when training with KD.
The results summarized in Table 2. Compared with KD, NetAug provides slightly higher ImageNet accuracy improvements: +0.5% on MobileNetV2- +0.8% on MobileNetV2 (w0.35, r160), +0.5% on MobileNetV3 (w0.35, r160), and +0.4% ProxylessNAS (w0.35, r160). In addition, we find NetAug's improvement is orthogonal to KD's, Combining NetAug and KD can further boost the ImageNet accuracy of tiny neural networks: 2.7% MobileNerV2-Tiny; 2.9%

---

Published as a conference paper at ICLR 2022
Model
Baseline
KD
NetAug
NetAug + KD
MobileNerV2-Tiime MobileNetV2 w0.35, r160
51.7%
52.8% (+1.1%)
53.3% (+1.6%)
54.4% (+2.7%)
56.3%
57.0% (+0.7%)
57.8% (+1.5%)
59.2% (+2.9%)
MobileNerV33 w0.35, r160
58.1 %
59.8%
(+1.7%)
60.3% (+2.2%)
61.5%
(+3.4%)
ProxylessNAS w0.35, r160
59.1%
60.4% (+1.3%)
60.8% (+1.7%
61.5% (+2.4%)
Table 2: Comparison with KD (Hinton et al., 2015) on ImageNet. NetAug is orthogonal to KD. Combining NetAug with KD further boosts the ImageNet accuracy of tiny neural networks.
Training Cost
ImageNet Top1 Acc AAce
Model (MobileNetV2-inny)
#Epochs
(GPU Hours)
Baseline
150
210
51.7%
Dropout (kp=0.9) (Srivastava et al., 2014)
150
210
51.0%
-0.7%
Dropout (kp=0.8) (Srivastava al., 2014)
150
210
50.3%
-1.4%
Baseline
300
420
52.4%
Mixup (Zhang et al., 2018a)
300
420
51.7%
-0.7%
AutoAugment (Cubuk et al., 2019)
300
440
51.0%
-1.4%
Randent (Cubuk al., 2020)
300
440
49.6%
-2.8%
DropBlock (Ghiasi et al., 2018)
300
420
48.7%
-3.7%
NetAug
300
490
53.7%
+1.3%
Table 3: Regularization techniques hurt the accuracy for MobileNetV2-Tingg while NetAug provides 1.3% topl accuracy improvement with only 16.7% training cost overhead.
on MobileNetV2 (w0.35, r160), 3.4% on MobileNetV3 (w0.35, and 2.4% on ProxylessNAS (w0.35, r160).
Comparison with Regularization Methods. Regularization techniques hurt the performance of tiny neural network (Table 3), even when the regularization strength very weak (e.g., dropout with keep probability 0.9). This is due to tiny networks has very limited model capacity. When adding stronger regularization (e.g., RandAugment, Dropblock), the accuracy loss gets larger (up to 3.7% accuracy loss). Additionally, we notice that Mixup, Dropblock, and RandAugment provide pperrpprraeetes to adjust the strength of regularization. We further studied these methods under different regularization strengths. The results reported in the appendix (Table 6) due the space limit. Similar to observations Table 3, we consistently find that: i) adding these regularization methods hurts the accuracy of the tiny neural network. ii) Increasing the regularization strength leads to higher accuracy loss.
Based on these results, conjecture that tiny neural networks suffer from the rather than the over-fitting issue. Applying regularization techniques designed to overcome over- fitting will exacerbate the under-fitting issue of tiny neural networks, thereby leading accuracy loss. In contrast, applying NetAug improves the ImageNet accuracy of MobileNetV2-TNe with zero inference overhead and only 16.7% training cost overhead. NetAug is more effective than regularization techniques in tiny deep learning.
issue
by 1.3%,
Discussion. NetAug improves the accuracy of tiny neural networks by alleviating under-fitting. However, for larger neural networks that do not suffer from under-fitting, applying NetAug may, on the contrary, exacerbate over-fitting, leading to degraded validation accuracy (as shown in Table I, ResNet50), We verify the idea by plotting the training and validation curves of tiny network (MobileNetV2-Timgg and a large network (ResNet50) in Figure 4, with and without applying NetAug. Applying NetAug improves the training and validation accuracy of MobileNetV2-1TT 2-Tyy,, demonstrating that NetAug effectively reduces the mnder-fitting issue. For ResNet50, NetAug improves the training accuracy while lowers the validation accuracy, showing signs of overfitting.
7

---

Published as a conference paper at ICLR 2022
MbV2-Tiny (train)
MbV2-Tiny (val)
ResNet50 (train)
ReaNet50 (val)
50
85
Top1 Acc (%)
48
+1.3%
1.6%%
Top1 Acc (%)
81
+ 16%
0 30%
46
51
78
75
44
NetAug
NetAug
74
NetAug Baseline
NetAug
Baseline
Baseline
Baseline
42
70
100
110
120
130 140 Epoch
150
100 110
120
130
140 150
50
70
80 Epoch
90
100
50
60
80 Epoch
90
100
Epoch
Figure 4: Learning curves on ImageNet. Left: NetAug alleviates the under-fitting issue of tiny neural networks (c.g., MobileNetV2-Tinys leading higher training and validation accuracy. Right: Larger networks like ResNet50 does not suffer from under-fitting; applying NetAug will exacerbate over-fitting (higher training accuracy, lower validation accuracy).
ImageNet Topl (%)
Fine-grained Classificaition: Top1 (%)
Det:
AP50 (%)
Method
Food101
Flowers 102
Cars
Cub200
Pets
VOC
COCO
Baseline (150)
56.3
76.4
90.8
76.9
69.0
81.9
60.4
24.7
Baseline (300)
57.0
76.5
90.3
75.8
69.6
81.7
60.8
Baseline (600)
57.5
76.8
89.7
74.7
69.5
81.7
61.3
MbV2
w0.35
KD
57.0
76.4
91.6
78.4
68.4
80.3
60.0
24.3
r6
NetAug
57.8
77.4
92.4
79.8
68.8
82.3
62.4
25,4
NetAug+KD
59.2
77.5
92.9
80.4
68.5
82.2
62.1
25.4
Baseline (150)
58.1
76.6
92.1
75.8
69.3
83.1
63.6
29.2
MbV3
KD
59.8
76.9
92.6
77.2
69.2
82.9
63.4
28.9
w0.35
NetAug
60.3
78.3
93.0
78.9
70.4
84.6
65.3
30.7
r160
NetAug+KD
61.5
77.1
93.0
78.6
70.5
84.5
66.0
30.8
Table 4: Transfer learning results of MobileNetV2 (w0.35, r160) and MobileNetV3 (w0.35, r160) with different pre-training methods. In most cases, models pre- te-itrained with NetAug provide the best transfer learning performance on fine-grained classification and object detection. Results that are worse than the 'Baseline (150)' in red, and results that are better than the 'Baseline (150)* are in green. Best results are highlighted in bold.
4.3 RESULTS ON TRANSFER LEARNING
Models pre-trained on ImageNet are usually used for initialization in downstream tasks such as fine-grained image classification (Cui et al., 2018 8; Kornblith al., 2019; Cai et al., 2020b) and object detection (Everingham et al., 2010; Lin et al., 2014). In this subsection, we study whether NetAug can benefit these downstream tasks using five fine-grained image classification datasets and two object detection datasets. The input image size is 160 for fine- tine-grained image classification and 416 for object detection.
The transfer learning results of MobileNetV2 w0.35 and MobileNetV3 w0.35 with different pre- trained weights are summarized Table We find that higher accuracy on the pre-training dataset (i.e., ImageNet in our case) does not always lead to higher performances on downstream tasks. For example, though adding KD improves the ImageNet accuracy of MobileNetV2 w0.35 and MobileNetV3 w0.35, using weights pre-trained with KD hurts the performances two fine- grained classification datasets (Cub200 and Pets) and all object detection datasets (Pascal VOC and COCO). Similarly, training models for epochs significantly improves the ImageNet accuracy of MobileNetV2 w0.35 but hurts the performances on three fine- mee-grained classification datasets.
Compared to KD and training for more epochs, we find models pre-trained with NetAug achieve clearly better transfer learning performances in most cases, though their ImageNet performances are similar. It shows that encouraging the tiny models work sub-model of larger models not only
8

---

Published as a conference paper at ICLR 2022
Fine-grained Classificaition: Topl (%)
Method
Food101
Hlwerss022
Cars
Cub200
Pets
MbV2
Baseline (150)
67.33
89.04
58.28
57.75
78.88
w0.35 r160
NetAug
68.67
90.40
60.02
58.39
79.37
MbV3
Baseline (150)
67.53
89.19
51.18
57.99
80.32
w0.35 r160
NetAug
69.74
90.73
56.21
58.94
82.04
Table 5: NetAug also benefits the tiny transfer learning (Cai al., 2020b) setting where pre-trained weights are frozen to reduce training memory footprint.
Pascal (AP50)
COCO (AP50)
64
28
YoloV3+MbV2
VoloVS-MoV3
Yolok3+MDV2
YolV3-MBVV
62
41% MACs
25
38% MACS
30
24
58
Baseline Mixup
NetAug
.5.8%% MAP
NetAug Baseline
23
NetAug Baseline
NetAug Baseline
Mixup
54
53
22
90
150
210
270
60
120
180
240
150
200
250
300
350
400
100
170 240
310 380 450 (M)
(M)
(M)
(M)
Figure 5: On Pascal VOC and COCO, models pre-trained with NetAug achieve a better performance- efficiency trade-off. Similar ImageNet classification, adding detection mixup (Zhang et al., 2019b) that is effective for large neural networks causes performance drop for tiny neural networks.
improves the ImageNet accuracy of the models but also improves the quality of learned representation. In addition to the normal transfer learning setting, we also test the effectiveness NetAug under the tiny transfer learning (Cai et al., 2020b) setting where the pre-trained weights are frozen while only updating the biases and additional lite residual modules to reduce the training memory footprint. As shown in Table 5, NetAug also benefit tiny transfer learning, providing consistently accuracy improvements over the baseline.
Apart from improving the performance, NetAug can also be applied for better inference efficiency. Figure 5 demonstrates the results YoloV3+MobileNettt w0.35 and Yolo Moboileebil w0.35 under different imput resolutions. Achieving similar AP50, NetAug requires a smaller input resolution than the baseline, leading to 41% % MACs reduction on Pascal VOC and 38% MACs reduction on COCO. Meanwhile, a smaller input resolution also reduces the inference memory footprint (Lin et al., 2020), which is also critical for running tiny deep learning models on memory-constrained devices.
Additionally, we experiment with detection mixup (Zhang al., 2019b) on Pascal VOC. Similar to ImageNet experiments, adding detection mixup provides worse mAP than the baseline, especially on YoloV3+MobileNer22 w0.35. It shows that the under-fitting issue tiny neural networks also exists beyond image classification.
5 CONCLUSION
We propose Network Augmentation (NetAug) for improving the performance of tiny neural networks, which suffer from limited model capacity. Unlike regularization methods that aim to address the over-fitting issue for large neural networks, NetAug tackles the under-fittie fitting problem of tiny neural networks. This is achieved by putting the target tiny neural network into larger neural networks to get auxiliary supervision during training. Extensive experiments on image classification and object detection consistently demonstrate the effectiveness of NetAug on tiny neural networks.
ACKNOWLEDGMENTS
We thank National Science Foundation, MIT-IBM Watson AI Lab, Hyundai, Ford, Intel and Amazon for supporting this research.

---

Published as a conference paper at ICLR 2022
REFERENCES
Internet of things (IoT) connected
worldwide from https://wwwk statista, com/ httm//sttss.cs//122//...
devices installed base
2015 to
2025.
Colby Banbury, Chuteng Zhou, Igor Fedorov, Ramon Matas, Urmish Thakker, Dibakar Gope, Vijay
Janapa Reddi, Matthew Mattina, and Paul Whatmough. Micronets: Neural network architectures for deploying tinyml applications on commodity microcontrollers. Proceedings of Machine Learning and Systems, 3, 2021. 5
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. arXiv preprint arXiv:2106.052.7
2021. 2
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-10l-mining discriminative compo- nents with random forests. In ECCV, 2014.
Han Cai, Ji Lin, Yujun Lin, Zhijian Liu, Kuan Wang, Tianzhe Wang. Ligeng Zhu, and Song Han. Automl for architecting efficient and specialized neural networks. IEEE Micro, 2019a. 3
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architcccturre search on target task and hardware. In ICLR, 2019b. URL https::/// http://axxvvv.i attpa//www....////////////2 00332 pdf. 3,
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once for all: Train one network and specialize it for efficient deployment. In ICLR, 2020a. URL https: /arxivv.iiv peg/pdd / 1908. 01..999911 pdf. 4
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems, volume 33, pp. 11285-11297, 2020b. 8,
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. In CVPR, 2019. 1,
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with a reduced search space. In NeurlPS, volume 33, 2020. 3, 7,
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained
categorization and domain-specific transfer learning. In CVPR, 2018. 8
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 1,
Terrance Vrics and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 17xiv:1708.04552 2017.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 88(2):303-338 2010. 8,8
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning. pp. 1607-1616.
PMLR, 2018. 2
Golnaz Ghiasi, Tsung- Yi Lin, and Quoe V Le. Dropblock: regularization method for convolutional networks. In NeurIPS, 2018. 3,
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. ECCV, pp. 544-560. Springer, 2020.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NeurIPS, 2015.
10

---

Published as a conference paper at ICLR 2022
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR, 2016. 3
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In ICCV, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0253: 2015. 7,
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In ICCV, 2019. 6
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. ECCV, 2016.
Forrest N landola, Song Han, Matthew Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint arXiv:1602.07366 2016. 3
Simon Korbblith, Jonathon Shlens, and Quoc Le. Do better imagenet models transfer better? In CVPR, 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, 2013. 5
Jungkyu Lee, Taeryun Won, Tae Kwan Lee, Hyemin Lee, Geonmo Gu, and Kiho Hong, Compounding the performance improvements of assembled techniques a convolutional neural network. arXiv preprint arXiv:200/.0622 2020. 6
Ji Lin, Wei-Ming Chen, John Cohn, Chuang Gan, and Song Han. Mcunet: Tiny deep learning iot devices. In NeurIPS, 2020. 1, 5, 6,
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014. 5, 8
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, 2017.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification of classes. In Sixth Indian Conference on Computer Vision, Graphics & Image Processing, 2008.
a large number
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. CVPR, 2012. 5
Ilija Radosavovic, Raj Prateck Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020. 4
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, pp. 525-542. Springer, 2016.
Joseph Redmon and Ali Farhadi. 17X.v.1804.02767. 2018. 2,5
Yolov3: An incremental improvement. arXiv preprint
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelni.-Manor. Imagenet- 22l pretraining for the masses. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL httpt:::/pppeeeeeeeeeee wwwp///eeeeeeeeeo//////// net /ww//eertonteeeeee fformm idd=kk __
Vc76o1.
11

---

Published as a conference paper at ICLR 2022
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015. 2
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint
TrXiv:1706.050900 2017. 4
Oindrila Saha, Aditya Kusupati, Harsha Vardhan Simhadri, Manik Varma, and Prateck Jain. Rnnpool: Efficient non-linear pooling for ram constrained inference. arXiv preprint 1.X.::2002.119211. 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chies Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. CVPR, 2018. 3,6
Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang-Ting Cheng, and Marios Savvides. label smoothing truly incompatible with knowledge distillation: An empirical study. In International Conference on Learning Representations, 2021. 2
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. JMLR, 15(1):1929-199)))) 195199999989,,---111111111189988 2014. 1,
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In CVPR, 2019. 3
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. Efficient object localization using convolutional networks. In CVPR, pp. 648-656, 2015.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-200 dataset. 2011.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-awar efficient convnet design via differentiable neural architecture search. In CVPR, 2019. 3
Taojiannan Yang, Sijie Zhu, and Chen Chen. Gradaug: A new regularization method for deep neural networks. In NeurIPS, 2020.
Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint ar 1xXv::103.11725. 2019. 3
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3903-3911, 2020. 2
Sangdoo Yun, Scong Joon Oh, Bycongho Hco, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. Re-labeling imagenet: from single to multi-labels, from global to localized labels. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2340-2350, 2021.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond em- pirical risk minimization. In ICLR, 2018a. URL https:://////..... httttp:://opnnevieew..t net wot./formmmmmm
r1Dép1-Rb. Rb. 3, 7,
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In ICCV, 2019a. 3
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018b. 3
Zhi Zhang, Tong He, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of freebies for training object detection neural networks. arXiv preprint 100.04490..04033 2019b.
12

---

Published as a conference paper at ICLR 2022
Denny Zhou, Mao Ye, Chen Chen, Tianjian Meng, Mingxing Tan, Xiaodan Song, Quoe Le, Qiang Liu, and Dale Schuurmans. Go wide, then narrow: Efficient training of deep thin networks. In International Conference Machine Learning, pp- 11546-11555. PMLR, 2020. 2
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. In ICLR. 2017. 3
A
ABLATION STUDY ON REGULARIZATION METHODS
ImageNet Topl Acc AAco
Model (MobileNetV2-Tiny)
Baseline
52.4%
Mixup (a=0.1) (Zhang al., 2018a)
52.2%
-0.2%
Mixup (a=0.2) (Zhang et al., 2018a)
51.7%
-0.7%
DropBlock (kp=0.95, block size=5) (Ghiasi et al., 2018)
50.6%
-1.8%
DropBlock (kp=0.9, block size=5) (Ghiasi et al., 2018)
48.7%
-3.7%
RandAugment (N=1, M=9) (Cubuk al., 2020)
51.5%
-0.9%
RandAugment (N=2, M=9) (Cubuk et al., 2020)
49.6%
-2.8%
NetAug
53.7%
+1.3%
Table 6: Ablation study on regularization methods. All models are trained for 300 epochs on ImageNet.
For DropBlock, we adopted the implementation from https://githuu www/mimmmmmmocccoemcco guelvr/ dropblock. Additionally, block size=7 is not applicable on MobileNetV2-Tinn, because the feature map size the last stage of MobileNetV2-Tiny is 5. For RandAugment, we adopted the implementation from https: : /tt//gtttii //gtttub...g com rwi httpt/aaappyttcccchhhee image-modes monoels.. 1s.
As shown in Table increasing the regularization strength results in a higher accuracy loss for the tiny neural network. Removing regularization provides better result.
B ABLATION STUDY ON TRAINING SETTINGS
In addition to NetAug, there several simple techniques that potentially alleviate the under- fitting issue of tiny neural networks, including using smaller weight decay, using weaker data augmentation, and using smaller batch size. However, as shown in Table 7, they not effective compared with NetAug on ImageNet.
Model (MobileNetV2--inn))
ImageNet Top1 Acc
Baseline
51.7%
Smaller Weight Decay (1e-5)
51.6%
Smaller Weight Decay (1e-6)
51.3%
Weaker Data Augmentation (No Aug)
50.8%
Smaller Batch Size (1024)
51.5%
Smaller Batch Size (256)
51.7%
NetAug
53.3%
Table Ablation study on training settings. All models are trained for 150 epochs on ImageNet.
13


Reviews:
[{"review_id": "tLPFIoxqtOJ", "reviewer": "Reviewer_co48", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n\n1) The authors show that large models benefit from data augmentation and dropout while tiny models suffer from these regularizations/augmentations.\n\n2) Augmenting the network width to improve the trainability of tiny models is an interesting approach.\n\n3) Extensive experiments on ImageNet, Pascal VOC and several down-stream tasks show the superiority of the proposed method.\n\n4) Ablation studies show that the proposed method can be combined with KD and pruning to get better results.\n\n[Weaknesses]\n\n1) The proposed method is similar to progressive shrinking used in OFANet in principle.\n\n2) Augmenting the width of the network in training  has been proposed before, such as \"Go Wide, Then Narrow: Efficient Training of Deep Thin Networks. ICML 2020.\". It would be better to discuss the key differences.\n", "summary_of_the_paper": "This work proposes a novel method to train tiny networks. The proposed method first augments the width of the tiny network to make the tiny network a bit larger. Then the gradients from the larger network are used as additional supervision. The authors show that the proposed method significantly improves the generalization performance of tiny models, is complementary to knowledge distillation and pruning. In addition, the authors show that data augmentation and dropout or similar regularization methods actually hurt the tiny models in training.", "summary_of_the_review": "While there are lots of works studying how to improve the accuracy of large models, there are relatively fewer works focusing on the tiny network training. This work demonstrates that the tiny models suffer from under-fitting rather than over-fitting therefore requires different training strategies. The proposed method, NetAug, is effective and simple to implement. It also works well with other techniques such as knowledge distillation and pruning. There are existing works sharing a similar idea of NetAug for large model training, which slightly hurts the novelty of this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_SHnmIPlImC", "reviewer": "Reviewer_PXn6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Generally, I like the idea proposed in this paper. It is clear, somewhat interesting and looks reasonable. While, it is essentially similar to the one-shot NAS with weight sharing method, even the authors emphasized that they are different.\n\nIn the weight sharing training, the sub-network can also be augmented by auxiliary supervision across different iterations, it seems this proposed method just aggregates the gradients from auxiliary supervisions (from larger models) in a single iteration training, more like a special case of the former.\n\n**Some of my other concerns are as follows:**\n\n***About novelty:***\n\nAs I mentioned above, the overall idea and implementation strategy are both similar to the weight sharing mechanism.\n\nAlso, only considering width dimension without depth in this paper is too limited for the method to be deployed on real-world applications.\n\n***About experiments:***\n\nIn the paper, the authors claimed MobileNetV2-Tiny is a tiny model so the method is effective and ResNet-50 is a large one so the method is harmful, this may be true from the perspective when compared to some compact networks. But if we consider efficientnet_l2, vit_large, GPT3, etc., ResNet-50 is also a *tiny* model in this case, it seems the proposed method only works well on the extremely low-level performance families, basically, these extremely tiny models may not have fully converged so they are relatively easy for improvement. I suggest the authors can try to train the tiny models with more epochs like 600 and better optimizers like AdamW to make sure the model has completely converged (a strong baseline), then check whether the proposed method is still effective in this case.\n\n***About KD:***\n\nFirst, I personally don’t agree with the claims in Sec. 2 of the paper that “Recent studies (Furlanello et al., 2018; Cho & Hariharan, 2019; Yuan et al., 2020) show that the teacher model does not necessarily need to be larger than the student model in KD.” These papers claimed the teacher scale is not necessary but there are also a large number of literatures stating that larger models can provide more precise supervision and are crucial for better distillation, such as: A good teacher is patient and consistent [1], Is label smoothing truly incompatible with knowledge distillation [2] ReLabel ImageNet [3], etc.\n\nBasically, this is not a well-explored problem and in different scenarios the requirements for the teachers in KD are definitely different, so it seems the claims in this paper are too arbitrary and not well motivated. I just went through all the mentioned related papers, it seems the performance of the models in these papers is still at a very low level and they mainly tested on small datasets like CIFAR, which means the models may not yet fully converge. I don’t think the derived conclusions from these papers are more solid than [1] which has true state-of-the-art accuracy on the large-scale ImageNet.\n\n[1] Lucas Beyer, Xiaohua Zhai, Am ́elie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. arXiv preprintarXiv: 2106.05237, 2021.\n\n[2] Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang-Ting  Cheng, and Marios Savvides. Is label smoothing truly incompatible with knowledge distillation: An empirical study. In ICLR, 2021.\n\n[3] Sangdoo Yun, Seong Joon Oh,  Byeongho Heo,  DongyoonHan, Junsuk Choe, and Sanghyuk  Chun. Re-labeling imagenet: from single to multi-labels, from global to localized labels. In CVPR, 2021.\n\nFurthermore, KD seems not related to the proposed method, I’m a little bit confused why the authors spend a lot of space to introduce KD in the related work and further involve it into all the experiments. To my perspective, some statements are even not accurate and also not yet verified.\n\nKD brings the significant contribution of improvement, but it will also involve much uncertainty and vagueness of the improvement. I think it's better for the authors to focus on the proposed method itself in the experiments. Considering the not very accurate statements for the KD introduction, I’m concerned about the solidness of the experiments in this paper.\n", "summary_of_the_paper": "This paper proposed to train and augment a tiny network by incorporating it into the larger networks with weight sharing training mechanism. The tiny neural network is learned with the auxiliary/additional supervisions from the larger models that wrap it. With this training strategy, the tiny model can perform better than the conventional training scheme on ImageNet and several downstream tasks.", "summary_of_the_review": "I think this is an interesting paper, but I still have a few concerns stated above, especially on the experimental part and some statements on knowledge distillation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "DopuMlxYb5g", "reviewer": "Reviewer_XSG2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduced a training mechanism that includes auxiliary forward flow and supervision from the very training of the very large model into the training of the tiny nets, this training mechanism improves the classification and object detection performance of tiny nets on several large datasets. The paper does not provide evidence that if a small sub-network of the larger network which is equivalent to the tiny network in network sizes will provide the same results as shown in the paper, in other words, does the auxiliary training necessary?", "summary_of_the_paper": "To improve the performance of compact neural networks with limited model capacity, the paper proposed Network Augmentation (NetAug) which addresses the under-fitting problem in small neural networks. This is accomplished by incorporating the target tiny neural network into bigger neural networks for additional training supervision. The paper demonstrates the effectiveness of\nNetAug on improving the effectiveness and the performance of tiny models image classification and object detection on several image datasets, achieving up to 2.1% accuracy improvement on ImageNet, and 4.3% on Cars. On Pascal VOC, NetAug provides 2.96% mAP improvement with the same computational cost.", "summary_of_the_review": "As I stated in my main review, I suspect that auxiliary training is necessary for getting the better performance of the tiny network. I suspect that an equivalent sub-network of the larger net will have similar results in comparing to the performance of the tiny network solely training on the large datasets (ImageNet, Pascal VOC etc).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4aFEG7pxzWb", "reviewer": "Reviewer_ME7S", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\nS1. This paper has a clear motivation. \\\nS2. This paper is easy-to-read and well-organized. \\\nS3. The proposed method is simple, so it has broad applicability. \\\nS4. The proposed method shows consistent improvements in a variety of settings and high compatibility with other frameworks, including knowledge distillation and network pruning.\n\n### Weaknesses\nW1. How does the proposed method solve the under-fitting issue? \\\nI agree with the under-fitting issue in training tiny neural networks. However, even though augmented networks are larger than the target tiny networks, the capacity of the target network is fixed (i.e., constant). Hence, the under-fitting issue of the target network still exists and might remain unsolved. I wonder why and how the proposed method can find a better optimum.\n\nW2. Missing baselines that can avoid under-fitting. \\\nThere are several simple techniques to avoid under-fitting. For example, one can use a smaller weight decay (e.g., 1e-5, 1e-6) or weaker data augmentations. Also, using a small batch size can be another option since randomness may prevent stuck in the local optimum.\n\n### Minor comments\n- The citation formats in tables are not consistent with that in the main body.\n", "summary_of_the_paper": "When training tiny neural networks, a major issue is under-fitting. This paper proposes Network Augmentation (NetAug), which randomly augments (enlarges) neural networks during training to solve the under-fitting issue. NetAug shows consistent improvements when training tiny neural networks in various settings, including ImageNet classification, fine-grained classification, and object detection.\n", "summary_of_the_review": "I think this paper's strengths are clear, but some important parts (explanation and baselines) are missing. Hence, my initial rating is weak reject. I will raise my rating if all the issues are resolved with a strong rebuttal in the discussion period.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of training tiny networks, by proposing a new training method called Network Augmentation (NetAug). The main challenge for training tiny networks lies in underfitting, which data augmentation and dropout etc. regularizations may suffer from for tiny networks. To overcome this hurdle, the proposed method first embeds or augments the tiny network as a subnet into a larger network, mostly by enlarging the width; then the gradients from the larger network are used as additional or auxiliary supervision. With this training strategy, the tiny model can perform better than the conventional training scheme on ImageNet and several downstream tasks. The proposed method is simple to implement and complementary with other techniques such as knowledge distillation and pruning. While there are lots of works studying how to improve the accuracy of large models, there are relatively fewer works focusing on the tiny network training. Despite that there are existing works sharing a similar idea of NetAug for large model training, which slightly hurts the novelty of this work, the majority of reviewers still like the idea and suggest to accept the paper.
Decision: Accept (Poster)