Growing Efficient Accurate and Robust Neural Networks on the Edge

## Abstract

Abstract The ubiquitous deployment of deep learning systems on resource-constrained Edge devices is hindered by their high computational complexity coupled with their fragility to out-of-distribution (OOD) data, especially to naturally occurring common corruptions. Current solutions rely on the Cloud to train and compress models before deploying to the Edge. This incurs high energy and latency costs in transmitting locally acquired field data to the Cloud while also raising privacy concerns. We propose GEARnn (Growing Efficient, Accurate, and Robust neural networks) to grow and train robust networks in-situ, i.e., completely on the Edge device. Starting with a low-complexity initial backbone network, GEARnn employs One-Shot Growth (OSG) to grow a network satisfying the memory constraints of the Edge device using clean data, and robustifies the network using Efficient Robust Augmentation (ERA) to obtain the final network. We demonstrate results on a NVIDIA Jetson Xavier NX, and analyze the trade-offs between accuracy, robustness, model size, energy consumption, and training time. Our results demonstrate the construction of efficient, accurate, and robust networks entirely on an Edge device.

## 1 Introduction

The ubiquitous practical deployment of deep neural networks is mainly hindered by their lack of robustness and high computational cost. Prior art has shown that these deep networks are extremely fragile to adversarial perturbations Szegedy et al. ()Goodfellow et al. () and out-of-distribution (OOD) data Hendrycks & Dietterich ()Mintun et al. (). Natural corruptions Hendrycks & Dietterich () (a specific type of OOD data) are more commonly encountered at the Edge where real-time data is being continually acquired, e.g., video sequences acquired by on-board cameras in autonomous agents (self-driving cars, field robots, drones), which tend to be distorted by weather and blur. The state-of-the-art defense against these corruptions employs robust data augmentation Hendrycks et al. (; ); Modas et al. () which incurs a huge computational cost when implemented on an Edge device. Fig. 1 indicates that it takes more than 2 days to robustly train a VGG-19 network Simonyan & Zisserman () on a simple CIFAR-10 dataset when implemented on NVIDIA Jetson Xavier NX Edge device NVIDIA (). Even for a small 5% VGG-19 network it takes more than a day, thus highlighting the non-trivial nature of the problem. This is a huge concern because Edge devices are typically battery-powered and such large training costs reduce their operational life-time.

Traditional solutions for reducing network complexity such as pruning Han et al. (); Li et al. (); Diffenderfer et al. (), quantization Rastegari et al. (); Hubara et al. () and neural architecture search (NAS) Liu et al. (); Zoph et al. () mainly target Edge inference, and are not suited for Edge training since they start with hard-to-fit over-parameterized networks that require the large computational resources of the Cloud. However, transmitting local data to the Cloud incurs energy and latency costs while raising privacy concerns, thus requiring training to happen completely on the Edge.

Figure: Figure 1: Improvements in robust accuracy, training time, and model size (area of circles) of our proposed GEARnn method measured on NVIDIA Jetson Xavier NX Edge device NVIDIA (). Robust accuracy is evaluated on CIFAR-10-C for GEARnn, full network baselines (VGG-19), and small network baselines (5% VGG-19 networks with same topology as GEARnn-2). For robust training, we employ AugMix Hendrycks et al. (). GEARnn demonstrates significant reduction in training complexity over robust baselines at similar robust and clean accuracies (shown in Section 6.2).
Refer to caption: x1.png

Given the above challenges, the primary objective of our work is: To design and train compact robust networks completely in-situ on Edge devices. Our proposed solution GEARnn (Growing Efficient, Accurate, and Robust neural networks) is based on the family of growth algorithms Chen et al. (); Wu et al. (); Evci et al. (); Yuan et al. () that gradually increase the size of an initial backbone network to reach the robust accuracy of a full network but at a fraction of its size, training complexity, and energy consumption.

Prior work on network growth Wu et al. (; ); Yuan et al. () do not consider robustness to common corruptions since they use clean data during training, while works
that consider robustness train fixed-sized networks using augmented data Hendrycks et al. (); Modas et al. () without considering the efficiency of robust training.
Hence, in order to grow robust networks on the Edge and achieve good robustness vs. training efficiency trade-off, we ask the following questions: Q1) should networks be grown using augmented data only (1-Phase), or should they be grown using clean data first and then trained with augmented data (2-Phase)? Q2) for growth, how many steps should be employed?
We answer these questions by proposing our method GEARnn to efficiently grow robust networks. Fig. 1 shows that GEARnn achieves significant improvements in robust accuracy over vanilla trained baselines while requiring much smaller training energy consumption compared to robustly trained baselines.

Contributions: We make the following contributions (Fig. 2):

- 1.
To the best of our knowledge, our work is the first to grow networks robust to common corruptions.
Additionally, we also consider the training efficiency of our methods.

- 2.
We answer Q1 as: 2-Phase (growth with clean data followed by robust training using augmented data) provides improved robustness over a 1-Phase (growth using augmented data) at iso-model size. This result indicates the importance of proper initialization for efficient robust training (Sections 6.1, 6.2 & 6.3).
- 3.
We answer Q2 as: One-Shot Growth (OSG) achieves the best training efficiency, clean and robust accuracies at iso-model size compared to $m$-Shot ($m>1$) Growth (Section 6.3).
- 4.
We propose two GEARnn (Growing Efficient Accurate and Robust neural networks) algorithms (see Fig. 2 and Section 4.3) by combining 1-Phase/2-Phase with OSG and Efficient Robust Augmentation (ERA).
- 5.
We explain the network topologies generated during OSG, and also provide rationale for the efficacy of 2-Phase approach and initialization using clean data (Section 8).
- 6.
We show that GEARnn generated networks shine on all four metrics simultaneously – clean accuracy, robust accuracy, training efficiency and inference efficiency – by implementing them on a real-life Edge device, the NVIDIA Jetson Xavier NX
(Section 6.2).

Figure: Figure 2: Proposed approach: GEARnn-1 performs One-Shot Growth (OSG) on augmented data ($\mathcal{D}_{\text{aug}}$) generated by Efficient Robust Augmentation (ERA) (using clean data ($\mathcal{D}_{\text{in}}$)) in a single phase (1-Phase). GEARnn-2 performs OSG using $\mathcal{D}_{\text{in}}$ first followed by parametric training on $\mathcal{D}_{\text{aug}}$ in two consecutive phases (2-phase). Here $\mathcal{L}_{\text{CE}}$ and $\mathcal{L}_{\text{aug}}$ denote the cross-entropy loss and augmented loss, respectively.
Refer to caption: x2.png

## 2 Background and Related Work

Robust Data Augmentation:
This is the most commonly used method for addressing corruptions due to its ease of integration into the training flow and ability to replicate low-level structural distortions. AugMix Hendrycks et al. (), PRIME Modas et al. () and FourierMix Sun et al. () combine chains of stochastic image transforms and enforce consistency using a suitable loss function to generate an augmented sample from a clean image. DeepAugment Hendrycks et al. () randomly distorts the parameters of an image-to-image network to generate augmented images. CARDs Diffenderfer et al. () combines data augmentation Hendrycks et al. () and pruning Frankle & Carbin () to find compact robust networks embedded in large over-parameterized networks. Adversarial augmentations Zhao et al. (); Rusak et al. (); Calian et al. () have also been proposed to handle common corruptions.
Unlike our proposed GEARnn algorithm, all these techniques significantly increase the complexity over vanilla training and are thus inappropriate for Edge deployment.

Growth Techniques:
A typical growth algorithm starts with a small initial backbone model whose size is gradually increased until the desired performance or network topology is reached. Neural network growth has been previously used in optimization Fukumizu & Amari (), continual learning Rusu et al. (); Hung et al. () and in speeding up the training of large networks Chen et al. (). Recent works Evci et al. (); Yuan et al. () look at improving the training dynamics and efficiency for growth by using better neuron initializations.
Others find efficient networks by growing the width Wu et al. (), depth Wen et al. () or both Wu et al. (); Yuan et al. (). However, none of these methods address the issue of robustness to common corruptions or demonstrate the utility for training on a resource-constrained Edge setting, which is our focus. Though our work GEARnn builds upon Firefly Wu et al. (), it is flexible and can incorporate other growth methods mentioned above.

## 3 Notation and Problem Setup

Notation: Let $f:\mathbb{R}^{d}\rightarrow[C]$ be a hard classifier which classifies input $\mathbf{x}\in\mathbb{R}^{d}$ into one of $C$ classes. We choose $f$ to be a convolutional neural network (CNN) with $L$ layers (depth), $\{w_{l}\}_{l=1}^{L}$ output channels (widths), and $(K,K)$ sized kernels. The network $f$ is trained on $n$ samples $(\mathbf{x},y)\sim\mathcal{D}_{\text{in}}$, where $(\mathbf{x},y)\in\mathbb{R}^{d}\times[C]$ and $\mathcal{D}_{\text{in}}$ denotes the “in-distribution” or “clean” data. $\mathcal{L}_{\text{CE}}$ represents the cross-entropy loss and $\mathcal{L}_{\text{aug}}=\mathcal{L}_{\text{CE}}+\lambda\mathcal{L}_{\text{JSD}}$ represents the augmentation loss where $\mathcal{L}_{\text{JSD}}$ is the Jensen-Shannon divergence loss described in Hendrycks et al. ().

During inference, $f$ can be exposed to samples from both $\mathcal{D}_{\text{in}}$ and $\mathcal{D}_{\text{out}}$ (“out-of-distribution” or “corrupted” data). In case of common corruptions, $(\mathbf{x_{\text{out}}},y)\sim\mathcal{D}_{\text{out}}$ is obtained by $\mathbf{x_{\text{out}}}=\kappa(\mathbf{x_{\text{in}}},s)$, where $(\mathbf{x_{\text{in}}},y)\sim\mathcal{D}_{\text{in}}$, $\kappa$ is a corruption filter and $s$ is the severity level of the corruption. We denote $p_{e}=\text{Pr}(\hat{y}\neq y)$ as the classification error at inference where $\hat{y}=f(\mathbf{x}_{\text{test}})$. When $(\mathbf{x}_{\text{test}},y)\sim\mathcal{D}_{\text{in}}$, we define $(1-p_{e})$ as clean accuracy $\mathcal{A}_{\text{cln}}$, and when $(\mathbf{x}_{\text{test}},y)\leavevmode\nobreak\ \sim\leavevmode\nobreak\
\mathcal{D}_{\text{out}}$, we define $(1-p_{e})$ as robust accuracy $\mathcal{A}_{\text{rob}}$. The value of $p_{e}$ is determined empirically in this work.

Problem: Our primary objective is to maximize the empirical clean and robust accuracies ($\mathcal{A}_{\text{cln}}$ and $\mathcal{A}_{\text{rob}}$) while ensuring the network complexity ($\sum_{l=1}^{L}w_{l}$) is small. Along with these two criteria, we also prioritize reduction in training time ($t_{\text{tr}}$) and training energy consumption ($E$) on hardware.

## 4 Growing Efficient Accurate and Robust Neural Networks (GEARnn)

As shown in Fig. 2, two flavors of GEARnn algorithms are proposed – GEARnn-1 and GEARnn-2. While GEARnn-1 leverages the 1-Phase (joint growth and robust training) training, GEARnn-2 employs the 2-Phase (sequential growth and robust training) approach. Both flavors incorporate One-Shot Growth (OSG) and Efficient Robust Augmentation (ERA) in different ways. In this section, we first describe OSG and ERA, and then formally present the GEARnn algorithms.

### 4.1 One-Shot Growth (OSG)

One-Shot Growth (OSG) employs labeled data to perform a single growth step sandwiched between two training stages. The initial backbone $f_{0}$ is first trained for $\mathcal{E}_{1}$ epochs. The resulting network $f_{1}$ is grown over $\mathcal{E}_{g}$ epochs to obtain the grown network $f_{g}$, i.e., $f_{g}=\mathcal{G}(f_{1}|\gamma,\mathcal{D},\mathcal{L},\mathcal{E}_{g})$, where $\mathcal{G}$ is the growth technique which is nominally Firefly Wu et al. () in our work. The final network $f_{2}$ is obtained by training $f_{g}$ over $\mathcal{E}_{2}$ epochs. Either clean ($\mathcal{D}_{\text{in}}$) or augmented ($\mathcal{D}_{\text{aug}}$) data can be used in OSG. For instance, OSG in GEARnn-1 and GEARnn-2 employs augmented data ($\mathcal{D}\sim\mathcal{D}_{\text{aug}}$) and clean data ($\mathcal{D}\sim\mathcal{D}_{\text{in}}$), respectively.

The growth technique $\mathcal{G}$ is described below:

$$ $\displaystyle\begin{split}\displaystyle f_{g}=\operatorname*{arg\,min}_{f}& \hskip 10.00002pt\mathcal{L}(f,\mathcal{D}|f_{1})\\ \text{s.t.}\hskip 10.00002pt\hskip 10.00002pt&f\in\partial(f_{1},\epsilon)\\ \mathcal{C}(f&)\leq(1+\gamma)\;\mathcal{C}(f_{1})\ \end{split}$ (1) $$

where $\partial(f_{1},\epsilon)$ represents the growth neighbourhood for topology search, $\mathcal{C}(f)=\sum_{l=1}^{L}w_{l}$ represents the complexity estimate of network $f$ and $\gamma$ denotes the growth ratio. The neighbourhood $\partial(f_{1},\epsilon)$ is expanded in two ways - splitting and growing new neurons - as described in Wu et al. (; ). We perform growth only in the width dimension and keep the number of layers $L$ and the kernel size $(K,K)$ constant for reasons described in Wu et al. () and Simonyan & Zisserman ().

Existing growth methods Wu et al. (; ); Evci et al. () use several growth steps (typically 10 steps) and large number of training epochs (typically 1600 total epochs) which makes them inefficient for training.
This directs us to pick OSG over multi-step growth (validated in Section 6.3) and reduce the training epochs significantly ($20\times$) compared to prior growth algorithms. The drop in accuracy observed due to these modifications is compensated for using a 2-Phase approach (Section 6.1).

Figure: Figure 3: OSG takes in labeled data ($\mathcal{D}$) and backbone network $f_{0}$, and performs a training step, a growth step, and a training step in sequence to generate network $f_{2}$. The 2-tuple $(\mathcal{L},\mathcal{E})=$ (loss function, number of epochs) employed in each step.
Refer to caption: x3.png

### 4.2 Efficient Robust Augmentation (ERA)

Figure: Figure 4: ERA takes in clean data ($\mathcal{D}_{\text{in}}$) as input and applies a set of stochastic transforms to generate augmented data ($\mathcal{D}_{\text{aug}}$) in an efficient manner.
Refer to caption: x4.png

Efficient Robust Augmentation (ERA) employs clean data ($\mathcal{D}_{\text{in}}$) to generate augmented data ($\mathcal{D}_{\text{aug}}$) in an efficient manner. The clean sample $\mathbf{x}$ (where ($\mathbf{x},y)\sim\mathcal{D}_{\text{in}}$) is passed through a set of transforms $a_{1},a_{2},...,a_{d_{j}}$ to obtain the transformed sample $A_{j}(x)$, which is then combined linearly with the clean sample to give the augmented sample $\mathbf{x}_{j}^{\text{aug}}$. We concatenate $(J-1)$ such augmented samples $\{\mathbf{x}^{\text{aug}}_{j}\}_{j=1}^{J-1}$ along with the clean sample to obtain our Efficient Robust Augmentation $\mathcal{R}((\mathbf{x},y)|\mathcal{T})$.

$$ $\displaystyle\begin{split}A_{j}(\mathbf{x})&=a_{1}\circ a_{2}\circ...\circ a_{ d_{j}}(\mathbf{x})\\ \mathbf{x}_{j}^{\text{aug}}&=p\mathbf{x}+(1-p)A_{j}(\mathbf{x})\\ \mathcal{R}((\mathbf{x},y)|\mathcal{T})=(\{\mathbf{x}_{1}^{\text{aug}},&..., \mathbf{x}_{J-1}^{\text{aug}},\mathbf{x}\},y)\implies\mathcal{D_{\text{aug}}}: =\mathcal{R}(\mathcal{D}_{\text{in}}|\mathcal{T})\\ \text{where }a_{i}\sim\text{Unif}(\mathcal{T}),\;p\sim\beta(1,1)&,\;d_{j}\sim \text{Unif}(\{1,...,D\}),\;j\in\{1,...,J-1\}\end{split}$ (2) $$

where $\mathcal{T}$ denotes the set of transforms, $\beta()$ and Unif() represent the beta and uniform distributions, respectively. SOTA robust data augmentation Hendrycks et al. (); Modas et al. (); Sun et al. () methods for common corruptions also employ stochastic chains of transforms with width $W$, depth $D$, and enforce consistency across $J-1$ augmented and clean samples using the $\mathcal{L}_{\text{aug}}$ loss function (described in Section 3). The SOTA augmentation framework increases the training time and energy by $3\times$$\text{ to }4\times$ compared to vanilla training. We choose $(W,D,J)=(1,3,4)$ based on our diagnosis (shown in Section B.1) to improve the efficiency without compromising on robustness compared to SOTA approaches Hendrycks et al. (); Modas et al. ().
In GEARnn-2, grown network $f_{2}$ (see Fig. 3) obtained using clean data OSG is trained for $\mathcal{E}_{r}$ epochs using $\mathcal{D}_{\text{aug}}$ generated by ERA.

### 4.3 GEARnn Algorithms

Figure: Algorithm 1 GEARnn-1

Algorithms 1 and 2 describe GEARnn-1 and GEARnn-2, respectively. Algorithms 1 and 2 output final compact and robust models $f_{1p}^{*}$ and $f_{2p}^{*}$, respectively.
For empirical results in Section 6, the growth technique $\mathcal{G}$ and the set of transforms $\mathcal{T}$ are chosen from Firefly Wu et al. () and AugMix Hendrycks et al. (), respectively, though other growth Yuan et al. (); Wu et al. () and augmentation Modas et al. (); Sun et al. () methods can be substituted to obtain different GEARnn variants.

## 5 Experimental Setup

Datasets and Architectures:
All results are shown on CIFAR-10, CIFAR-100 Krizhevsky et al. () and Tiny ImageNet Le & Yang () ($\mathcal{D}_{\text{in}}$) datasets. CIFAR-10-C, CIFAR-100-C and Tiny ImageNet-C Hendrycks & Dietterich () ($\mathcal{D}_{\text{out}}$) are used to benchmark corruption robustness. Convolutional neural network architectures MobileNet-V1Howard et al. (), VGG-19Simonyan & Zisserman (), ResNet-18He et al. () are employed to demonstrate the results.

Hardware:
For the server-based experiments, we use a single NVIDIA Quadro RTX 6000 GPU with 24GB RAM, 16.3 TFLOPS peak performance and an Intel Xeon Silver 4214R CPU. This machine is referred to as “Quadro”. For the Edge-based experiments, we use the NVIDIA Jetson Xavier NX NVIDIA () which has a Volta GPU with 8GB RAM, 21 TOPS peak performance and a Carmel CPU. We refer to this device as “Jetson”.

Metrics:
Clean accuracy $\mathcal{A}_{\text{cln}}(\%)$ measured on clean test data $\mathcal{D}_{\text{in}}$, and robust accuracy $\mathcal{A}_{\text{rob}}(\%)$ measured on corrupted test data $\mathcal{D}_{\text{out}}$, are used as accuracy metrics (both computed using RobustBench Croce et al. ()). The number of floating-point parameters (model size), wall-clock training time $t_{\text{tr}}$ (in minutes), per-sample wall-clock inference time $t_{\text{inf}}$ (in seconds) and energy consumption $E$ (in Joule) are used as the efficiency metrics. Size $(\%)$ represents the fraction of the full model size. In case of growth algorithms, training times include both the time taken for training and growth. The power is measured from the Quadro and Jetson using Nvidia-SMI NVIDIA () and Jetson Stats , respectively, and the energy $E$ is computed by summing the mean power values polled.

Baselines: In the absence of prior work on robust growth, we propose our own baselines Small ($\mathcal{D}_{\text{in}}$) and Small ($\mathcal{D}_{\text{aug}}$), both of which use 160 training epochs to be consistent with Diffenderfer et al. ().
They are networks with the same size and topology as the final GEARnn-2 network ($f_{2p}^{*}$ in Fig. 2) trained with random initialization on clean data and augmented data (AugMix Hendrycks et al. (), unless specified otherwise), respectively.

We pick Small ($\mathcal{D}_{\text{aug}}$) as the main baseline for a fair comparison with GEARnn as it depicts a typical private-Edge training scenario. We do not compare with compression techniques since they have been shown to have worse training efficiency compared to growth Yuan et al. (), and require a robust-trained full baseline, and this is clearly more expensive than training Small ($\mathcal{D}_{\text{aug}}$) (see  Fig. 1).

**Table 1: GEARnn hyperparameters for different networks and datasets.**
| Dataset | Growth Ratio ($\gamma$) | Small($\mathcal{D}$) | GEARnn-1 | GEARnn-2 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mob. | VGG | Res. | $\mathcal{E}$ | $\mathcal{E}_{1}$ | $\mathcal{E}_{g}$ | $\mathcal{E}_{2}$ | $\mathcal{E}_{1}$ | $\mathcal{E}_{g}$ | $\mathcal{E}_{2}$ | $\mathcal{E}_{r}$ |  |
| CIFAR-10 | 1.8 | 0.9 | 0.6 | 160 | 40 | 1 | 40 | 40 | 1 | 40 | 40 |
| CIFAR-100 | 2.0 | 1.5 | 0.8 | 160 | 50 | 1 | 50 | 40 | 1 | 40 | 50 |
| Tiny ImageNet | 2.0 | 1.5 | 0.8 | 160 | 50 | 1 | 50 | 40 | 1 | 40 | 50 |

## 6 Main Results

In this section, we first compare the performance of GEARnn across different network architectures and datasets on Quadro. We then show results for CIFAR-10 and CIFAR-100 using VGG-19 and MobileNet on Jetson. Finally, we compare OSG with $m$-shot growth methods on Jetson.

### 6.1 Results across Network Architectures and Datasets

Figure: (a)
Refer to caption: x5.png

Table 2 shows GEARnn is consistently better in terms of training time and training energy consumption over the best baseline Small ($\mathcal{D}_{\text{aug}}$) over multiple network architectures and datasets. Specifically, an average reduction in training time (energy consumption) of $\mathbf{3.5\times}$, $\mathbf{2.9\times}$ and $\mathbf{1.8\times}$ ($\mathbf{3.7\times}$, $\mathbf{2.0\times}$ and $\mathbf{2.0\times}$) is observed for CIFAR-10, CIFAR-100 and Tiny ImageNet, respectively. Furthermore, we find GEARnn-1 is inferior to GEARnn-2 on all the four metrics thereby answering Q1 in Section 1 – 2-Phase approach is better than 1-Phase approach for efficiently growing robust networks.

A key reason underlying GEARnn-2’s training efficiency is the reduction in the number of robust training epochs $\mathcal{E}_{r}$ made possible by the OSG initialization in Phase-1. Fig. 5 shows that for the same training time, GEARnn-2 provides better robustness than Small ($\mathcal{D}_{\text{aug}}$) and GEARnn-1. Similar results were obtained for CIFAR-10 and other network architectures as shown in Appendix C.

**Table 2: Comparison of accuracy, robustness, and efficiency between the baselines and GEARnn across various network architectures for CIFAR-10, CIFAR-100 and Tiny ImageNet on Quadro. See Fig. 5 for robustness comparison between Small ($\mathcal{D}_{\text{aug}}$) and GEARnn-2 at similar training cost.**
| Architecture |  | CIFAR-10 | CIFAR-100 | Tiny ImageNet |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| (full model | Method | Size | Accuracy | Training | Size | Accuracy | Training | Size | Accuracy | Training |  |  |  |  |  |  |
| size) | (%) | $\mathcal{A}_{\text{cln}}(\%)$ | $\mathcal{A}_{\text{rob}}(\%)$ | $t_{\text{tr}}$(min) | $E$(kJ) | (%) | $\mathcal{A}_{\text{cln}}(\%)$ | $\mathcal{A}_{\text{rob}}(\%)$ | $t_{\text{tr}}$(min) | $E$(kJ) | (%) | $\mathcal{A}_{\text{cln}}(\%)$ | $\mathcal{A}_{\text{rob}}(\%)$ | $t_{\text{tr}}$(min) | $E$(kJ) |  |
|  | Small ($\mathcal{D}_{\text{in}}$) | 8 | 92.28 | 66.31$\downarrow$ | 42 | 192 | 8 | 67.66 | 39.04$\downarrow$ | 45 | 274 | 8 | 55.13 | 18.48 $\downarrow$ | 262 | 2030 |
| MobileNetV1 | Small ($\mathcal{D}_{\text{aug}}$) | 8 | 92.90 | 83.21 | 211 | 1130 | 8 | 68.88 | 54.95 | 212 | 1330 | 8 | 56.46 | 28.17 | 765 | 7200 |
| (3M) | GEARnn-1 | 7 | 90.64 | 80.71 | 88 | 379 | 8 | 65.07 | 51.46 | 93 | 651 | 8 | 54.57 | 27.46 | 506 | 4410 |
|  | GEARnn-2 | 8 | 91.35 | 81.96 | 56 | 270 | 8 | 67.95 | 53.28 | 72 | 432 | 8 | 56.16 | 28.56 | 429 | 3565 |
|  | Small ($\mathcal{D}_{\text{in}}$) | 5 | 92.69 | 70.57$\downarrow$ | 31 | 241 | 9 | 68.07 | 41.24$\downarrow$ | 38 | 335 | 9 | 53.9 | 17.78 $\downarrow$ | 218 | 2040 |
| VGG-19 | Small ($\mathcal{D}_{\text{aug}}$) | 5 | 93.08 | 85.73 | 215 | 1140 | 9 | 70.01 | 56.94 | 219 | 927 | 9 | 55.51 | 30.01 | 668 | 7120 |
| (20M) | GEARnn-1 | 5 | 91.25 | 82.86 | 86 | 552 | 9 | 65.73 | 52.68 | 111 | 779 | 9 | 54.38 | 28.56 | 428 | 4220 |
|  | GEARnn-2 | 5 | 92.18 | 83.77 | 53 | 298 | 9 | 68.44 | 54.31 | 65 | 566 | 9 | 56.19 | 29.79 | 357 | 3219 |
|  | Small ($\mathcal{D}_{\text{in}}$) | 6 | 93.34 | 68.85$\downarrow$ | 61 | 546 | 7 | 68.74 | 40.83$\downarrow$ | 67 | 490 | 7 | 54.72 | 18.11 $\downarrow$ | 381 | 3390 |
| ResNet-18 | Small ($\mathcal{D}_{\text{aug}}$) | 6 | 94.18 | 86.50 | 217 | 1730 | 7 | 71.97 | 57.30 | 219 | 1250 | 7 | 54.50 | 25.74 | 1103 | 12400 |
| (12M) | GEARnn-1 | 6 | 92.36 | 83.86 | 108 | 747 | 8 | 69.15 | 55.62 | 142 | 1020 | 7 | 53.17 | 24.93 | 898 | 9100 |
|  | GEARnn-2 | 6 | 93.14 | 84.45 | 77 | 567 | 7 | 70.94 | 56.54 | 97 | 905 | 7 | 54.79 | 26.64 | 649 | 7270 |

### 6.2 Results on the Edge

**Table 3: Comparison of accuracy, robustness, inference and training efficiency between the baselines and GEARnn for CIFAR-10 and CIFAR-100 using MobileNet-V1 and VGG-19 on Jetson. Due to computational limitations, the results for Tiny ImageNet and ResNet-18 are excluded for Jetson.**
|  |  | CIFAR-10 | CIFAR-100 |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Network | Method | Accuracy | Inference | Training | Accuracy | Inference | Training |  |  |  |  |  |  |
| $\mathcal{A}_{\text{cln}}(\%)$ | $\mathcal{A}_{\text{rob}}(\%)$ | Size% | $t_{\text{inf}}$(ms) | $t_{\text{tr}}$(min) | $E$(kJ) | $\mathcal{A}_{\text{cln}}(\%)$ | $\mathcal{A}_{\text{rob}}(\%)$ | Size% | $t_{\text{inf}}$(ms) | $t_{\text{tr}}$(min) | $E$(kJ) |  |  |
| MobileNet-V1 | Small ($\mathcal{D}_{\text{in}}$) | 91.88 | 68.35$\downarrow$ | 7 | 0.9 | 675 | 175 | 68.59 | 39.47$\downarrow$ | 8 | 0.9 | 744 | 166 |
| Small ($\mathcal{D}_{\text{aug}}$) | 92.58 | 83.84 | 7 | 0.9 | 1216 | 511 | 69.24 | 54.84 | 8 | 0.9 | 1333 | 586 |  |
| GEARnn-1 | 90.20 | 79.65 | 7 | 0.9 | 560 | 238 | 65.48 | 50.46 | 8 | 1.0 | 704 | 226 |  |
| GEARnn-2 | 91.43 | 81.64 | 7 | 0.9 | 553 | 162 | 67.42 | 52.39 | 8 | 0.9 | 690 | 216 |  |
| VGG-19 | Small ($\mathcal{D}_{\text{in}}$) | 92.97 | 71.08$\downarrow$ | 5 | 1.0 | 533 | 128 | 67.92 | 40.49$\downarrow$ | 9 | 1.4 | 714 | 187 |
| Small ($\mathcal{D}_{\text{aug}}$) | 93.36 | 85.73 | 5 | 1.0 | 1543 | 522 | 70.07 | 56.68 | 9 | 1.4 | 2016 | 678 |  |
| GEARnn-1 | 90.94 | 82.25 | 5 | 1.2 | 652 | 207 | 62.89 | 49.63 | 9 | 1.5 | 936 | 281 |  |
| GEARnn-2 | 92.07 | 83.45 | 5 | 1.0 | 596 | 155 | 67.59 | 53.64 | 9 | 1.4 | 884 | 328 |  |

We now study GEARnn when mapped onto the Edge device NVIDIA Jetson Xavier NX. The training hyperparameters for Jetson are described in Appendix A. Results on Jetson (Table 3) show similar trends to those on Quadro (Table 2).

Specifically,  Table 3 shows that GEARnn-2 achieves comparable clean and robust accuracies to the baseline Small ($\mathcal{D}_{\text{aug}}$) but at a fraction of its training cost – a $\mathbf{2.3\times}$ ($\mathbf{2.8\times}$) reduction in training time (training energy) when averaged across both networks and datasets. Additionally, GEARnn-2 beats GEARnn-1 on almost all metrics, again confirming our answer to Q1 in favour of 2-Phase. Interestingly, GEARnn-2 achieves a clean accuracy within $1\%$ of Small ($\mathcal{D}_{\text{in}}$) at a similar training cost.
These results confirm that it is possible to grow efficient and robust networks on the Edge.

### 6.3 One-Shot vs. Multi-Shot Growth

Since GEARnn employs OSG (One-Shot Growth) for growing networks, it begs the question if we are missing anything if multiple growth steps ($m$-Shot Growth) were to be permitted, i.e., question Q2 from Section 1.
To answer this question, we compare the clean and robust accuracies along with training time and energy for different growth steps between GEARnn-1 and GEARnn-2 in Table 4. All $m$-Shot Growth methods start with the same initial backbone $f_{0}$ (1.4% of full model size) and perform growth to reach $f_{2}$ (5% of full model size) using different growth ratios. All methods use VGG-19 model and perform 80 epochs parametric training during the growth phase. The experiments are done on CIFAR-10 data and the hardware measurements are taken from Jetson.

**Table 4: Comparison of training complexities, clean and robust accuracies for different growth methods implemented using VGG-19 and CIFAR-10 on Jetson. 2-Phase approach and OSG provide the best solution for growing robust networks on the Edge.**
| Growth | GEARnn-1 | GEARnn-2 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Steps | $\mathcal{A}_{\text{cln}}(\%)$ | $\mathcal{A}_{\text{rob}}(\%)$ | $t_{\text{tr}}$ (min) | $E$ (kJ) | $\mathcal{A}_{\text{cln}}(\%)$ | $\mathcal{A}_{\text{rob}}(\%)$ | $t_{\text{tr}}$ (min) | $E$ (kJ) |
| 1 | 90.94 | 82.25 | 652 | 207 | 92.07 | 83.45 | 596 | 155 |
| 2 | 90.01 | 81.92 | 640 | 191 | 91.94 | 83.34 | 593 | 157 |
| 3 | 89.73 | 80.86 | 653 | 194 | 91.79 | 83.05 | 624 | 177 |
| 4 | 89.90 | 81.08 | 845 | 223 | 91.65 | 82.75 | 645 | 173 |

Table 4 indicates that OSG is comparable or better than the other $m$-Shot Growth methods in all the metrics, thereby answering Q2. This result can be attributed to the lower training overhead of growth stage in OSG compared to the $m$-Shot Growth methods. It should be noted that as the growth steps increase, the accuracies go down and training cost goes up, thus indicating that the optimal solution cannot be found by further increasing the growth steps. Another comparison that is highlighted by Table 4 is the one between GEARnn-1 and GEARnn-2. For each growth step, GEARnn-2 is better than the corresponding GEARnn-1 solution on all the metrics. The numbers highlighted in red indicate the best solution across the table. Thus, Table 4 clearly highlights that 2-Phase approach using One-Shot Growth is the best combination to grow robust networks efficiently on the Edge. More comparisons between OSG and Multi-Shot growth are shown in Section B.3.

## 7 Ablation Study

In this section, we look at the generalization of GEARnn to other robust augmentations and then understand the robustness and efficiency breakdowns for GEARnn.

### 7.1 Generalization across robust augmentation methods

The results thus far employed AugMix Hendrycks et al. () tranforms ($\mathcal{T}$) to generate $\mathcal{D}_{\text{aug}}$ for robust training. In this section, we see if the benefits of GEARnn are maintained across other augmentation transforms. Table 5 compares the implementation of PRIME Modas et al. () augmentation across different methods. The accuracy and efficiency trend observed are similar to the results in Table 2. The important aspect to notice is the increase in training complexity gap ($\sim 2\times$) between GEARnn-1 and GEARnn-2. This is because OSG with PRIME is more expensive than OSG with AugMix.

Table: Table 5: Accuracy and Efficiency comparisons for PRIME ($\mathcal{D}_{\text{aug}}$) augmentation implemented for VGG-19 and CIFAR-10 on Quadro.

| Method | $\mathcal{A}_{\text{cln}}(\%)$ | $\mathcal{A}_{\text{rob}}$ (%) | $t_{\text{tr}}$ (min) | $E$ (kJ) |
| --- | --- | --- | --- | --- |
| Small ($\mathcal{D}_{\text{in}}$) | 92.69 | 70.57$\downarrow$ | 31 | 241 |
| Small ($\mathcal{D}_{\text{aug}}$) | 91.30 | 87.01 | 829 | 2550 |
| GEARnn-1 | 88.37 | 83.18 | 458 | 1410 |
| GEARnn-2 | 90.26 | 84.45 | 234 | 856 |

Table: Table 6: Training time and energy breakdown for GEARnn on CIFAR-10 using VGG-19 on Quadro.

| Quantity | GEARnn-1 | GEARnn-2 |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| OSG-1 | OSG-2 | Total | OSG-1 | OSG-2 | ERA | Total |  |
| training | 38 | 48 | 86 | 5 | 10 | 38 | 53 |
| time (min) | 44% | 56% | 100% | 9% | 19% | 72% | 100% |
| energy | 180 | 372 | 552 | 26 | 71 | 201 | 298 |
| (kJ) | 33% | 67% | 100% | 9% | 24% | 67% | 100% |

### 7.2 Efficiency and Robustness breakdown

**Table 7: Impact of using OSG and ERA for CIFAR-100 and VGG-19 on Quadro.**
| Phase-1 ($\mathcal{D}_{\text{in}}$) | Phase-2 ($\mathcal{D}_{\text{aug}}$) | $\mathcal{A}_{\text{rob}}(\%)$ | $t_{\text{tr}}$(min) | $E$ (kJ) |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| vanilla | OSG | AugMix | ERA |  |  |  |
| ✓ |  |  |  | 38.72 | 18 | 161 |
|  | ✓ |  |  | 38.01 | 16 | 118 |
|  |  | ✓ |  | 46.50 | 62 | 385 |
|  |  |  | ✓ | 46.13 | 46 | 406 |
| ✓ |  | ✓ |  | 53.74 | 79 | 534 |
|  | ✓ |  | ✓ | 54.31 | 64 | 515 |

Table 6 shows the breakdown of energy and training time for different stages of GEARnn-1 and GEARnn-2. OSG-1 involves the training of backbone $f_{0}$ and OSG-2 includes both the growth stage and training of $f_{g}$. The key aspect to notice in Table 6 is the small fraction of training cost required by OSG-1 and OSG-2 in GEARnn-2 to provide a good initialization.

Table 7 shows the ablation studies of different components used in GEARnn-2 and compares it with a fixed network robust training. Firstly, we notice that OSG is more efficient than vanilla (fixed network) training, both in terms of training time and energy while achieving comparable accuracy. Similar observations can be made for ERA over AugMix. Performing 2-Phase approach by using either vanilla or OSG as initialization provides a significant boost in robustness while incurring a minimal overhead in training cost. Thus the 2-Phase approach is a clear winner over the 1-Phase approach, and in particular the combination of OSG and ERA used for GEARnn-2 is optimal. More comparisons between AugMix and ERA on Jetson are shown in Section B.2.

## 8 Discussion

Until now we have looked at extensive empirical simulations that highlight the efficacy of GEARnn-2. In this section, we will look at the inner workings of this algorithm. Specifically, we will see what network topologies are generated when OSG designs compact networks, and also understand why clean data initialization benefits robust training.

### 8.1 Impact of OSG on Network Topology

Figure: (a)
Refer to caption: x7.png

In this section, we look at the growth topology patterns ($\{w_{l}\}_{l=1}^{L}$) as a function of layer index $l$. Specifically, we investigate these patterns in the simple setting of OSG ($\mathcal{D}_{\text{in}}$) implemented on CIFAR-10 for ($\mathcal{E}_{1},\mathcal{E}_{2}$) = (40, 40) and an initial backbone $f_{0}$ with $\{w_{l}\}_{l=1}^{L}=45$. The bar plots represent the mean width ($\mathbb{E}[w_{l}]$) across four random seeds.

Backbone architecture: For plain CNNs like VGG-19 Simonyan & Zisserman () - the initial layers have higher number of convolutional filters compared to final layers. This correlates with the observations seen in quantization Sakr & Shanbhag () where the initial layers require higher precision compared to the final layers.
However, in case of residual networks like ResNet-18, the pattern is largely invariant to network depth and is oscillating as shown in Fig. 6(a). The invariance in depth can be attributed to the direct gradient flow facilitated by the shortcut connections making each residual block act independently of the depth.
In each residual block, the macro-level pattern in plain CNNs is observed at a micro-level, i.e. initial layer has more output channels than the final layer.

Growth Epochs and Data: All the above experiments were performed for a single growth epoch ($\mathcal{E}_{g}$ = 1) and on clean data. The effect of increasing $\mathcal{E}_{g}$ to 50 and using ERA data for growth (GEARnn-1) is shown in Fig. 6(b) and Fig. 6(c). The topology pattern in both cases remains roughly the same as OSG ($\mathcal{D}_{\text{in}}$) $\mathcal{E}_{g}=1$.

### 8.2 Rationale for 2-Phase Approach

In this section, we provide insights for the efficacy of GEARnn-2 and the 2-Phase approach. In particular, we highlight why training or growth done on clean data provides a good initialization for robust training. We look at the loss curves for the 1-Phase approaches (Small (AugMix), Small (ERA), GEARnn-1) and the 2-Phase approach (GEARnn-2) in Fig. 7. The initial dip in GEARnn-2 loss function in Fig. 7(a) is due to the loss landscape being different for Phase-1 done on clean data compared to Phase-2 done on augmented data. One can clearly see that GEARnn-2 achieves a lower loss at a faster rate compared to the other 1-Phase approaches, thus justifying the importance of clean growth initialization. We also plot the filter normalized loss curves Li et al. () in Fig. 7(b) to observe the loss landscapes around the converged weights. GEARnn-2 finds the smallest minima while also having a wide curve which enables better generalization Li et al. ().

The above explanation illustrates why GEARnn-2 has a good training and generalization performance. However, in order to understand why initialization with clean data aids faster convergence of robust training, we look at the Fourier spectrums of the clean, augmented and corrupted images in Fig. 8. Fig. 8(a) indicates that the clean images lie in the low frequency domain, while the corrupted samples occupy a wide range of frequencies (Figs. 8(b) & 8(c)). Crucially, the spectrum containing all the augmentations (in AugMix) Fig. 8(d) and all the corruptions (in CIFAR-10-C) Fig. 8(e) is also in the low-frequency domain, similar to the clean image spectrum Fig. 8(a). This is unlike the scenario of adversarial or Gaussian noise perturbations, which lie in the high-frequency domain Yin et al. () and hence may not benefit from clean data initialization. Thus, robust training for common corruptions benefits from initialization with clean data.

Figure: (a)
Refer to caption: x10.png

Figure: (a) clean data
Refer to caption: x12.png

## 9 Limitations and Broader Impacts

While our work has conclusively shown that a 2-Phase approach for growing robust networks is computationally efficient, a theoretical convergence analysis for this result is currently lacking. Such a result would help identify favorable initial conditions for robust training to achieve high accuracy in fewer epochs.

The impact of our work is broadly positive since it enables efficient robust training on Edge devices. We do not see any direct negative impact of our work.

## 10 Conclusion

We addressed the problem of growing robust networks efficiently on Edge devices. Specifically, we concluded that a 2-Phase approach with distinct clean growth and robust training phases is significantly more efficient than a 1-Phase approach which employs augmented data for growth. We encapsulated this result into the GEARnn algorithm and experimentally demonstrated its benefits on a real-life Edge device. An interesting and non-trivial extension of our work would be to use unlabeled data for growing efficient and robust networks. Another extension would be to design robust networks for complex tasks such as object detection on highly resource-constrained Edge platforms.

## 11 Reproducibility Statement

We list all the experimental setup details in Section 5 and Appendix A. We use fixed seeds during the simulation runs so that our results can be reproduced. We will make the code public along with the software versions if the paper is accepted so that the community can use and reproduce our results.

## Appendix / Supplemental material

## Appendix A Training Setup

Hyperparameters: The setup for growth and robust augmentation follows closely with what is described in Firefly Wu et al. () and AugMix Hendrycks et al. (), respectively. The parametric training is done for 160 epochs using a batch-size of 128 and an initial learning rate of 0.1. The learning rate scheduler decays by 0.1 at half and three-fourths of the total number of epochs. We use the Swish loss function for MobileNet-V1 as used in  Wu et al. (), while employing ReLU for the other two networks. Instead of using three fully-connected layers at the end of VGG-19, we use only one as done in Wu et al. (). Stochastic Gradient Descent (SGD) optimizer is used with momentum 0.9 and weight decay $10^{-4}$. As for the standard growth process, we use a Root Mean Square Propagation (RMSprop) optimizer with momentum 0.9, alpha 0.1 and initial learning rate of $9\times 10^{-5}$. The number of workers is chosen as 4. For ERA, $(W,D,J)=(1,3,4)$ is picked. The augmentation transforms $\mathcal{T}$ are same as that of AugMix Hendrycks et al. () for all the results except Table 5, where we pick the transforms from PRIME Modas et al. (). As specified in AugMix, we also do not use any augmentations which are directly present in the corrupted test dataset.

In case of OSG, the initial backbone $f_{0}$ is chosen as a network with $w_{l}=45$ for all $l=\{1,...,L\}$ and is thus extremely small.
The number of randomly initialized neurons at each growth stage is 70. We ensure that $\mathcal{E}_{2}$ of GEARnn-1 and $\mathcal{E}_{r}$ of GEARnn-2 are same for a fair comparison. $\mathcal{E}_{g}$ is chosen as 1 based on Firefly Wu et al. ().

Jetson Training: The two changes to the GEARnn algorithm when implementing on NVIDIA Jetson Xavier are - one we use $j=3$ instead of $j=4$, and two, we allow only 40 randomly initialized new neurons per layer in the growth step (as compared to 70 in Wu et al. ()). These measures are taken to stay within the memory constraints of the Edge device. We also reduce the batch size (and learning rate) appropriately in case the above measures are insufficient.

## Appendix B Ablation Studies

### B.1 Diagnostics of Robust Augmentation Methods

In this section, we investigate which aspects of the robust augmentation framework described in Section 4.2 contribute most to the robustness while being training efficient. Table 8 shows different modifications of the stochastic chains obtained by varying $(W,D,J)$ values. It can be observed that the basic version with $(W,D,J)=(1,1,0)$ (uses only standard cross entropy loss with the label and augmented data as input) has the least training time, but suffers a significant drop in $\mathcal{A}_{\text{rob}}$ compared to standard AugMix. Crucially, we note that increase in $D$ and $J$ has more impact on robustness at a lesser training cost compared to $W$. For ERA, we pick the modification with $(W,D,J)=(1,3,4)$ as it provides the highest robustness while simultaneously reducing training time over AugMix.

| Experiment | $W$ | $D$ | $J$ | $\mathcal{A}_{\text{rob}}(\%)$ | $t_{\text{tr}}$(min) |
| --- | --- | --- | --- | --- | --- |
| Basic | 1 | 1 | 0 | 77.74 | 10 |
| + width | 3 | 1 | 0 | 78.51 | 16 |
| + depth | 1 | 3 | 0 | 80.31 | 12 |
| + JSD-3 | 1 | 1 | 3 | 82.43 | 20 |
| + width + depth | 3 | 3 | 0 | 80.47 | 21 |
| + width + JSD-3 | 3 | 1 | 3 | 82.27 | 32 |
| + depth + JSD-3 | 1 | 3 | 3 | 83.67 | 22 |
| + depth + JSD-2 | 1 | 3 | 2 | 82.41 | 13 |
| + depth + JSD-4 | 1 | 3 | 4 | 84.10 | 29 |
| AugMix Hendrycks et al. () | 3 | 3 | 3 | 84.05 | 41 |

Table: Table 8: Impact of training AugMix-variants on the robust accuracy and training time. Network $f_{2}$ from OSG is used as the starting network and $\mathcal{E}_{r}$ = 40. All the methods are implemented for CIFAR-10 and 5% VGG-19 network on Quadro. $W,D,J$ represent the width, depth and consistency samples used in the stochastic chains.

### B.2 Benefits of ERA on Jetson

**Table 9: Comparison of ERA versus AugMix on Jetson for VGG-19 and CIFAR-10 when trained for 160 epochs**
| Method | $\mathcal{A}_{\text{cln}}(\%)$ | $t_{\text{tr}}$ (min) | $E$ (kJ) |  |
| --- | --- | --- | --- | --- |
| Small (AugMix) | 93.36 | 85.73 | 1543 | 522 |
| Small (ERA) | 93.42 | 85.74 | 1542 | 486 |

In this section we will look at the benefits of using ERA over AugMix. Previously, we had looked at this comparison on Quadro using CIFAR-100 and VGG-19 in Table 7. Here, we will look at these results for CIFAR-10 and VGG-19 on Jetson when training a fixed-size Small ($\mathcal{D}_{\text{aug}}$) network for 160 epochs. Table 9 indicates that ERA is better than AugMix on all the metrics. Thus our choice of ERA over AugMix is justified.

### B.3 OSG versus Multi-Shot Growth Comparisons

In this section, we first look at clean data growth comparisons on Jetson in Table 11. Then we look at robust data growth comparisons on Quadro in Table 11. When comparing various growth methods on clean data in Table 11, we also include the Small ($\mathcal{D}_{\text{in}}$) results to highlight the efficiency benefits of growth. We can see that OSG has comparable or better training efficiency than all the methods including Small ($\mathcal{D}_{\text{in}}$). In case of clean accuracy, we observe that OSG has the highest among growth methods while being slightly lower than Small ($\mathcal{D}_{\text{in}}$). Looking at Table 11, we see that for both datasets OSG again provides comparable or best solution among all the growth methods across all metrics. Thus our choice of OSG over other Multi-Shot Growth methods is justified.

**Table 10: Comparison of training complexities and clean accuracy for different growth methods implemented using VGG-19 and clean CIFAR-10 data on Jetson.**
| Growth Steps | $\mathcal{A}_{\text{cln}}(\%)$ | $t_{\text{tr}}$ (min) | $E$ (kJ) |
| --- | --- | --- | --- |
| Small ($\mathcal{D}_{\text{in}}$) | 91.96 | 267 | 73 |
| 1 | 90.80 | 210 | 51 |
| 2 | 90.49 | 209 | 53 |
| 3 | 90.31 | 231 | 59 |
| 4 | 90.08 | 275 | 65 |

## Appendix C Impact of Robust Training Epochs

In Section 6 and Fig. 5 we observed that GEARnn-2 can achieve high robustness even when the robust training epochs are low. This is due to better initialization provided by OSG. We show the same results ablated for both VGG-19 and MobileNet-V1 for CIFAR-10 and CIFAR-100 in Fig. 9.

Figure: (a)
Refer to caption: x17.png

## Peer Reviews

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: Please refer to the weakness part
soundness: 3
strengths: 1. It is critical to produce and optimize an accurate neural network on resource-constrained edge devices, the paper provides a good solution that addressing the challenge of deploying efficient, accurate, and robust deep learning models on the Edge.
2. The two-phase growth approach can provide efficiency and robustness.
3. The proposed method is verified on real devices NVIDIA Jetson Xavier NX
4. The proposed approach addresses the key challenges of high computational complexity and fragility to out-of-distribution (OOD) data.
summary: The paper proposes GEARnn (Growing Efficient, Accurate, and Robust neural networks) to grow and train robust networks completely on resource-constrained Edge devices. The two questions raised by the paper are critical, whose solutions eventually achieve the overall objective. The paper is well organized and written. The idea is novel and experimental verification seems convincing.
weaknesses: 1. I am expecting more real edge devices to demonstrate the feasibility and also the performance difference. 
2. Is it possible to include other datasets such as multimodal cases.
3. Is it possible to show some analysis regarding on the performance of the neural training method?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Besides the above comments, there are also some minors:

- Have you explored combining the solution with other efficiency techniques like quantization? This could potentially further improve the results for edge deployment.

- What modifications would be needed for growing self-attention layers and handling multi-head structures?
soundness: 3
strengths: - The paper addresses a real-world problem of training robust networks on resource-constrained edge devices

- The paper also provides comprehensive empirical evaluation results.
summary: This paper introduces a method for training robust neural networks directly on edge devices. It designs a one-shot growth (OSG) approach to efficiently grow networks using clean data, and an efficient robust augmentation (ERA) method to enhance robustness. The authors demonstrate that the solution achieves competitive accuracy and robustness while reducing training time and energy consumption on edge devices compared to traditional approaches.
weaknesses: - More theoretical analysis is expected. For example, the rationale provided in Section 8 (referenced but not shown in the excerpt) appears mostly empirical. What is the fundamental challenge and uniqueness of the technical contribution? 

- More implementation details can be used. For example, for the ERA method, more specifics about transform operations and hyperparameter choices can help the readers.

- More analysis analysis for some designs can help. For example, the hyperparameter growth ratio γ: more details will improve the assessment of the algorithm's stability. 

- Is it possible to assess its applicability to architectures like ViT, given ViT's distinct characteristics like self-attention mechanisms and multi-head structure, what are the implications?

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please check above.
soundness: 2
strengths: The paper is very well written. The problem is interesting. The performance analysis shows the benefit of applying GEAR and provides interesting insights on how grow techniques work.
summary: The paper studies on-device learning, proposing training solutions that are robust to common corruptions of data while requiring low resource footprint. It uses growth techniques that gradually increase the network sizes to reach to a robust accuracy, and applies robust data augmentation techniques to improve robustness against common corruptions. The paper provides interesting insights, but it lacks the necessary novelty, as it is heavily based on the previously proposed techniques.
weaknesses: The paper lacks the novelty, the proposed solution is a combination of different techniques proposed before. Also, it has limited applicability, as the evaluation results are shown for CCN models only, and it is not clear how the proposed solution can be extended to more recent models, such as transformers. 

Besides, I found isolated on-device training rather impractical: i) it is very slow (as also mentioned by the authors), and ii) in many cases, there might not be enough data at the device to train the model accurately. The current focuses are therefore on 

- Model fine-tuning (especially for large models such as LLMs and ViTransformers): where a model is trained using publicly available data, and fine-tuned at devices using local data. Techniques such as LORA can be applied to reduces the resource cost of such re-training.

- Federated Learning:  where multiple edge devices are training similar model, learning from each other data, significantly improving the convergence and accuracy of the model. In an FL system, you can train a large model, with devices training subset of the parameters only.   

Please provide some insights on the advantages of performing isolated on-device compared with other techniques (especially model fine-tuning technique). Is there a use case where the latter cannot be applied? 

Moreover, the problem formulation is not very clear. What does mean ensuring network complexity is small? How small? Should not this constraint be defined by the limitations at the device (e.g. by memory and computation constraints at the device)?

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. How can you achieve the efficient, accurate, and robust networks simulatenesouly on an Edge device?

2. How is theoretical convergence for  the GEARnn algorithms?

3. How about the implementation code? Can you share it on public repo?

4. What are the main strengths and weaknesses of your approach compared to existing state-of-the-art approaches.
soundness: 2
strengths: 1. It was said to be the first work to grow networks robust to common corruptions. 

2.The proposed GEARnn method is measured on NVIDIA Jetson Xavier NX, Edge device NVIDIA.

3. The experiments show some promising results of the proposed GEARnn architecture.
summary: In order to address the problem of growing robust networks efficiently on Edge devices, this paper proposes GEARnn (Growing Efficient, Accurate, and Robust neural networks) to grow and train robust networks in-situ. It was said to be the first work to grow networks robust to common corruptions. Experimental results on a NVIDIA Jetson Xavier NX demonstrate the construction of efficient, accurate, and robust networks entirely on an Edge device.
weaknesses: 1. The organization of this paper can be greatly enhanced. There are many times double-columns and single-columns mixed together. 

2. The authors make six bullets of contributions, however, some of them are trivial contributions and can be merged together, as Q1 and Q2, which are part of GEARnn.

3. The proposed solution GEARnn is based on a family of growth algorithms, which limits the novelty in algorithm design. 

4. The objective of this paper lies in the improvements in robust accuracy, training time, and model size. However, these metrics can not be achieved simultaneously. 

5. The trade-offs between accuracy, robustness, model size, energy consumption, and training time are not well discussed both theoretically and experimentally.

6. Insufficient Comparative Analysis: The comparative analysis lacks depth. The authors only propose their own baselines Small (Din)
and Small (Daug). The authors need to expand the comparison to include state-of-the-art methods beyond the baselines mentioned, to position the paper's contributions within the current research landscape.

7. The complexity of the proposed solutions, GEARnn-1 and GEARnn-2 may be too high for practical deployment in resource-constrained environments. The paper does not provide sufficient detail on the computational overhead and latency introduced by the frequent updates and training cycles of GEARnn-1 and GEARnn-2 models, which may hinder real-time performance.

8. A theoretical convergence analysis for this result is currently lacking. It is unconvincing to present a large number of carefully designed experiments, but a rigorous convergence proof to test whether the GEARnn algorithms can yield a good convergent performance.

9. The full name should be put in front of the abbreviation, e.g., GEARnn (Growing Efficient, Accurate, and Robust neural
networks) should be Growing Efficient, Accurate, and Robust neural networks  (GEARnn).

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. As for the experiment baselines, "In the absence of prior work on robust growth" (line 270) is not an appropriate reason for excluding SOTA methods from comparison. To be more convincing, the authors are supposed to compare the robust accuracy between GEARnn and the SOTA methods (e.g. [5-7]) to show the robustness of GEARnn empirically. 
2. The authors divide GEARnn into GEARnn-1 and GEARnn-2, with one and two growing phases respectively. However, the authors emphasize the superiority of the 2-phase approach (i.e. GEARnn-2) in the experiment and discussion. In this case, what's the meaning of proposing GEARnn-1 if it is completely outperformed by GEARnn-2? It would be better if you eliminate the concept of "GEARnn-1" and simply use "GEARnn" to represent GEARnn-2. If you consider GEARnn-1 and GEARnn-2 as two separate contributions, you need to analyze the advantages of GEARnn-1 over GEARnn-2.


[1] Kim, Jiwon, Jung Kwon Lee, and Kyoung Mu Lee. "Accurate image super-resolution using very deep convolutional networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[2] Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: a general approach for growing neural networks. Advances in neural information processing systems, 2020.

[3] Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. Advances in Neural Information Processing Systems, 2019.

[4] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.

[5] Xin Yuan, Pedro Henrique Pamplona Savarese, and Michael Maire. Accelerated training via incrementally growing neural networks using variance transfer and learning rate adaptation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 

[6] Utku Evci, Bart van Merrienboer, Thomas Unterthiner, Max Vladymyrov, and Fabian Pedregosa. Gradmax: Growing neural networks using gradient information. arXiv preprint arXiv:2201.05125, 2022.

[7] Ding, N., Tang, Y., Han, K., Xu, C., & Wang, Y. Network expansion for practical training acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
soundness: 1
strengths: 1. This paper is well-structured and written.
2. This paper is well-motivated. In practice, addressing the problem of corrupted training data and high computation complexity is quite meaningful for edge devices.
3. The experiment results demonstrate the effectiveness of training the grown model with augmented data in terms of robust accuracy.
summary: This paper proposes GEARnn, a robust and efficient neural network growth method based on One-Shot Growth (OSG). Given a small backbone model, GEARnn constructs a large grown model via a single growth step, which is more computationally efficient than traditional multi-shots growth methods. Besides, this paper also introduces efficient robust augmentation (ERA) to obtain an augmented dataset. Training the grown model on the augmented dataset can increase the model’s robust accuracy on corrupted and out-of-distribution datasets.
weaknesses: 1. The ideas of OSG and ERA in this paper are non-innovative. Firstly, ERA advocates for adding a clean data residual onto augmented data. This idea is not original and has already been explored. For instance, in super-resolution, [1] proposes to add clean samples (high-resolution images) to corrupted samples (low-resolution images) to improve the accuracy of super-resolution models. Secondly, OSG simply applies Firefly [2] as the growing method and does not present any new model-growing technique. In this case, the only difference between OSG and existing works is that OSG straightforwardly obtains a fully-grown model with fewer growing steps, which is too simple and lacks novelty.
2. As OSG derives a fully-grown model from a backbone model in one single growth step, one potential problem is that it might generate a model that is overly complex. For example, a device with simple data distribution might only need an intermediate-level model to process its data. In this case, training a fully-grown overparametrized model with larger training complexity and memory consumption, might become unnecessary. 
3.  The authors' statement that GEARnn is robust across different augmentation methods is doubtful. In the experiment, the authors only test GEARnn with two augmentation methods (AugMix and Prime), which is insufficient. To support this claim, at least two more augmentation methods, such as Gaussian noise perturbation [3] and DeepAugment [4], should be included in the ablation study.
4. Some presentations in this paper are ambiguous and should be more clear. For example, with respect to the growth technique $\mathcal{G}$, the authors should at least introduce how Firefly works briefly, rather than simply say they follow Firefly (Section 4.1). Similarly, the authors should briefly introduce how AugMix works in Section 7.1.  Besides, it is unclear how to evaluate the loss $\mathcal{L}$ in equation (1). Since a model $f$ is obtained by expanding the backbone model $f\_{0}$, before expansion, how can you calculate $\mathcal{L}(f)$ without actually having $f$?

Meta-review: The paper studies on-device learning, proposing training solutions that are robust to common corruptions of data while requiring low resource footprint. It uses growth techniques that gradually increase the network sizes to reach to a robust accuracy, and applies robust data augmentation techniques to improve robustness against common corruptions.

The paper is the first to look at the efficient training of robust networks for common corruptions completely on the edge device. The proposed GEARnn method is measured on NVIDIA Jetson Xavier NX, Edge device NVIDIA, and the results show some promising results of the proposed GEARnn architecture.

The paper has marginal novelty though the problem is quite new. The proposed solution is a combination of different techniques proposed before, and it has limited applicability. The novelty is the most important reason for accept/reject.

Decision: Reject