Growing Efficient Accurate and Robust Neural Networks on the Edge

Authors: Vignesh Sundaresha, Naresh Shanbhag
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Corruption Robustness, Growth, Efficient Training, Edge

Abstract:
The ubiquitous deployment of deep learning systems on resource-constrained Edge devices is hindered by their high computational complexity coupled with their fragility to out-of-distribution (OOD) data, especially to naturally occurring common corruptions. Current solutions rely on the Cloud to train and compress models before deploying to the Edge. This incurs high energy and latency costs in transmitting locally acquired field data to the Cloud while also raising privacy concerns. We propose GEARnn (Growing Efficient, Accurate, and Robust neural networks) to grow and train robust networks in-situ, i.e., completely on the Edge device. Starting with a low-complexity initial backbone network, GEARnn employs One-Shot Growth (OSG) to grow a network satisfying the memory constraints of the Edge device using clean data, and robustifies the network using Efficient Robust Augmentation (ERA) to obtain the final network. We demonstrate results on a NVIDIA Jetson Xavier NX, and analyze the trade-offs between accuracy, robustness, model size, energy consumption, and training time. Our results demonstrate the construction of efficient, accurate, and robust networks entirely on an Edge device.

Decision: reject
Reviewer scores: 8, 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1729260807084

thread_type:
official_review

<thread_object>
creator: reviewer_prmj

<content>
summary:
This paper proposes GEARnn, a robust and efficient neural network growth method based on One-Shot Growth (OSG). Given a small backbone model, GEARnn constructs a large grown model via a single growth step, which is more computationally efficient than traditional multi-shots growth methods. Besides, this paper also introduces efficient robust augmentation (ERA) to obtain an augmented dataset. Training the grown model on the augmented dataset can increase the model’s robust accuracy on corrupted and out-of-distribution datasets.

strengths:
1. This paper is well-structured and written.
2. This paper is well-motivated. In practice, addressing the problem of corrupted training data and high computation complexity is quite meaningful for edge devices.
3. The experiment results demonstrate the effectiveness of training the grown model with augmented data in terms of robust accuracy.

weaknesses:
1. The ideas of OSG and ERA in this paper are non-innovative. Firstly, ERA advocates for adding a clean data residual onto augmented data. This idea is not original and has already been explored. For instance, in super-resolution, [1] proposes to add clean samples (high-resolution images) to corrupted samples (low-resolution images) to improve the accuracy of super-resolution models. Secondly, OSG simply applies Firefly [2] as the growing method and does not present any new model-growing technique. In this case, the only difference between OSG and existing works is that OSG straightforwardly obtains a fully-grown model with fewer growing steps, which is too simple and lacks novelty.
2. As OSG derives a fully-grown model from a backbone model in one single growth step, one potential problem is that it might generate a model that is overly complex. For example, a device with simple data distribution might only need an intermediate-level model to process its data. In this case, training a fully-grown overparametrized model with larger training complexity and memory consumption, might become unnecessary. 
3.  The authors' statement that GEARnn is robust across different augmentation methods is doubtful. In the experiment, the authors only test GEARnn with two augmentation methods (AugMix and Prime), which is insufficient. To support this claim, at least two more augmentation methods, such as Gaussian noise perturbation [3] and DeepAugment [4], should be included in the ablation study.
4. Some presentations in this paper are ambiguous and should be more clear. For example, with respect to the growth technique $\mathcal{G}$, the authors should at least introduce how Firefly works briefly, rather than simply say they follow Firefly (Section 4.1). Similarly, the authors should briefly introduce how AugMix works in Section 7.1.  Besides, it is unclear how to evaluate the loss $\mathcal{L}$ in equation (1). Since a model $f$ is obtained by expanding the backbone model $f\_{0}$, before expansion, how can you calculate $\mathcal{L}(f)$ without actually having $f$?

questions:
1. As for the experiment baselines, "In the absence of prior work on robust growth" (line 270) is not an appropriate reason for excluding SOTA methods from comparison. To be more convincing, the authors are supposed to compare the robust accuracy between GEARnn and the SOTA methods (e.g. [5-7]) to show the robustness of GEARnn empirically. 
2. The authors divide GEARnn into GEARnn-1 and GEARnn-2, with one and two growing phases respectively. However, the authors emphasize the superiority of the 2-phase approach (i.e. GEARnn-2) in the experiment and discussion. In this case, what's the meaning of proposing GEARnn-1 if it is completely outperformed by GEARnn-2? It would be better if you eliminate the concept of "GEARnn-1" and simply use "GEARnn" to represent GEARnn-2. If you consider GEARnn-1 and GEARnn-2 as two separate contributions, you need to analyze the advantages of GEARnn-1 over GEARnn-2.


[1] Kim, Jiwon, Jung Kwon Lee, and Kyoung Mu Lee. "Accurate image super-resolution using very deep convolutional networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[2] Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: a general approach for growing neural networks. Advances in neural information processing systems, 2020.

[3] Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. Advances in Neural Information Processing Systems, 2019.

[4] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.

[5] Xin Yuan, Pedro Henrique Pamplona Savarese, and Michael Maire. Accelerated training via incrementally growing neural networks using variance transfer and learning rate adaptation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 

[6] Utku Evci, Bart van Merrienboer, Thomas Unterthiner, Max Vladymyrov, and Fabian Pedregosa. Gradmax: Growing neural networks using gradient information. arXiv preprint arXiv:2201.05125, 2022.

[7] Ding, N., Tang, Y., Han, K., Xu, C., & Wang, Y. Network expansion for practical training acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730295434002

thread_type:
official_review

<thread_object>
creator: reviewer_ywgt

<content>
summary:
In order to address the problem of growing robust networks efficiently on Edge devices, this paper proposes GEARnn (Growing Efficient, Accurate, and Robust neural networks) to grow and train robust networks in-situ. It was said to be the first work to grow networks robust to common corruptions. Experimental results on a NVIDIA Jetson Xavier NX demonstrate the construction of efficient, accurate, and robust networks entirely on an Edge device.

strengths:
1. It was said to be the first work to grow networks robust to common corruptions. 

2.The proposed GEARnn method is measured on NVIDIA Jetson Xavier NX, Edge device NVIDIA.

3. The experiments show some promising results of the proposed GEARnn architecture.

weaknesses:
1. The organization of this paper can be greatly enhanced. There are many times double-columns and single-columns mixed together. 

2. The authors make six bullets of contributions, however, some of them are trivial contributions and can be merged together, as Q1 and Q2, which are part of GEARnn.

3. The proposed solution GEARnn is based on a family of growth algorithms, which limits the novelty in algorithm design. 

4. The objective of this paper lies in the improvements in robust accuracy, training time, and model size. However, these metrics can not be achieved simultaneously. 

5. The trade-offs between accuracy, robustness, model size, energy consumption, and training time are not well discussed both theoretically and experimentally.

6. Insufficient Comparative Analysis: The comparative analysis lacks depth. The authors only propose their own baselines Small (Din)
and Small (Daug). The authors need to expand the comparison to include state-of-the-art methods beyond the baselines mentioned, to position the paper's contributions within the current research landscape.

7. The complexity of the proposed solutions, GEARnn-1 and GEARnn-2 may be too high for practical deployment in resource-constrained environments. The paper does not provide sufficient detail on the computational overhead and latency introduced by the frequent updates and training cycles of GEARnn-1 and GEARnn-2 models, which may hinder real-time performance.

8. A theoretical convergence analysis for this result is currently lacking. It is unconvincing to present a large number of carefully designed experiments, but a rigorous convergence proof to test whether the GEARnn algorithms can yield a good convergent performance.

9. The full name should be put in front of the abbreviation, e.g., GEARnn (Growing Efficient, Accurate, and Robust neural
networks) should be Growing Efficient, Accurate, and Robust neural networks  (GEARnn).

questions:
1. How can you achieve the efficient, accurate, and robust networks simulatenesouly on an Edge device?

2. How is theoretical convergence for  the GEARnn algorithms?

3. How about the implementation code? Can you share it on public repo?

4. What are the main strengths and weaknesses of your approach compared to existing state-of-the-art approaches.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730385140838

thread_type:
official_review

<thread_object>
creator: reviewer_4xz4

<content>
summary:
The paper studies on-device learning, proposing training solutions that are robust to common corruptions of data while requiring low resource footprint. It uses growth techniques that gradually increase the network sizes to reach to a robust accuracy, and applies robust data augmentation techniques to improve robustness against common corruptions. The paper provides interesting insights, but it lacks the necessary novelty, as it is heavily based on the previously proposed techniques.

strengths:
The paper is very well written. The problem is interesting. The performance analysis shows the benefit of applying GEAR and provides interesting insights on how grow techniques work.

weaknesses:
The paper lacks the novelty, the proposed solution is a combination of different techniques proposed before. Also, it has limited applicability, as the evaluation results are shown for CCN models only, and it is not clear how the proposed solution can be extended to more recent models, such as transformers. 

Besides, I found isolated on-device training rather impractical: i) it is very slow (as also mentioned by the authors), and ii) in many cases, there might not be enough data at the device to train the model accurately. The current focuses are therefore on 

- Model fine-tuning (especially for large models such as LLMs and ViTransformers): where a model is trained using publicly available data, and fine-tuned at devices using local data. Techniques such as LORA can be applied to reduces the resource cost of such re-training.

- Federated Learning:  where multiple edge devices are training similar model, learning from each other data, significantly improving the convergence and accuracy of the model. In an FL system, you can train a large model, with devices training subset of the parameters only.   

Please provide some insights on the advantages of performing isolated on-device compared with other techniques (especially model fine-tuning technique). Is there a use case where the latter cannot be applied? 

Moreover, the problem formulation is not very clear. What does mean ensuring network complexity is small? How small? Should not this constraint be defined by the limitations at the device (e.g. by memory and computation constraints at the device)?

questions:
Please check above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730642308760

thread_type:
official_review

<thread_object>
creator: reviewer_fqxa

<content>
summary:
This paper introduces a method for training robust neural networks directly on edge devices. It designs a one-shot growth (OSG) approach to efficiently grow networks using clean data, and an efficient robust augmentation (ERA) method to enhance robustness. The authors demonstrate that the solution achieves competitive accuracy and robustness while reducing training time and energy consumption on edge devices compared to traditional approaches.

strengths:
- The paper addresses a real-world problem of training robust networks on resource-constrained edge devices

- The paper also provides comprehensive empirical evaluation results.

weaknesses:
- More theoretical analysis is expected. For example, the rationale provided in Section 8 (referenced but not shown in the excerpt) appears mostly empirical. What is the fundamental challenge and uniqueness of the technical contribution? 

- More implementation details can be used. For example, for the ERA method, more specifics about transform operations and hyperparameter choices can help the readers.

- More analysis analysis for some designs can help. For example, the hyperparameter growth ratio γ: more details will improve the assessment of the algorithm's stability. 

- Is it possible to assess its applicability to architectures like ViT, given ViT's distinct characteristics like self-attention mechanisms and multi-head structure, what are the implications?

questions:
Besides the above comments, there are also some minors:

- Have you explored combining the solution with other efficiency techniques like quantization? This could potentially further improve the results for edge deployment.

- What modifications would be needed for growing self-attention layers and handling multi-head structures?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730745890170

thread_type:
official_review

<thread_object>
creator: reviewer_mr6f

<content>
summary:
The paper proposes GEARnn (Growing Efficient, Accurate, and Robust neural networks) to grow and train robust networks completely on resource-constrained Edge devices. The two questions raised by the paper are critical, whose solutions eventually achieve the overall objective. The paper is well organized and written. The idea is novel and experimental verification seems convincing.

strengths:
1. It is critical to produce and optimize an accurate neural network on resource-constrained edge devices, the paper provides a good solution that addressing the challenge of deploying efficient, accurate, and robust deep learning models on the Edge.
2. The two-phase growth approach can provide efficiency and robustness.
3. The proposed method is verified on real devices NVIDIA Jetson Xavier NX
4. The proposed approach addresses the key challenges of high computational complexity and fragility to out-of-distribution (OOD) data.

weaknesses:
1. I am expecting more real edge devices to demonstrate the feasibility and also the performance difference. 
2. Is it possible to include other datasets such as multimodal cases.
3. Is it possible to show some analysis regarding on the performance of the neural training method?

questions:
Please refer to the weakness part
</content>
</thread_object>

</thread>
<thread>
creation date:
1732154236317

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response by Authors

comment:
All the changes in the latest revision of the manuscript have been indicated in blue color. The line numbers mentioned in the rebuttals are with respect to the original submission unless specified otherwise.

## Results on other Edge devices
We show the performance of our GEARnn algorithm on *NVIDIA Jetson Orin Nano* in **Appendix B.5** (along with the previous results on NVIDIA Jetson Xavier NX and NVIDIA Quadro RTX 6000) to highlight that GEARnn performance translates across Edge devices. GEARnn-2 again beats GEARnn-1 on all metrics and achieves an average reduction of $2.2\times$ and $2.8\times$ in training time and energy compared to Small($\mathcal{D}_{\text{aug}}$) at similar accuracies.

## Achieving efficient, accurate and robust neural networks simultaneously
We first explain each trade-off individually and then collectively understand why our method achieves best results on all metrics simultaneously. In case of common corruptions, the trade-off between clean accuracy and robust accuracy is minimal since clean images and augmented images (data used to improve robustness) are low frequency images as shown in Figure 8 of Section 8.2. This is observed even in AugMix [4]. One-Shot Growth ensures we choose a network topology which causes minimal trade-off between parameter size and the accuracies. Similar effect is observed in Firefly [6], but for multi-shot growth and only clean accuracy. Lastly and most importantly, the trade-off between training efficiency and the accuracies is overcome by using an efficient initialization (through clean data training) for the robust training i.e. the 2-Phase approach. Thus we are able to achieve good results simultaneously on all metrics, which is something that is not achieved by any of the prior works.

## Strengths and Contributions
Our work is the first to look at the efficient training of robust networks for common corruptions completely on the Edge device. Our key novelty lies in the proposed 2-Phase approach which provides a strong and efficient initialization for robust training. We also explain the rationale behind the efficacy of the 2-Phase approach (Section 8.2). Lastly, our work is the first to simultaneously achieve good clean accuracy, robust accuracy, inference efficiency and training efficiency. We have modified the **Contributions in Section 1** to clearly highlight this.

## Theoretical Analysis
Performing theoretical analysis for compression methods like pruning [12] and quantization [13] is relatively simpler since performance bounds can be provided relative to the trained over-parametrized network. However, growth-based parameter-efficient models are obtained during the training (not post-training like compression) of these networks. Thus firstly, we do not have a reference over-parametrized network to provide bounds against. Secondly, since GEARnn is focused on training efficiency, one has to look at convergence analysis during training of deep networks. Existing literature [14, 15] has shown that convergence analysis and generalization of deep networks are extremely hard to explain analytically. Since our motivation is grounded in practical implementation, extreme simplification of deep networks to linear classifiers or single-layered networks does not provide much relevance to our results. Hence, our work is mainly empirical and we provide substantial experiments to validate our results. However, we acknowledge this as a limitation in **Section 9** of our paper.

[12] Malach, Eran, et al. "Proving the lottery ticket hypothesis: Pruning is all you need." International Conference on Machine Learning. PMLR, 2020.

[13] Sakr, Charbel, Yongjune Kim, and Naresh Shanbhag. "Analytical guarantees on numerical precision of deep neural networks." International Conference on Machine Learning. PMLR, 2017.

[14] Nagarajan, Vaishnavh, and J. Zico Kolter. "Uniform convergence may be unable to explain generalization in deep learning." Advances in Neural Information Processing Systems 32 (2019).

[15] Zhang, Chiyuan, et al. "Understanding deep learning (still) requires rethinking generalization." Communications of the ACM 64.3 (2021): 107-115.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper studies on-device learning, proposing training solutions that are robust to common corruptions of data while requiring low resource footprint. It uses growth techniques that gradually increase the network sizes to reach to a robust accuracy, and applies robust data augmentation techniques to improve robustness against common corruptions.

The paper is the first to look at the efficient training of robust networks for common corruptions completely on the edge device. The proposed GEARnn method is measured on NVIDIA Jetson Xavier NX, Edge device NVIDIA, and the results show some promising results of the proposed GEARnn architecture.

The paper has marginal novelty though the problem is quite new. The proposed solution is a combination of different techniques proposed before, and it has limited applicability. The novelty is the most important reason for accept/reject.