Leveraging AutoML for Sustainable Deep Learning: A Multi-Objective HPO Approach on Deep Shift Neural Networks

## Abstract

Abstract Deep Learning (DL) has advanced various fields by extracting complex patterns from large datasets.
However, the computational demands of DL models pose environmental and resource challenges. Deep shift neural networks (DSNNs) offer a solution by leveraging shift operations to reduce computational complexity at inference.
Following the insights from standard DNNs, we are interested in leveraging the full potential of DSNNs by means of AutoML techniques.
We study the impact of hyperparameter optimization (HPO) to maximize DSNN performance while minimizing resource consumption. Since this combines multi-objective (MO) optimization with accuracy and energy consumption as potentially complementary objectives, we propose to combine state-of-the-art multi-fidelity (MF) HPO with multi-objective optimization. Experimental results demonstrate the effectiveness of our approach, resulting in models with over 80% in accuracy and low computational cost.
Overall, our method accelerates efficient model development while enabling sustainable AI applications.

## 1 Introduction

Deep Learning (DL) is considered one of the most promising approaches to extracting information from large data collections with complex structures. This includes performing computations in IoT environments and on edge devices (Li et al., ; Zhou et al., ).
With the ever-increasing size and performance of such models due to the progress in science and industry, a computational cost is associated with them (Sze et al., ).
Minimizing this cost directly affects a model’s environmental impact (Schwartz et al., ).
Thus, resources are freed up and can be used for other tasks, e.g., those with resource limitations (Howard et al., ). With our approach, we contribute to DL in such low-resource settings.

Deep shift neural networks (DSNNs) offer great potential in reducing power consumption compared to traditional deep learning models (Elhoushi et al., ).
Instead of floating point arithmetic, they leverage shift operations — specifically, bit shifting — as the computational unit, which boosts efficiency by replacing costly multiplication operations in convolutional networks.
We suspect that the configuration of DSNNs has a huge impact on both performance and computational efficiency.

One of the key challenges with DSNNs is determining the appropriate level of precision for the shift operations to minimize quantization errors without excessively increasing the computational load.
In this study, we leverage DSNNs in combination with automated machine learning (AutoML) to find the optimal configuration of DSNNs in the spirit of Green AutoML (Tornede et al., ).
This is achieved by hyperparameter optimization (HPO) using the framework SMAC3 proposed by Lindauer et al. (), which automates the search for optimal model configurations.
Integrating multi-fidelity (MF) and multi-objective optimization (MO) techniques (Belakaria et al., ) facilitates an optimal exploration of the hyperparameter space that jointly prioritizes performance and energy consumption (Deb, ).
SMAC’s MO implementation effectively balances the trade-off between achieving high predictive accuracy and minimizing energy consumption.
The MF aspect allows for the efficient use of computational resources by evaluating configurations at varying levels of detail.
Employing tools like CodeCarbon (Lacoste et al., ; Lottick et al., ) during the training and evaluation phases provides real-time insights into the energy consumption and carbon emissions associated with each model configuration.

Overall, we make the following contributions:

- 1.
A configuration space specific for DSNNs,
- 2.
a Green AutoML approach for efficiency-driven model building,
- 3.
insights into the design decisions for obtaining optimal tradeoffs between accuracy and energy efficiency, and
- 4.
combining MO and MF approaches in SMAC to improve optimization performance and computational resource utilization.

## 2 Background

The following chapter introduces foundational concepts for our approach.

### 2.1 Deep Shift Neural Networks

Deep shift neural networks (DSNNs) are a novel approach to reducing the computational and energy demands of deep learning (Elhoushi et al., ). They achieve a considerable reduction in latency time by simplifying the network architecture such that they replace the traditional multiplication operations in neural networks by bit-wise shift operations and sign flipping, making DSNNs suitable for computing devices with limited resources.
There are two methods for training DSNNs (Elhoushi et al., ): DeepShift-Q (Quantization) and DeepShift-PS (Powers of two and Sign).
DeepShift-Q involves training regular weights constrained to powers of 2 by quantizing weights to their nearest power of two during both forward and backward passes.
DeepShift-PS directly includes the values of the shifts and sign flips as trainable parameters.

The DeepShift-Q approach obtains the sign matrix $S$ from the trained weight matrix $W$ as $S=\mathit{sign}(W)$.
The power matrix $P$ is the base-2 logarithm of $W$’s absolute values, i.e., $P=\log_{2}(|W|)$.
After rounding $P$ to the nearest power of two, $P_{\mathit{r}}=\mathit{round}(P)$, the quantized weights $\tilde{W}_{q}$ are calculated by applying the sign from $S$, shown as $\tilde{W}_{q}=\mathit{flip}(2^{P_{\mathit{r}}},S)$.
The DeepShift-PS approach optimizes neural network weights by directly adapting the shift ($\tilde{P}$) and sign ($\tilde{S}$) values.
The shift matrix $\tilde{P}$ is obtained by rounding the base-2 logarithm of the weight values, $\tilde{P}=\mathit{round}(P)$, and the sign flip $\tilde{S}$ is computed as $\tilde{S}=\mathit{sign}(\mathit{round}(S))$.
Weights are calculated as $\tilde{W}_{\mathit{ps}}=\mathit{flip}(2^{\tilde{P}},\tilde{S})$, where the sign flip operation $\tilde{S}$ assigns values of $-1$, 0 0, or $+1$ based on $s$.

### 2.2 Hyperparameter Optimization

The increasing complexity of deep learning algorithms enhances the need for automated hyperparameter optimization (HPO) to increase model performance (Bischl et al., ).
Consider a dataset $\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{N}\in\mathbb{D}\subset\mathcal{X}\times
\mathcal{Y}$, where $\mathcal{X}$ is the instance space and $\mathcal{Y}$ is the target space, and a hyperparameter configuration space $\Lambda=\{\lambda_{1},\ldots,\lambda_{L}\}$, $L\in\mathbb{N}$. In our study, $\mathcal{M}$ denotes the space of possible DSNN models. An algorithm $\mathcal{A}:\mathbb{D}\times\Lambda\rightarrow\mathcal{M}$ trains a model $M\in\mathcal{M}$, instantiated with a configuration of $L$ hyperparameters sampled from $\Lambda$, on a training subset of $\mathcal{D}$. The dataset $\mathcal{D}$ is split into training, validation, and test sets: $\mathcal{D}_{\textit{train}},\mathcal{D}_{\textit{val}},$ and $\mathcal{D}_{\textit{test}}$ respectively. The performance of the algorithm is assessed via an expensive-to-evaluate loss function $\mathcal{L}:\mathcal{M}\times\mathbb{D}\rightarrow\mathbb{R}$. The optimization objective of HPO is to find the configuration $\lambda^{*}\in\Lambda$ with minimal validation loss $\mathcal{L}$, such that:

$$ $\lambda^{*}\in\operatorname*{arg\,min}_{\lambda\in\Lambda}\mathcal{L}\big{(} \mathcal{A}(\mathcal{D}_{\textit{train}},\lambda),\mathcal{D}_{\textit{val}} \big{)}.$ (2.1) $$

This process tunes hyperparameters based on validation set performance.
Models are trained on $\mathcal{D}_{\textit{train}}$ and optimized using $\mathcal{D}_{\textit{val}}$. Finally, the model’s performance is assessed on $\mathcal{D}_{\textit{test}}$.

### 2.3 Bayesian Optimization

Bayesian Optimization (BO) is a strategy for global optimization of black-box loss functions $\mathcal{L}:\mathcal{M}\times\mathbb{D}\xrightarrow{}\mathbb{R}$ that are expensive to evaluate (Jones et al., ).
BO uses a surrogate model $\mathcal{S}$, a probabilistic model to approximate the loss function, commonly given by a Gaussian Process or a Random Forest (Rasmussen & Williams, ; Hutter et al., ).
An acquisition function $\mathcal{C}:\Lambda\xrightarrow{}\mathbb{R}$ guides the search for the next optimal evaluation points by balancing the exploration-exploitation trade-off, based on the set of previously queried points $\{(\lambda_{1},\mathcal{L}_{1}),...,(\lambda_{m-1},\mathcal{L}_{m-1})\}$ at time $m$.
$\mathcal{L}$ is only evaluated at certain points, with which the surrogate model is updated.

### 2.4 Multi-Fidelity Optimization

Since it is not feasible to fully train multiple configurations of DSNNs for comparison due to computational efficiency, we employ a multi-fidelity (MF) approach (Li et al., ), which is a common strategy in AutoML to navigate the trade-off between performance and approximation error (Hutter et al., ).
MF approaches train cheap-to-evaluate proxies of black-box functions following different heuristics, e.g., allocating a small number of epochs to many configurations in the beginning and training the best-performing ones on an increasing number of epochs.
Formally, we define a space of fidelities $\mathcal{F}$ and aim to minimize a high-fidelity function $F\in\mathcal{F}$ (Kandasamy et al., ):

$$ $\min_{\lambda\in\Lambda}F(\lambda)\,.$ (2.2) $$

We aim to approximate $F\in\mathcal{F}$, using a series of lower-fidelity, less expensive approximations
$\{f(\lambda)_{1},\ldots,f(\lambda)_{j}\}$, where $j$ denotes the total number of fidelity levels.
The allocated resources for evaluating a model’s performance at various fidelities are referred to as a budget.
MF assumes that the highest fidelities approximate the black-box function best.

### 2.5 Multi-Objective Optimization

Multi-Objective Optimization (MO) addresses problems involving multiple, often competing objectives. This approach is used in scenarios where trade-offs between two or more conflicting objectives must be navigated, such as, in the context of DSNNs, enhancing accuracy alongside reducing energy consumption. MO aims to identify Pareto-optimal solutions (Deb, ). To ensure effective approximation of the Pareto front, new points are added based on the current observation dataset $\mathcal{D}_{\mathit{obs}}=\{(\lambda_{1},\mathcal{L}(\lambda_{1})),\ldots,(
\lambda_{n},\mathcal{L}(\lambda_{n}))\}$ at time $n$. These points augment the surface formed by the non-dominated solution set $D^{\star}_{n}$, which satisfies the condition for $d$ objective variables and a loss function $\mathcal{L}=(\mathcal{L}_{1},\ldots,\mathcal{L}_{d})$ (Wada & Hino, ): For all $\lambda,(\lambda,\mathcal{L}(\lambda))\in\mathcal{D}^{\star}_{n}\subset
\mathcal{D}_{n}$ and $\lambda^{\prime},(\lambda^{\prime},\mathcal{L}(\lambda^{\prime}))\in\mathcal{D
}_{n}$ exists $k\in\{1,\ldots,d\}$, such that $\mathcal{L}_{k}(\lambda)\leq\mathcal{L}_{k}(\lambda^{\prime})$.

## 3 Approach

To computationally enhance Deep shift neural networks (DSNN) via AutoML, we employ multi-fidelity optimization (MF).
We introduce different fidelities to the training process by increasing the number of shift layers in the model.
The initially trained models will include fewer shift layers, which will increase during the optimization process. We assume this will guide the search toward the highest-performing shift networks since a low number of shift layers introduces the least numerical inaccuracy and mathematical uncertainty to the model. We aim to investigate whether picking the best-performing models under low model-intrinsic uncertainty continues to show improved performance when combined with more shift layers.

In Section 2, we explained the MF approach. One algorithm that formalizes its heuristics is successive halving (Jamieson & Talwalkar, ), where $n_{c}$ configurations are trained on an initial small budget $b_{I}$.
After that, the $\nu/(\nu+1)$ best-performing configurations are trained on a budget of $\nu b_{I}/(\nu+1)$ until the best configuration is determined.
We address the trade-off between $b_{I}$ and $n_{c}$, or between approximation error and exploration inherent in successive halving, using the HyperBand algorithm for MF.
HyperBand (Li et al., ) runs successive halving in multiple brackets, where each bracket provides a combination of $n_{c}$ and a fraction of the total budget per configuration so that they sum up to the total budget.

We extend this to multi-fidelity multi-objective optimization (MFMO). We simultaneously address the accuracy of the model as well as its energy consumption.
The goal is to optimize the performance of the DSNNs, ensuring they achieve high accuracy and robustness in their predictive capabilities; secondly, to minimize energy consumption, a critical factor given the environmental implications of computational efficiency.
To this end, we implement a two-dimensional objective function $f_{MO}:\Lambda\xrightarrow{}\mathbb{R}^{2},\quad f_{M0}(\lambda)=\big{(}f_{
loss}(\lambda),f_{emission}(\lambda)\big{)}$, where, given a configuration $\lambda\in\Lambda$, $f_{loss}(\lambda)$ aims to minimize the loss, enhancing the model’s accuracy, and $f_{emission}(\lambda)$ seeks to minimize the energy consumption during training and inference, promoting environmental sustainability.
We aim to solve the following optimization problem:

$$ $\operatorname*{arg\,min}_{\lambda\in\Lambda}f_{M0}(\lambda)\,.$ (3.1) $$

SMAC3 uses a mean aggregation strategy by calculating the objective’s arithmetic mean to aggregate multiple objectives into a single scalar value for MF optimization.
We use the ParEGO algorithm (Knowles, ) to compute Pareto optimal objectives. It transforms the multi-objective problem into a series of single-objective problems by introducing varying weight parameters for the objectives in each iteration of HyperBand. Thus optimizing a different scalarization per evaluation to approximate the Pareto front.
The resulting single-objective optimization function can now be evaluated in an MF setting.

## 4 Experiments

In the following section, we detail the setup and methodology used to evaluate our approach discussed in Section 3, focusing on optimizing deep shift neural networks (DSNNs) through multi-fidelity, multi-objective optimization (MFMO) as well as multi-fidelity optimization (MF). We discuss how our approach successfully navigates the model performance and environmental impact trade-offs.

### 4.1 Evaluation Setup

We train and evaluate our models on the Cifar10 dataset (Krizhevsky et al., ), using NVIDIA A100 GPUs.
For hyperparameter optimization (HPO), we rely on using SMAC3 (Lindauer et al., ).
To incorporate the environmental impact into our HPO workflow, we use the CodeCarbon emissions tracker (Lacoste et al. (), Lottick et al. ()) to track carbon emissions from computational processes by monitoring energy use and regional energy mix in $g\mathit{CO}_{2}\mathit{eq}$, grams of $\mathit{CO}_{2}$ equivalent (Tornede et al., ).
We chose a pre-trained ResNet20 (He et al., ).

### 4.2 Results

Figure: Figure 1: Test accuracy of the default DSNN and a configured DSNN

Figure 1 displays the Top-1 test accuracy of a DSNN trained with the default configuration from Table 1 as defined by Elhoushi et al. (), as well as the test accuracy from the configured DSNN according to SMAC3 with MF using HyperBand, with the number of shift layers inducing the fidelities. The Pareto-optimal Solution No. 1 configuration from Table 1 is used. The MF algorithm, trained on the same seed as MFMO, identifies it as the optimal model configuration out of fifty evaluated configurations. While volatile to a certain degree, the plot clearly shows that the model’s performance with the optimal configuration surpasses that of the model instantiated with the default configuration by at least three percent. This verifies our approach that MF can successfully be executed with DSNN-specific budgets such as the shift depth.

Furthermore, it confirms our assumption from Section 3 that the models can be evaluated under low model-intrinsic uncertainty and continue to perform well when instantiated with more shift layers.

Table 1 shows two Pareto optimal solutions out of 33 evaluated configurations with our MFMO approach.
Evaluating solution No. 1 results in a Top-1 accuracy of 83.50% and 0.1661 $g\mathit{CO}_{2}\mathit{eq}$. Solution No. 2 yields a Top-1 accuracy of 84,67%, producing 0.1673 $g\mathit{CO}_{2}\mathit{eq}$.
Note that they were trained for fewer epochs due to computational limitations. Both configurations yield good performance results while keeping emissions low.

The DSNN-specific model hyperparameters include activation integer bits, activation fraction bits, weight bits, and shift depth.
Solution No.1 includes fewer shift layers with fewer bits representing the weights activation function values.
Activation bits and weight bits seem to affect emissions and model performance similarly.
Fewer shift layers increase the computational demand of a model.
However, the reduced number of bits used to replace floating-point operations (FLOPs) seems to compensate for this since solution No. 1 produces approximately the same amount of emissions as solution No. 2.
This one contains more than three times the number of shift layers, using only four convolutional layers with FLOPs, but uses substantially more representation bits.
There seems to be a trade-off between the number of shift layers and the number of representation bits that, when navigated efficiently, yields model configurations with satisfying performance and energy consumption.
This supports our claim that our MFMO approach is promising for building high-performance, energy-efficient DSNNs.

**Table 1: MFMO findings, incl. two Pareto optimal solutions (POS 1 & 2)**
| Parameter | Search Space | Default | POS 1 | POS 2 |
| --- | --- | --- | --- | --- |
| Batch Size | [32, 128] | 128 | 97 | 102 |
| Optimizer | SGD, Adam, Adagrad, Adadelta, RMSProp, RAdam, Ranger | SGD | Ranger | SGD |
| Learning Rate | [0.001, 0.1] | 0.1 | 0.0847 | 0.0798 |
| Momentum | [0.0, 0.9] | 0.9 | 0.5016 | 0.6738 |
| Epochs | [5, 100] | 200 | 67 | 91 |
| Weight Bits | [2, 8] | 5 | 4 | 7 |
| Activation Integer Bits | [2, 32] | 16 | 4 | 29 |
| Activation Fraction Bits | [2, 32] | 16 | 4 | 10 |
| Shift Depth | [0, 20] | 20 | 5 | 16 |
| Shift Type | Q, PS | PS | Q | Q |
| Rounding | deterministic, stochastic | deterministic | stochastic | deterministic |
| Weight Decay | [1e-6, 1e-2] | 0.0001 | 0.0026 | 0.00012 |
| Accuracy (in %) |  |  | 83.50 | 84.67 |
| Emissions (in $g\mathit{CO}_{2}\mathit{eq}$) |  |  | 0.1661 | 0.1673 |

## 5 Conclusion

In this work, we present our Green AutoML approach towards the sustainable optimization of DSNNs through a multi-fidelity, multi-objective (MFMO) HPO framework.
We address the critical intersection between advancing the capabilities of Deep Learning and environmental sustainability. By leveraging AutoML tools and integrating the environmental impact as an objective, we navigate the trade-off between model performance and efficient resource utilization.

Our experimental results highlight the potential of our approach. We successfully optimized a DSNN to achieve high accuracy while minimizing energy consumption.
We have delivered a thorough configuration space for DSNNs, introduced a Green AutoML approach for efficiency-driven model development, and provided valuable insights into the design decisions. Future work includes extending our approach to multiple benchmarks and neural network architectures to sufficiently verify our approach on a broad spectrum of model designs and applications. This way, we hope to gain more insight into critical design decisions for DSNNs and other models built for energy-efficient computations. Revisiting our MFMO implementation to find a more efficient way for ParEGO and Hyperband to intertwine, e.g. by finding a more effective way to assign budgets and weights to configurations, is subject to further work to lighten the computational load when computing the MFMO Pareto fronts. Furthermore, we will investigate more DSNN-specific fidelity types and MO algorithms to achieve further reductions in model emissions.

#### Acknowledgments

This work was supported by the German Federal Ministry of the Environment, Nature Conservation, Nuclear Safety and Consumer Protection (GreenAutoML4FAS project no. 67KI32007A).

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: - Could you clarify the novelty of your optimization approach for DSNNs? Specifically, are there any unique adaptations or modifications in the way DGEMO and SMAC are applied to DSNNs, beyond existing AutoML methods?
- Can you provide more details on the types of layers being quantized and the specific impact of bit-shift quantization across these layers? Are certain layers more affected by quantization than others?
- Have you considered testing your framework on additional model architectures and datasets to demonstrate its generalizability? If not, could you share insights into why ResNet20 was chosen as the sole model for evaluation?
- Could you elaborate on the rationale for the specific hyperparameters chosen for optimization? Given that only a few design hyperparameters appear to have a significant effect on the results, did you consider omitting those with minimal contributions?
soundness: 1
strengths: - The paper sheds light on an interesting design for quantized neural networks, DSNNs, which significantly reduce the computational demands at inference, positioning them as a sustainable AI solution.
- Through multi-fidelity hyperparameter optimization and multi-objective optimization, the proposed approach achieves configurations that improve DSNN performance by 20% while reducing emissions by 10% over baselines.
- The findings challenge conventional beliefs in model quantization, showing that selectively quantizing smaller portions of the network at low precision yields optimal trade-offs between energy efficiency and model accuracy.
summary: This paper investigates the potential of DSNNs to reduce computational complexity and environmental impact in deep learning (DL) models. Through AutoML techniques, the authors explore ways to optimize DSNNs by balancing multiple objectives, including accuracy and energy consumption. By applying multi-fidelity hyperparameter optimization and multi-objective optimization, they identify Pareto-optimal configurations that improve DSNN performance by 20% while reducing emissions by 10% compared to default models. Notably, their findings challenge traditional expectations around quantization, showing that selectively quantizing smaller portions of the network at low precision can yield the best trade-offs between accuracy and energy savings. These results, validated across different model architectures, underscore DSNNs' potential in designing energy-efficient, high-performing DL models.
weaknesses: - The paper's contribution in terms of novelty is somewhat limited and could benefit from further clarification. The proposed optimization approach for DSNNs appears to directly apply existing optimization techniques (DGEMO and SMAC) for hyperparameter tuning, without substantial adaptation or refinement specific to the DSNN topology or its unique optimization landscape. As it stands, the work reads more as a straightforward application of existing AutoML methods rather than an innovative extension or advancement in the field. To improve the paper's originality, I suggest the authors consider developing an algorithm that more effectively integrates various design dimensions, such as backbone architecture and bit-shifting hyperparameters, with specific tailoring for the DSNN architecture. This could provide a more distinct and impactful contribution to the AutoML research.
- The paper would benefit from a more detailed architectural analysis, particularly w.r.t. the types of layers being quantized and the specific optimizations for bit-shifting. Currently, there is a lack of discussion on how bit-shift quantization affects various layers, which limits the depth of the architectural insights provided. To strengthen the contribution, I suggest the authors include a breakdown of the impact of bit-shift quantization on different layers and specify precisely which layers are being quantized. 
- The evaluation section is somewhat limited, as it currently includes results on only a single model architecture (ResNet20). This narrow focus makes it challenging to draw robust conclusions about the effectiveness and generalizability of the proposed AutoML framework. To strengthen the evaluation, I recommend that the authors test the approach on a broader range of models and datasets. This would provide a more comprehensive assessment of its applicability and impact across different scenarios.
- The discussion on hyperparameter importance is valuable; however, its placement later in the paper limits its impact. Presenting this ablation study earlier would help justify the choice of hyperparameters targeted for optimization, especially since only a few key design hyperparameters significantly influence the results while others contribute minimally. Moving this section forward and providing either empirical or theoretical justification for selecting these specific hyperparameters would enhance the clarity and rigor of the paper's contribution.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the comments/questions above under "Weaknesses".
soundness: 3
strengths: 1. The paper is clear and well-written and provides both sufficient background and context for the work.
2. The paper is targeting improved efficiency for DSNNs, which has the potential for positive environmental impact.
3. The paper identifies improved configurations for DSNNs, and explores the Pareto front to understand the tradeoffs between configurations.
4. The impact of specific hyperparameters are analyzed.
summary: The paper reports on the tuning and optimization of deep shift neural networks (DSNNs) for both accuracy and energy efficiency. Automatic hyperparameter and architecture search is employed to tune models and explore the Pareto front of the accuracy/energy tradeoff for the models. The paper defines the relevant search spaces and discovers configurations improving upon the state-of-the-art for DSNNs.
weaknesses: 1. It is not clear to me what the novelty and broader impact of the work is. It seems, essentially, to use relatively standard automatic tuning approaches to optimize an existing architecture. While this provides some value, it also seems very specific to the DSNNs considered here. To what extent are there generalizable lessons or takeaways from the work that are more broadly applicable?
2. I am unclear on how exactly energy efficiency is measured in order to incorporate it into the optimization process, which is critical for evaluating the paper's methodology (and might be useful for others). CodeCarbon typically requires assumptions about both the hardware being used and the regional energy mixture, as well as experimental measurements. The paper should provide some additional details and discussion on this.
3. Some of the hyperparameter importance measurements w.r.t. emissions (Figure 3) seem strange to me. In particular, how does the learning rate impact the emissions?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: I am curious about how the authors deployed shift-based computations instead of traditional addition and multiplication on the NVIDIA A100 GPU. Can this type of computation be extended to more networks, even though LLMs may be larger in size, what about smaller BERT models?
soundness: 3
strengths: 1. The paper offers detailed quantitative results, hyperparameter importance studies, providing a robust empirical basis for the proposed approach.

2. The motivation of the proposed method is to use shift computing to reduce the computational consumption of DSNNs. The article uses emissions as the target metric, which is interesting and has application value.

3. The work effectively showcases that AutoML can yield DSNN configurations that outperform baseline models in both performance and energy efficiency, emphasizing practical implications for low-resource settings.
summary: The paper explores the optimization of Deep Shift Neural Networks (DSNNs) using Automated Machine Learning (AutoML) for balancing performance and energy efficiency. The authors propose an approach that leverages Multi-Fidelity (MF) and Multi-Objective (MO) Hyperparameter Optimization (HPO) to identify Pareto-optimal configurations of DSNNs. Their methodology integrates tools like SMAC3 and CodeCarbon to simultaneously optimize accuracy and reduce carbon emissions. The experiments conducted on architectures such as ResNet20, MobileNetV2, and GoogLeNet across datasets like CIFAR10 and Caltech10.
weaknesses: 1. For readers who are not familiar with AutoML or optimization methods, the technicality and depth of this paper may be difficult to grasp. A more detailed introduction in the Background section and a clearer writing style would help convey the motivation of the article. For instance, in Section 3, the description of DSNNs only briefly introduces the weight transformation of DSNNs through Equation 3.3 and Equation 3.4 for the simple flip(·) function. However, as introduced in [1], the core calculation of the shfit function is not explained in the article, making it difficult for readers to understand the true characteristics of DSNNs. From a writing logic perspective, there is not a strong connection between the backgrounds in Sections 3.1-3.3, and there are several typos in the article, such as '$tildeW_q$' in line 143.

2. The core contribution of the paper is the ability to quantify the emissions impact of training and inference of DSNNs, a topic that is environmentally friendly. However, energy efficiency is dependent on the chosen computing platform. The authors conducted tests using the A100. The reliance on energy and emissions tracking tools like CodeCarbon is beneficial, but the paper could improve by discussing limitations or assumptions inherent in these measurements and potential variability due to different hardware.

 3. A small-size model like ResNet20 on A100 may not be comprehensive enough, and researchers would like to see the relationship between the computational energy of different types of neural network structures on various energy-efficient computing devices and the shift configuration.

[1] DeepShift: Towards multiplicationless neural networks.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: no
soundness: 3
strengths: AutoML based solutions can reduce the design cycle for DSNN model development.
Pareto-optimal design  is a standard way. However it offers flexibility for the performance-energy trade-off.
summary: The paper aimed to address the resource and environmental challenge introduced by the great computational cost of DL models. In particular, they focused on the DNNs using shift operations. They used multi-fidelity (MF) HPO with multi-objective optimization to find some pareto-optimal designs for DSNNs. With this approach, the performance is increased by 20% with 10% less carbon emission.
weaknesses: Lacks novelty. The major techniques HPO used for autoML is not new.  
The improvement is marginal (only 20% for performance and 10% for energy reduction). But acceptable when compared with human experts.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: Mentioned in weakness section.
soundness: 3
strengths: 1. The paper describes the background literature work in good details.
2. The paper discusses the details of results and importance of each choice in optimization landscape. (Section 5).
summary: The paper studies Deep Shift Neural Networks and how they can be used for computationally efficient and energy-friendly deployment. Particularly, the paper extends the prior work on DeepShiftQ and DeepShift-PS and uses a multi-fidelity, multi-objective optimization algorithm to better search the optimization space of deep shift networks while ensuring good task accuracy and energy consumption is reduced. The efficacy of the proposed algorithm is demonstrated on ResNet20 architecture.
weaknesses: 1.  The paper should include a section of how energy efficiency compares to tensor core operations with reduced precisions on modern GPUs (A100, H100). Since these GPUs have high specific dense tensor cores in SMs, these are very energy efficient in comparison to equivalent operations that execute on CUDA cores (this where bit shift operations would execute). If the motivation is design custom ASICS, then the paper should definitely include a section explaining this in depth.
2. The paper conducts experiments on very small number of networks that are specifically image recognition focused. The paper should definitely include more architectures specifically transformer architectures (vision transformers if the paper only wants to deal with vision related tasks).
3. Paper should include a pseudo code section describing how various components of the optimization algorithms used.
4. Figure 1 is hard to understand with few data points and huge space on x axis wasted (what is data point indicated by "x" corresponding to?).
5. The paper should include error bars in the results and use the results averaged over multiple seeds (3 seeds in this case). Since quantization methods often have higher variance, providing the mean and standard deviation of the reported numbers would provide a better understanding of the experimental results.
6. The paper seems hard to read and understand in one go for a new reader. I guess its because the core contribution requires dependence on too many methods from Background and these are not described with concrete examples in Background literature.


Nit:
The explanation of DeepShift-PS at line 146 can definitely be improved by adding an example or pseudo code.
Line 143 => tildeWq (looks like wrong text since the formula there on Line 144).

Meta-review: This paper presents an exploration into optimizing Deep Shift Neural Networks (DSNNs) for balancing performance and energy efficiency using Automated Machine Learning (AutoML) techniques. The authors propose a multi-fidelity, multi-objective hyperparameter optimization approach to identify Pareto-optimal configurations for DSNNs, aiming to reduce computational complexity and environmental impact.

One of the primary concerns raised by the reviewers is the lack of novelty in the optimization techniques applied. While the paper claims to offer flexibility for the performance-energy trade-off, the major techniques used for AutoML are not new, and the improvement in performance and energy reduction, though present, is considered marginal. The paper's contribution in terms of novelty is somewhat limited and could benefit from further clarification. The proposed optimization approach for DSNNs appears to directly apply existing optimization techniques without substantial adaptation or refinement specific to the DSNN topology or its unique optimization landscape. The comparison with existing methods and the generalizability of the findings were also noted as insufficient by some reviewers. The evaluation section is limited, as it currently includes results on only a few model architectures, which makes it challenging to draw robust conclusions about the effectiveness and generalizability of the proposed AutoML framework. The paper would benefit from testing the approach on a broader range of models and datasets to provide a more comprehensive assessment of its applicability and impact across different scenarios.

The reviewers expressed mixed opinions, with some leaning towards acceptance due to the relevance and potential of the work in the field of sustainable deep learning, while others were more skeptical due to the lack of innovation and concerns about the practicality of the proposed methods. The paper shows promise, but it would benefit from further refinement, particularly in addressing the concerns about hardware implementation, providing more robust comparisons with existing methods, and demonstrating the generalizability of the approach across different architectures and datasets.

This paper received an averaged score of 5.4, which is not competitive among this year's submissions. Given the balance of strengths and weaknesses, the final recommendation is to reject this submission in its current form. The paper has the potential to make a significant contribution to the field of sustainable deep learning, but it would benefit from an additional round of revisions that address the concerns about the novelty of the approach, the generalizability of the findings, and the robustness of the comparisons with state-of-the-art models.

Decision: Reject