Leveraging AutoML for Sustainable Deep Learning: A Multi- Objective HPO Approach on Deep Shift Neural Networks
Authors: Leona Hennig
Marius Lindauer
Venue: Accepted by TMLR ()

Abstract: Deep Learning (DL) has advanced various fields by extracting complex patterns from large
 datasets. However, the computational demands of DL models pose environmental and
 resource challenges. Deep Shift Neural Networks (DSNNs) present a solution by leveraging
 shift operations to reduce computational complexity at inference. Compared to common
 DNNs, DSNNs are still less well understood and less well optimized. By leveraging AutoML
 techniques, we provide valuable insights into the potential of DSNNs and how to design them
 in a better way. We focus on image classification, a core task in computer vision, especially
 in low-resource environments. Since we consider complementary objectives such as accuracy
 and energy consumption, we combine state-of-the-art multi-fidelity (MF) hyperparameter
 optimization (HPO) with multi-objective optimization to find a set of Pareto optimal trade-offs
 on how to design DSNNs. Our approach led to significantly better configurations of DSNNs
 regarding loss and emissions compared to default DSNNs. This includes simultaneously
 increasing performance by about 20% and reducing emissions, in some cases by more than 60%.
 Investigating the behavior of quantized networks in terms of both emissions and accuracy,
 our experiments reveal surprising model-specific trade-offs, yielding the greatest energy
 savings. For example, in contrast to common expectations, quantizing smaller portions
 of the network with low precision can be optimal with respect to energy consumption
 while retaining or improving performance. We corroborated these findings across multiple
 backbone architectures, highlighting important nuances in quantization strategies and offering
 an automated approach to balancing energy efficiency and model performance.

Published in Transactions on Machine Learning Research (07/2025)
Leveraging AutoML for Sustainable Deep Learning: A Multi- Objective HPO Approach on Deep Shift Neural Networks
Leona Hennig
Lhennig@ai, unw.oooooonhooooo.d.dd
Institute of Artificial Intelligence Leibniz Universily Hanover
Marius Lindauer
m. ww..idduurrdeee daai uni. wmi.hammover.com de
Instibute of Artificial Intellignnce
Leibniz University Hanover
L3S Research Center
Reviewed on OpenReview: /ttps://openreee net/ forum? id= ld=bkh11DD
Abstract
Deep Learning (DL) has advanced various fields by extracting complex patterns from large datasets. However, the computational demands of DL models pose environmental and resource challenges. Deep Shift Neural Networks (DSNNs) present a solution by leveraging shift operations to reduce computational complexity at inference. Compared to common DNNs, DSNNs are still less well understood and less well optimized. By leveraging AutoML techniques, provide valuable insights into the potential of DSNNs and how design them in better way. We focus on image classification, in low-resource environments. Since we consider complementary objectives such as accuracy and energy consumption, we combine state-of-the-art multi-fidelity (MF) hyperparameter optimization (HPO) with multi-objective optimization to find a set of Pareto optimal trade-offs on how to design DSNNs. Our approach led to significantly better configurations of DSNNs regarding loss and emissions compared default DSNNs. This includes simultancously increasing performance by about 20% and reducing emissions, in some cases by more than 60%. Investigating the behavior of quantized networks in terms of both emissions and accuracy, our experiments reveal surprising model-specific tradc-offs, yiclding the greatest energy savings. For example, in contrast to common expectations, quantizing smaller portions of the network with low precision can be optimal with respect to energy consumption while retaining or improving performance. We corroborated these findings across multiple backbone architectures, highlighting important muances in quantization strategics and offering automated approach to balancing energy efficiency and model performance.
task in computer vision, especially
1 Introduction
Deep Learning (DL) promising approach to extracting information from large datasets with complex structures. This includes performing computations in loT environments and on edge devices (Li al. 2018 Zhou et al. 2019), which can come with strict limitations energy consumption. This is especially true for the ever-increasing size and performance of such models due to the progress in science and industry. Running these models is not free of computational costs (Sze et al. 2017). and minimizing this cost directly affects the environmental impact of a model (Schwarlz el. al. 2020). Even resource consumption should not be considered crucial in view of environmental impact, efficiently designed neural networks free up resources that can be used for other tasks, c.g., edge computing or computations for advanced driver assistance systems (Howard al. (217)). With our approach, we contribute DL by minimizing its environmental footprint and allowing applications in low-resource settings.

---

Published in Transactions on Machine Learning Research (07/2025)
In this context, we focus image classification, one of the core tasks in computer vision and particularly relevant. for low-power applications. Image classification is widely common in scenarios where resources are limited, such as edge computing, automated driving, and industrial production environments. These applications require fast, accurate, and energy-efficient models that can process visual information in real time. Because it is well understood and broadly benchmarked, image classification enables reproducible evaluation of architectural and optimization strategies with minimal confounding of task-specific variables.
Of particular interest to are Deep Shift Neural Networks (DSNNs), which offer great potential to reduce power consumption compared to traditional DL models, e.g., by reducing the inference time by factor of 4 Elloouhh et al., 2021). Instead of expensive floating-poing arithmetic, they leverage cheap shift operations specifically, bit shifting as the computational unit, which boosts efficiency by replacing costly multiplication operations in convolutional networks. Although DSNNs offer great promise, so far, different design decisions, including training hyperparameters and shift architecture, have not been well studied, and there is little knowlodge about their full potential. We suspect that the configuration of DSNNs has huge impact on both performance and computational efficiency.
One of the key challenges with DSNNs is determining the appropriate level of precision for shift operations to minimize quantization errors without excessively increasing the computational load. To address this challenge, we propose to apply automated machine learning (AutoML) to DSNNs to find their optimal configuration. This is achieved by hyperparameter optimization (HPO) (Bischl et al. 2023 and a neural architecture search on a macro-level (Elsken et al.] 2019). Integrating multi-fidelity (MF) and multi-objective optimization (MO) techniques (Belakaria et al.J 2020) facilitates an optimal exploration of the configuration space that trades off' predictive performance and energy consumption (Deb) 2014). To this end, we extended the SMAC3
approach (Lindauer et al. 2022 state-of-the-art HPÒ package (Eggensperger et al. 2021), so that its MO implementation effectively balances the trade-off between achieving high predictive accuracy and minimizing energy consumption. Using tools such as CodeCarbon (Lacoste et al. 2019 Lottick et al. 2019) during the training and evaluation phases provides insight into the energy consumption and carbon emissions associated with each model configuration. The MF aspect allows for the efficient use of computational resources by evaluating configurations at varying levels of approximation. Our work is in the spirit of Green AutoML (Tornede et al. 2023 considering efficient AutoML to gain insight into the design of efficient DSNNs
Contributions
Overall, we contribute to Green AutoML w.r.t. DSNNs by:
1. Specifying the first configuration space tailored to DSNNs, which is efficiently optimized by a combination of multi-objective and multi-fidelity AutoML approaches;
2. Empirically exploring how specific design choices in DSNNs lead to different trade-offs between accuracy and energy efficiency, enabling stakeholders and researchers to leverage these findings to develop energy-efficient applications that maintain high computational accuracy; and
3. Identifying specific configurations of DSNNs that surpass the baseline results in both dimensions of the performanco-fficiincyy optimization problem.
The corresponding repository is available at https://github com/ auttma//uttooSSI
2 Related Work
Both multi-fidelity optimization (MF) (Bicchl et al., 2023) and multi-objective optimization (MO) (Morales Hernández et al. ) for AutoML have gotten lot of traction in recent years. The combined integration of multi-fidelity multi-objective optimization (MFMO) has seen some advancements to enhance the efficiency of model training while minimizing environmental impact. [Beakariia et al. (2020) proposed an acquisition function based on output space entropy search for multi- filtiiiiiiiity objecivo Bayesian optimization (MFMO- BO-OSES). Their method addresses the calooration-xpllitaaati data points that significantly reduce the entropy of the Pareto front. This approach enables more strategic sampling decisions and leverages lower-fidelity evaluations to approximate the Pareto front effectively, aligning
dilemma by prioritizing the acquisition of
2

---

Published in Transactions on Machine Learning Research (07/2025)
with the sustainability goals of Green AutoML. Similarly, Schmucker et al. (020) considered a combination of multi-objective and multi-fidelity optimization but focused on fairness as the second objective. With our MFMO approach, we contribute an algorithm tailored specifically for performing efficient HPO tasks using BO, directly minimizing emissions in the process.
A well-explored technique for reducing the computational complexity of neural networks themselves is network quantization. It involves lowering the precision of weights and activations, which decreases the model's memory footprint and accelerates inference. Works by Courbariaux et al.] (2015) and Rastegari et al. (2016), for example, have demonstrated that techniques such as BinaryConnes and XNOR-Net not only reduce computational requirements but also maintain near-staaccoo-thh--rrt performance, underscoring the potential of quantization to balance performance with computational efficiency. This has also been transferred into the field of LLMs, where 1-Bit transformer architectures are used to address the challenges of increasing model size (Wang et al., 2023). Strongly related to network quantization, Deep Shift Neural Networks (DSNNs), proposed by Elhoushi et al. (2021). represent an advancement towards quantization combined with efficient operators. DSNNs employ bitwise shift operations instead of traditional multiplications, thus further reducing the computational overhead and power consumption. This innovation is particularly crucial for deploying Deep Learning models in power-sensitive or resource-constranned environments, further contributing to the envirommental sustainability of AI technologies.
DSNNs offer a more balanced trade-off between computational efficiency and accuracy compared to binary methods like BinaryConnect and XNOR-Net by replacing multiplications with bitwise shifts, which are more power-efficient yet maintain greater precision, reducing the accuracy loss typically associated with binary quantization. This allows DSNNs to achieve better performance in resource-constrained environments while still minimizing computational overhead (Elhoushi et al. 2021
AdderNet (Chen et al. 2020a is another approach for reducing the amount of computationally expensive multiplications during network training and inference by using the f--norm distance between input and filter vectors to compute activations. There are efforts in mixed--recisson quantization, where different bit-widths are assigned to different. layers or channels of the network. It. allows for higher precision where necessary and lower precision where possible, optimizing the trade-off between accuracy and efficiency (Motetti et al.) 2024). Similarly, ternary quantization (TTQ) constrains the weights to three discrete valuos: {-1,0,+1}, To mitigate the accuracy loss from reduced precision, TTQ introduces learned scaling factors, allowing networks to maintain performance comparable to their full-precision counterparts while significantly reducing memory and power usage [Rokh et al. 2023). HPO combined with DSNNs further optimizes both performance and resource efficiency by tailoring shift depths and quantization strategies, allowing for fine-tuned control over energy consumption and accuracy in constrained environments, making them ideal for mixed-precision and quantization-awar neural network applications.
3 Background
The following chapter introduces foundational concepts for our approach.
3.1 Deep Shift Neural Networks
A Deep Shift Neural Network (DSNN) is an approach to reduce the computational and energy demands of Deep Learning (Elhoushi et al., 2021). DSNNs achieve considerable reduction in latency time by simplifying the network architecture such that they replace the traditional multiplication operations in neural networks with bit-wise shift operations and sign flipping, making DSNNs suitable for computing devices with limited resources. There are two methods for training DSNNs (Elhoushi et al. 2021 DeepShift-Q (Quantization) and DeepShift-PS (Powers of two and Sign). DecpShift-Q involves training regular weights constrained to powers of 2 by quantizing weights to their nearest power of two during both forward and backward passes. In DeepShift-Q, the weights are quantized to powers of two by rounding the logarithm of the absolute weights to the nearest integer. This process simplifies the weight representation and ensures compatibility with bitwise shift operations. The sign is then applied to preserve the original weight polarity. DeepShift-PS directly includes the values of the shifts and sign flips as trainable hyperparameters, offering finer control over weight.
3

---

Published in Transactions on Machine Learning Research (07 (07/2025)
adaptation. This approach removes the need for explicit rounding during training, potentially leading to improved precision al the cost of additional parameter updates.
The DeepShift-Q approach obtains the sign matrix S from the trained weight matrix W as S sign W). The power matrix P is the base-2 logarithm of W's absolute values, ie., P= logz([W]). After rounding P to the nearest power of two, P = round(P), the quantized weights W, are calculated by applying the sign S:
Wg flip(2?.S) .S)
(3.1)
The DeepShift-PS approach optimizes neural network weights by directly adapting the shift (P) and sign (5) values. The shift matrix P is obtained by rounding the base-2 logarithm of the weight values, P round(P), and the sign flip S is computed as $ sign( (round(S)). Weights are calculated as
Wp - flip(2",5) .))
(3.2)
where the sign flip operation S assigns values of -1, , O, +1 based on S. Directly training shift and sign values could allow for more precise control in optimizing the network's compulational efficiency by reducing mathematical imprecision. However, training the floating point weights and only rounding them during the forward and backward passes might increase the precision and reduce the error in training the weights.
3.2 Hyperparameter Optimization
The incrcasing complexity of Deep Learning algorithms enhances the need for automated hyperparameter oplimization (HPO) to increase model performance (Bischl et al. 2023). Consider a dataset D = {(z,, wz) w)){ E D C. X x y, where X' is the instance space and y is the target space, and a hyperparameter configuration space A= =f),..... ) ; L E N. In our work, M denotes the space of possible DSNN models. The datasot D split into disjoint training, validation, and test sets: Dirain. Deal, and Diest respectively. An algorithm A: D A- - M trains a model M E M, instantiated with configuration of L, hyperparameters sampled from A, on the training data Drrain. The performance of the algorithm is assessed via an expensive-to-evaluuee loss function L: .M XD= + R, which involves both the training on Drrain and the evaluation of the model on Deat- The optimization objective of HPO is to find configurations X E A with minimal validation loss L:
X E arg min L [(A(Drrainn) (Drriin A), Devt).
(3.3)
XEA
Finally, the model's final performance is assessed on Drest-
3.3 Bayesian Optimization
For a given dataset, Bayesian Optimization (BO) for HPO is a strategy for global optimization of black-box loss functions L: MM - TR that are expensive to evaluate (Jones et al., 1998).
BO uses probabilistic surrogate model S to approximate the loss function, commonly given by Gaus- sian Process or Random Forest (Rasmussen & Williams| 2006 Hutter et al 2011 Shahriari et al. 2016). An acquisition function : A R guides the search for the next optimal evaluation points by balancing the exploration-exploitatio trade-off, based on the set of previously evaluated configurations L(M,, .),,. (Am-1, ((Mmm_..
at time m. Common choices for acquisition functions include ex- pected improvement (FT) (Jones et al. 1998) since it calculates the expected improvement the objective function value and guides the scarch towards regions where improvements are most likely.
Eatropy-based methods like Entropy Search (ES) (Hennig & Schuler 2012) and Predictive Entropy Search (PES) Hhernandez-Lobatoo Lobato et al.) 2014) aim to reduce the entropy of the posterior distribution of the maximizer, focusing on information-rich regions. The Knowledge Gradient (KG) (Frazirr et al. 2009) offers a strategy for maximizing the expected improvement of the objective, considering all polential oulcomes, valuable in scenarios with noisy measurements. BÔ is particularly well-suited for hyperparameter optimization in Deep Learning, where evaluating the performance of a model configuration can be computationally expensive because of the training of each configuration. BO sample-efficient in evaluating C on only few configurations.
4

---

Published in Transactions on Machine Learning Research (07 (225))
3.4 Multi-Fidelity Optimization
To reduce the cost of fully training multiple DSNN conligurations, employ multi-fidelily (MF) approach (Li et al., 2017), which is a common strategy in AutoML to navigate the trade-off between performance and approximation error (Hutter et al. 2019). MF approaches train cheap-to-evaluato proxies of black-box functions following different heuristics, e.g., allocating a small number of epochs to many configurations in the beginning and training the best-performing ones on an increasing number of epochs. Formally, we define space of fidelities F and aim to minimize function F F (Kandasamy et al. 2019
min F(A). XEA
(3.4)
We approximate E F, using a series of lower-fidelity, i.c., less expensive approximations {f(A),,..... ..... f(A); F(A)}, where j denotes the total nunber of fidelity levels. The target function F E F corresponds to the loss function L of HPO and BO. The allocated resources for evaluating a model's performance at various fidelities are referred to as budget, e.g., training DNN for only n E N epochs instead of until convergence. MF typically assumes that the highest fidelitics approximate the black-box function best. The longer a model is trained, the more accurate ils approximation of an underlying function gets.
3.5 Multi-Objective Optimization
Multiiobjective optimization (MO) addresses problems involving multiple, often competing, objectives. This approach used in scenarios where trade-offs between two or more conflicting objectives must be navigated, such as in the context of DSNNs, enhancing accuracy alongside reducing energy consumption. MO aims identify Pareto optimal solutions (Deb, 2014). New points are added based the current observation datasel Dobs {(A1, ((M,..)..... (An: ((M })} at time m + 1. These points augment the surface formed by the non-dominated solution set D which satisfies the condition for d objective variables and a loss function
L = CC(C,,... Ra), where Lg corresponds to the loss regarding objective le (Wada & Hino 2019):
aA, (A, (())) E D; C Da, (X, (())) E Dn E (I,...,d}: (x S Lx()).
(3.5)
W.l.o.g,, assume the minimization of all objectives. The observation dataset Dots is iteratively updated to search for solutions that approximate the Pareto front.
4 Approach
Our goal is to provide deeper insights into the structure of Deep Shift Neural Networks (DSNNs). We optimize these networks with respect to both performance and efficiency, highlighting how their hyperparameters influence the optimization process and resulting trade-ofls.
4.1 Configuration Space Exploration
The foundation of our approach lies in defining and exploring a robust configuration space tailored specifically to DSNNs. This space includes a range hyperparameters that influence the network's performance and energy efficiency. Key hyperparameters under consideration include:
1. Shift Depth: determines the number of network layers converted to employ shift operations, replacing conventional floating point operations and thereby reducing computational overhead.
2. Shift Type: selects the method of shift operation, either quantization (DeepShift-Q) or direct training
of shifts (DeepShift-PS), impacting the network's training dynamics and inference efficiency.
3. Bit Precision for Weights and Activations: inffuences the network's accuracy and the gramularity of its computations, affecting both performance and power consumption.
4. Rounding Type: affects how weight adjustments are handled during training, with options for deterministic or stochastic rounding, each offering trade-offs in terms of stability and performance.
5

---

Published in Transactions on Machine Learning Research (07/2025)
Table details the configuration space for ResNet20 model adapted for DSNNs, outlining the range and default values of each hyperparameter considered in our study.
4.2 Multi-Fidelity Multi-Objection Optimization Framework
To computationally enhance DSNNs via AutoMI. we employ multi-fidelity optimization (MF), see Section 3 A well-known MF algorithm is successive halving (Jamieson & Talwalkar 2016), where configurations are trained on an initial small budget b,. It addresses the trade-off between by and The, or between approximation eιror and exploration inherent in successive halving, using the HyperBand algorithm for MF. HyperBand (Li et al., 2017) runs successive halving in multiple brackets, where each bracket provides a combination of ne and fraction of the total budget per configuration so that they sum up to the total budget. We extend this to multi-fidelity multi-objective optimization (MFMO). We simultaneously address the accuracy of the model well as its energy consumption using two-dimensional objective function:
LMO: A R2 (MO(A) (Cinss (A), Lemission (A)),
(4.1)
where, given a configuration À E A, (A) aims to minimize the loss, enhancing the model's accuracy, and Cemission (A) seeks to minimize the energy consumption during training and inference, promoting environmental sustainability. We aim to solve the following optimization problem:
arg min (M) xEλ
(4.2)
Table 1: Configuration search space of ResNet20. The first half includes commonly used training hyperparameters for deep learning, whereas the second half is specific to DSNNs.
Hyperparameter
Scarch Space
Default
Standard Training Hyperparamcters
Batch Size
[32, 128]
128
Optimizer Learning Rate
(SGD, Adam, Adagrad, Adadelta, RMSProp, RAdam, Ranger)
SGD
[0.001, 0.1]
0.1
Momentum
[0.0, 0.9]
0.9
Weight Decay
[1e-6, 1e-2]
0.0001
DSNN-Specific Hyperparameters
Weight Bits
[2,
Activation Integer Bits
[2, 32]
16
Activation Fraction Bits
[2, 32]
16
Shift Depth
[0, 20]
20
Shift Type
(Q, PS}
PS
Rounding
(doterministic, stochastic]
deterministic
Our approach leverages the MFMO framework to efficiently explore the configuration space using computa- tionally inexpensive proxies for full training. This enables broader and more tractable exploration of the hyperparameter landscape within practical resource constraints. The method is inherently architecture- and task-agnostic, roquiring no modifications when applied to different models or datascts, aside from adjusting the DSNN's quantization mechanism to align with the specific target architecture and task.
4.3 Algorithmic Implementation of MFMO
We use the ParEGO algorithm Knowles 2006) to compute Pareto optimal configurations. It transforms the multi-objective problem into series of single-objective problems by introducing varying weights for

---

Published in Transactions on Machine Learning Research (07 (07/2025)
the objectives in each iteration of HyperBand, thus optimizing different scalarization per evaluation to approximate the Pareto front. The resulting single-objective optimization function can then be evaluated in an MF setting. All configurations having survived a successive halving bracket are checked against the current Pareto front approximation and the Pareto set is updated if necessary. The computational strategy initially involves computing a broad array configurations and leveraging the successive halving method to efficiently narrow down the field to the most promising candidates. We specifically target solutions that represent both extremes of the Pareto front, those that excel in one objective at the potential expense of the other, and configurations that provide balanced compromise between the two objectives. We included pseudocode of our algorithmic implementation in Algorithm in the appendix.
5 Experiments
In the following section, we detail the setup and methodology used to evaluate our approach discussed in Section + focusing on optimizing Deep Shift Neural Networks (DSNNs) through multi-fidelity, multi-objective optimization (MFMO), and extending the DSNN objective function to multi-objective compute a Pareto front for optimality regarding performance and efficiency. We discuss how approach successfully navigates the model performance and environmental impact trade-ofTs. From this, we gain insights into DSNNs and how specific design choicos might affect their performance. By identifying optimal configurations for both or either objectives, we draw conclusions about how the DSNN-specific hyperparameters in the network architecture interact with each other.
5.1 Evaluation Setup
We train and evaluate our models on the CIFAR10 dataset (Krizhessyy et al., 2009 and the Caltech101 dataset (Fei-Fei et al.] 2004]. Our experiments conducted on the NVIDIA A100, widcly used GPU, slandard in research and industry for some time. Its computational capabilities and availability in many perfoppeformmacc clusters make it popular choice in the machine learning and deep learning communities. Using the A100, we ensure that our findings are broadly applicable and relevant to real-world scenarios, aligned with hardware commonly utilized for training and deploying advanced neural networks.
For hyperparameter optimization (HPO) with multi-fidelity optimization, we extend SMAC3 (Hutter et al., 2011 Lindauer et al., )022), as well-known Mtate-of-theaad HPO package (Eggensperger et al. 2021). As starting point, chose the well-known ResNet20 (He et al. 2016) architecture used by Elhoushi et al. (2021). Overall, this architecture is well understood and allows us to study DSNNs with few confounding
factors. Additionally, we evaluate our approach using the well-known GoogLeNet (Szegedy et al. 2015 and MobileNetV2 architectures (Sandler et al. 2018). We follow the model implementation of Elhoushi et al. (2021) to ensure comparability. The configuration space given in Table for which we focus on the DSNN Sspccifci hyperparamoters (lower part of the table) and general training hyperparametors (upper part).
The fidelities are the number of epochs.
For multi-objective optimization, we aim to compute Pareto front of optimal configurations for performance and energy consumption. To incorporate the environmental impact into our HPO workflow, we use the CodeCarbon emissions tracker (Lacoste et al. 2019 Lottick el al. 2019) to track carbon emissions from computational processes, in our case inference, by monitoring energy use and regional energy mix in kgColeq, grams of CO2 equivalents. We use power consumption as a proxy for CO2 emissions, ackuowledging that external factors such as energy source and grid transmission losses are not directly controlled. These emission values incorporated into SMAC3 alongside DSNN's performance metric. The loss is 1- accuracy.
5.2 Results
5.2.1
Quantitative Results
We first discuss the quantitive results, then the importance of the optimized hyperparameters, and finally, the implications for the configuration space. Note that we focus on the insights gained regarding DSNNs and not on how efficient our (or others') HPO approach is. In Figure 2 we present the computed Pareto fronts of
7

---

Published in Transactions on Machine Learning Research (07/2025)
ResNet20,, MobileNetV2 and GoogLeNet architecture, optimized with multi-fidelity multi-objective (MFMO) framework, on the CIFARIO and Caltech101 datasets. Shown is a diverse set of optimal configurations that either minimize or balance the primary objectives of model accuracy and energy consumption. These are aggregated results over three seeds. The Pareto front shown results from aggregating the three individual Parelo fronts and extracting the Pareto optimal points. The default value the mean over the loss and emissions of the default configuration on the three seeds. The Pareto fronts in Figure 2 depict how cach configuration performs relative to the others within the defined hyperparameter space. The configurations were evaluated regarding classification loss and emissions emitted during inference of the model instantiated with the respective configuration. Throughout this paper, when referring to emissions, measured emissions at inference. Although we expect that Elhoushi et al. (2021) optimized their hyperparameters at least manually, we can show that our AutoML approach found even better trade-ofTs of the two objectives. The default configuration for the DSNN, as designed by Elhoouhi et al. (2021). is fact not part of the Pareto front in Figure 2 This holds for all architcctures on both datasots. This mcans that there are better configurations that dominate the default configuration regarding both loss and emissions on MobileNet (Figures 2a and
2b
GoogLeNet (Figures 2c and 2d] and ResNet20 (Figures 2e and 21 Caltech101 and CIFAR10.
Having closer look at Figure 2i the underlying goal of our MFMO optimization remains to balance performance and ef- ficiency. Hence, the configu- rations the bottom left of the Pareto front especially relevant since they minimize both objectives simultaneously instead of heavily prioritizing either. There is an absolute re- duction loss of up to 20% these configurations compared to the defaults. At the same time, relative emission reduc- tion ranges from aрprox. 10% to more than 60% in Figures 2b and 2e Additionally, we achieved a maximum loss re- duction of about 20% for the ResNet20 CIFAR10. This
0.0020
Emissions
0.0019
Aggregated Pareto Front
Uncertainty Bounds
0.0018
Median Pareto Front
Average Default
0.0017
0.0016
0.6
0.7
0.8
0.9
Loss
Figure 1: Pareto front for E'fficiettNet V2 on CIFAR10 over multiple seeds. We show the loss % and the emissions in kgCO2eq. The plots include median Pareto fronts with uncertainty bars, as well as an aggregated Pareto front of Pareto-optimal solutions across all runs. The star denotes the averaged performance of the default DSNN configuration.
validates the need for proper HPO tuning since we found better configurations that take the energy-effciennt DSNNs a significant step further by improving their accuracy and energy consumption.
Additionally, we optimized an ViffcientNeeVV--aeeeee DSNN on CIFAR-10 to evaluate whether our method can yield further improvements when applied to an architecture already optimized for efficiency, thereby testing robustness and generalizability. The results are shown in Figure [ with the corresponding Parcto-optimal solutions in Table 10 in the appendix. Again, we were able to identify multiple configurations that surpass the default DSNN w.r.t. prediciton performance and emissions at inferece, using our MOMF approach. While Efficient V2 TTn & Le (021) is inherently designed for optmizing for performance and efficiency, using compound scaling, the need for carefully tuning the quantization hyperparameters is apparent. Compared to the default. NfficientNetV2, we were able to reduce emissions by more than 20%.
5.2.2 Hyperparameter Importances
Another crucial aspect is the analysis of hyperparameter importance to learn their influence on a model and lay the foundation of our DSNN design insights in the next subsoction. We use DeepCAVE (Sass al., 2022) for analyzing the Pareto front of our MFMO analysis in Figure 2f and computing the hyperparameter importance using TANOVA (Hutter et al. 2014).
8

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Aggregated Pareto Front
0.0016
Average Default
0.00065
Uncertainty Bounds
0.0014
Emissions
0.00060
Median Pareto Front
Emissions
0.0012
0.00055
0.0010
0.0008
0.00050
0.0006
0.00045
0.4
0.6 Lass
0.8
0.2
0.4
0.6
0.8
Loss
(a) MobileNet on Caltech101
(b) MobileNet on CIFAR10
0.0014
0.0026
0.0013
Emissions
Emissions
0.0012
0.0024
0.0011
0.0010
0.0022
0.0009
0.0020
0.4
0.5
0.6
0.7
0.8
0.9
0.14
0.i5
0.16 Lass
0.17
0.18
Loss
(c) GoogLeNet on Caltech101
(d) GoogLeNet on CIFAR10
0.00080
0.0012
0.00078
Emissions
0.0010
Emissions
0.00076
0.00074
0.0008
0.00072
0.0006
0.00070
0.00068
0.0004
0.5
0.6
0.7
0.8
0.15
0.20
0.25
0.30
0.35
Loss
Loss
(e) ResNet20 on Caltech101
(I) ResNet20 on CIFAR10
Figure 2: Comparison of Pareto fronts for MobileNet, GoogLeNet, and ResNet20 on Caltech101 and CIFAR10 datasots over multiple scods. We show the loss % and the emissions in kgCO2eq. The plots include median Pareto fronts with uncertainty bars, as well as an aggregated Pareto front of Pareto-optimal solutions across all runs. The star denotes the averaged performance of the default DSNN configuration.

---

Published in Transactions on Machine Learning Research (07 (025))
FANOVA fits random forest surrogate model to the hyperparameter optimization landscape and decomposes the model's variance into components corresponding to hyperparameters. This allows IANOVA to estimate the marginal impact of individual hyperparameters or pairs of hyperparameters. For an extension to MO optimization, fANOVA can be applied to cach objective's performance surface separately. The hyperparameter importances are then computed for different weightings of the objectives.
Importance
0.3
0.2
0.1
shift depth
momentum
weight, decay
activation fraction, bits
activation Integer bitts
weight bits
shift. type
rounding
(a) Hyperparameter importance with respect to loss.
0.25
Importance
0.2
0.15
0.1
0.05
activation Integer. bits
activation. fraction bits
momentum
rounding
weight, decay
shift. depth
weight bits
(b) Hyperparameter importance with respect to emissions.
Figure 3: Hyperparameter importance according to fANOVA for ResNet20 on CIFARIO. (a) Importance with respect to loss. (b) Importance with respect to emissions.
Figures 3a and 3b show the hyperparameter importances of a ResNet20 on Cifar10 w.r.t. loss and emissions, respectively. The MO-FANOVA analysis for different weightings the objectives loss and emissions can be found in Figure TT the appendix. The most important DSNN-specific hyperparameters for emissions Figure 3b include activation integer and fraction bits. This hints at the precision of the activation quantization being the most controlling factor for energy elficiency. Naturally, precision key factor since it controls the amount of operations in the network. Regarding loss in Figure 3a the shift depth is the most important hyperparameter. The proportion of the network converted to perform shift operations naturally controls the amount of information retained in the network. This is crucial for the overall performance of the network.
Notably, the shift type has a low importance value Figure Ba and is not among the most important hyperparameters in Figure 3b This could indicate that the shift type is marginally relevant for both objectives. practice, this insight could aid in model training by allocating fewer resources to tuning this hyperparameter, allocating resources tailored to the case. This is further supported by looking Tables
10

---

Published in Transactions on Machine Learning Research (07/2025)
About 50% of the Pareto optimal configurations use either shift type, meaning they are not leaning toward either to maximize either objective.
to 10
With the analysis of hyperparameter importance in our study, we offer a baseline of hyperparameters to include for future training and inference purposes. Inchuding only the significant ones is a promising way of further boosting the energy efficiency of the DSNNs and the optimization process (Probst et al. 2019 Additional plots showing the hyperparameter importance for each architecture can be found in Appendix AA. DpNN-specific hyperparameters consistently place among the most influential, alongside weight decay and learning rate. This consistent importance underscores the relevance of understanding and optimizing DSNNs.
5.2.3 Insights into the Design of DSNNs
When looking at the configurations of the ResNet20 architecture on CIFARIO in Table $ (see Appendix for similar tables for the other architectures and datasets), most solutions have a surprisingly small shift depth E {1,3}, compared to 20 as the maximal value and the setting of the default. At the same time, the number of activation fraction bits is often rather high. This leads to the assumption that the bulk of information is retained in the fraction part of the activation value. valid expectation since we used batch normalization in the ResNet20, same as Elhoushi et al. (202I). In batch normalization, the layer inputs are re-scaled and re-centered using the mean and variance of the corresponding dimension (loffe & Szegedy) 2015). This usually leads to small weights, highlighting the importance of activation fraction precision, which is higher in Pareto optimal configurations. It ranges from 8 to 32, likely a contributing factor to the cmission reduction.
While we initially hypotee a direct proportionality between shift depth and emission savings, our analysis reveals a more nuanced relationship between hyperparameters, especially the shift depth and the bit precision in weights and activations. These hyperparamcters interact in a non-lincar manner, jointly influencing both energy consumption and model performance in ways that are not immediately intuitive. The results of our Pareto front analysis indicate that the choice of shift layers, which we expected to correlate positively with performance gains and negatively with loss, does not exhibit such a straighttfrrwad relationship. Instead, our findings highlight the intricate dependencies among architectural design choices, hyperparameter configurations, and their combined impact on energy efficiency. Notably, increasing the number of shift layers does not consistently result in greater emission savings. Weight precision also influences emissions, but no consistent trend was observed in the ResNet20 configurations, indicating the need for datasct-specifie tuning.
These insights are corroborated by looking at the additional experiments in the appendix. Additionally to the Resnet20 on CIFAR10, we computed the Pareto fronts of our MOMF analysis MobileNetV2 and GoogLeNet on CIFARI0. The corresponding results are shown in Figures 2b and 2d The overall Pareto optimal configurations from multiple seeds can be found Appendix A.1 in Tables and
Again, the shift depths are generally very low, either one or three, with two exceptions of seven and fourteon. The number of activation fraction bits is usually close to the upper bound of 32 bits.
We have also computed Pareto fronts of our MFMO approach on the Caltech dataset, for ResNet20, MobileNetV2 and GoogLeNet. The results can be seen in Figures 2a 2c and 2c] The configurations on the Pareto fronts are detailed in Tables 7 g and g Again, the vast majority of configurations have a low shift depth in the range of 8 E 21, 4, 5, 6). Only two GoogT.eNet. configurations have shift depth of eight and 9. This is still relatively low, given that GoogLeNet is a complex architecture with 22 layers in total. As with the previously discussed results, this contributes to the reduction of emissions while not impacting the performance. Generally, the number of activation fraction and integer bits increases with lower shift depth and vice versa. This confirms our previous findings from the thoroughly discussed ResNet20 on CIFAR10.
In Figures 3a and 3b| the learning rate is surprisingly identified as important hyperparameter both for loss and emissions. To shed light on how the learning rate is related to emissions, we investigated this further. To this end, we instantiated ResNet20 on CIFARIO with the default configuration, but varied the learning by 100 values randomly sampled from the configuration search space. Analyzing the resulting emissions leads to a mean value of 0.00106 kgCO2eq, which exceeds the range of our Pareto fronts. The standard deviation 0.00156, which is itself close to the mean value. This shows, indeed, that randomizing the learning rate introduced great variability in the energy measurements. As a control measurement to rule out other confounding factors, we measured the energy required for inference for a model instantiated with the default.
11

---

Published in Transactions on Machine Learning Research (07/202) 20225)
0.016
0.014
0.012
Emission Values
0.010
0.008
0.006
0.004
0.002
0.000
Random
Fixed LR
Figure 4: Emissions from different learning rates.
learning rate 100 times over. As expected, this leads to mean of 0.00076 kgCo2eq, matching the evaluation of the default configuration in Figure 21 and very low standard deviation of about 0.00006 (see Figure 4
This leads to the conclusion that the learning rate is indeed an influential HP on the emissions since it can drastically lead to an increase in emissions when chosen suboptimally. Our AutoMI approach offers a solution to exactly this problem, since we aim to automate the identification and optimal choice of hyperparameters.
5.3 Transferability of Configurations
Understanding whether optimal configurations found in one setting remain effective in others is critical for practical and sustainable AutoML, In particular, the ability to transfer configurations across datasets can lead to substantial reductions in computational cost and associated emissions. Tf previously opt configurations remain near-optimal despite changes in the data, users avoid restarting the entire optimization process, thus saving both time and resources. To investigate this, we evaluated the Pareto-optimal configurations ified for a ResNet20 model on CIFAR-10 trained on ImageNet100, a subset of ImageNet, Tian et al. (2020). We measure loss and emissions at inference. The results, shown in Figure demonstrate that several of the original Pareto configurations remain competitive on Imagenet100, continuing to outperform the default configuration. Notably, the default remains Pareto-dominated while the transferred configurations yield improvements in both loss and emissions. This suggests degree of robustness and transferability, which could be leveraged in practice to reduce the frequency of full re-optimization runs.
6 Conclusion
In this work, we presented our Green AutoML approach towards the sustainable optimization of DSNNs through multi-fidelity, multi-objective (MFMO) HPO framework. Our approach effectively addressed the critical intersection between advancing the capabilities of Deep Learning and environmental sustainability, By leveraging AutoMI tools and integrating the environmental impact as an objective, we adeptly navigated the trade-off between model performance and efficient resource utilization. Our experimental results focused on a better understanding of DSNNs. We successfully optimized DSNNs to achieve higher accuracy while minimizing energy consumption, surpassing the default configuration scttings in both aspects. Through systematic experimentation, we identified key hyperparameters that significantly influence performance and emissions, such shift depth and number of weight bits.
12

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Configurations Pareto Front
0.0040
Default
0.0039
Emissions
0.0038
Config 82
0.0037
Config 49
Config 116
0.0036
Config 38
Config 28
Config 124
0.0035
Config 71
0.70
0.75
0.80
0.85 Loss
0.90
0.95
Figure 5: ResNet20 trained on Imagenet100, instantiated with Pareto-optimal configurations from CIFARIO (sce Table 4.. We show the loss in % and the emissions in kgCOzeq.
By optimizing these hyperparameters, our MFMO approach did not just improve one dimension of the problem concurrently enhanced both model loss and energy efficiency, showcasing balanced improvement across essential performance metrics. We have thoroughly explored the configuration space for DSNNs, introduced a Green AutoML approach for efficiency-driven model development (Tornede et al.) 2023 and provided valuable insights into the design decisions impacting DSNN performance.
The generalizable insights from our work extend beyond the need to carefully select bitwidths for quantization. Our results reveal that optimal configurations for DSNNs are often counterintuitive and highly dependent on the intricate relations between hyperparameters. For example, we found that low shift depths often achieve superior trade-offs between accuracy and energy efficiency, challenging assumptions about full quantization of networks. Additionally, our analysis highlights the importance of prioritizing specific hyperparameters for different objectives, providing a targeted approach to DSNN optimization. These findings are consistent. across multiple architectures and datasets, demonstrating their broader applicability. Overall, study shows the need for systematic, automated approach to detecting these insights, which are relevant not only for DSNNs but also for other quantized and resourco-cfficient neural networks.
Limitations While our use of CodeCarbon provides valuable insights into the emissions impact of training and inference for DSNNs, it is important to acknowledge the limitations and assumptions inherent in these measurements. CodeCarbon relies real-time power draw metrics from tools like nvidia-smi. However, these estimates assume a steady power draw during computation and do not account for fluctuations in hardware utilization or dynamic changes in the energy grid, We consider CodeCarbon a reasonable approximation for measuring energy consumption. This reffected in recent literature, where studies compared CodeCarbon measurements to wattmeters that directly measure power consumption on machines (Bouza et al., 2023)
We acknowledge that energy consumption varies significantly based on regional energy mixes and the infrastructure of data centers. Emissions are typically calculated by multiplying the energy consumed by carbon intensity factor. For given energy mix, this factor remains constant. As result, emissions scale lincarly with energy consumption when the energy mix is fixed. Thus, once we compute Parcto front based on energy consumption, the emission axis can be scaled accordingly to reflect the specific carbon intensity of the region. In dynamic scenarios where energy mixes change over time, our approach enables rapid recalculation of Parcto fronts reflect these variations so that stakcholders can accommodate real-time or regional changes energy conditions. While data centers may often rely on carbon-neutral energy sources, the broader applicability of our method extends to embedded devices, such as those used in automobiles
13

---

Published in Transactions on Machine Learning Research (07/2025)
or IoT systems. These devices operate in highly varied environments, with energy sources dependent on local conditions. Our approach enables stakeholders to determine location-specific or even time-sensitive Pareto fronts, providing a tailored basis for decision-making. This empowers users to balance trade-offs between accuracy and emissions based on financial, ecological, or operational considcrations. While testing on hardware such as low-power chips or alternative GPUs could offer additional insights into hard hardware-specife performance trade-offs, we maximize the potential impact and accessibility of our research by training on NVIDIA A100 GPUs, which is commonly-used hardware in academia and industry.
The DSNN configurations that outperform the default baselines were identified using our MFMO approach. This strategy enabled efficient exploration of broader region of the search space while keeping training costs manageable. Importantly, the models were not trained to full convergence, by design. Although we expect the accuracy gap between configurations to narrow with full training, our optimal configurations consistently demonstrate higher accuracy in the early stages of training, indicating faster convergence. This approximation inferring full-training behavior from carly-stage performance assumption in multi-fidelity optimization. The early stage performance advantage is particularly valuable during optimization, as it enables the rapid identification of promising candidates with significantly reduced computational overhead. Crucially, we also expect the relative emissions savings of these configurations to persist after full training, since the computational graph, operation types, and numerical precision remain fixed throughout training and thus continue to govern energy consumption consistently.
a widely accepted
We do not focus on transformer-based architectures in this work. While transformers have become prominent in computer vision, particularly Vision Transformers (ViTs), they typically require substantial computational resources and are less common in highly constrained environments. In scenarios such edge computing and embedded systems for image classification, smaller convolutional architectures remain a predominant choice due to their lower memory footprint, faster inference, and mature optimization toolchains (Mauricio et al.) 2023). Although there are emerging quantization strategies for transformers (Scherer et al.) 2024), our goal is not to replicate or extend the DSNN framework to support these architoctures. In line with the principles of AutoML, we are conscious of the compute budget required for large-scale search and deliberately avoid high-cost models like transformers to ensure that our pipeline aligns with low-resource deployment scenarios.
Weighing the additional cost of employing AutoML with the savings achieved the impact of our approach. We argue that we offset the additional cost. of the AuloML training process, namely the generation of the Pareto front for each model, through our savings in inference cost in a negligible amount of time. We target an application environment like automated driving or automated production, where inference needs to happen at near real-time or real-time. To provide sufficient evidence, we provide an upper bound calculation based on the experiments we ran. Our longest. AutoMT. optimization run, GoogLeNet on Caltech101, took approximately 48 hours to determine an approximated Pareto front, which is by far the longest; most other runs were completed in about half that time. Assuming a maximum 300W power draw an A100 GPU according to NVIDIA and local carbon intensity factor of 0.475 RgCO2//WI, this results in an estimated emission of about 6.84 kgCO2eq, based on the CodeCarbon formula:
result is vital to assossing
Emissions (kgCO2eg) 0.3 kW * 48 h 0.475 kgCO2/kWh 6.84 kgCO2eq.
(6.1)
For GoogLeNet, our optimized model reduces inference emissions by approximately 0.0004 kgCO2eq per inference, meaning the optimization overhead is amortized after around 17,100 inferences, or about 4.75 hours assuming conservative real-time inference rate of one image per second. This is much slower than actual rates in automated driving or production contexts, which is what we are targeting. For other models in our study, per-inference emission savings are significantly higher, reaching the break-even point even carlier.
In this work, we focus exclusively on image classification tasks. Other vision tasks, such as object detection, semantic segmentation, or visual tracking, are relevant for energy-effccient computing in real-world applications, particularly in our intended domains like automated driving and industrial monitoring. However, extending our methodology to these more complex (asks is beyond the scope of this study.
Future Work could focus on revisiting the multi sintii-ideitty-miuie wwltimmmmtttioiiieeeeee implementation to find a more efficient way for ParEGO and HyperBand to intertwine, such as by finding a more effective way to assign
14

---

Published in Transactions on Machine Learning Research (07/2025)
budgets and weights of objectives. Further investigations could include exploring more DSNN-speccfie fidelity types and multi-objective algorithms to achieve even greater reductions in model emissions. We consider it especially interesting to use the number of weight bits as a fidelity type. By controlling the precision of the weight quantization, training can be sped up in the earlier fidelity while regaining as much information as possible, use this for full training of the most promising configurations al maximum precision. Through these future initiatives, the underlying optimization methodology and the environmental benefits of our optimized DSNNs can be further improved, thereby contributing significantly sustainable Al.
Acknowledgements
The authors were supported by the German Federal Ministry for the Environment, Climate Action, Nature Conservation and Nuclear Safety (BMUKN) (GreenAutoMTAFFAS. project no. 67KT32007A). The authors gratefully acknowledge the computing time provided to them the high-pprformmnce computers Noctua2 at the NHR Center PC2 under the project ipc-prf-intexml. These are funded by the Federal Ministry of Education and Research and the state governments participating on the basis of the resolutions of the GWK for the national high performance computing at universities (ww..nhr www.nhh--errin.do vercin.de /
15

---

Published in Transactions on Machine Learning Research (07 (225))
References
S. Belakaria, A. Deshwal, and J. Doppa. Multi-Fidelity Multi-Objective bayesian optimization: An output space entropy search approach. AAAI, 34(06):10035 10043, 2020.
B. Bischl, M. Binder, M. Lang, T. Pielok, J. Richter, S. Coors, J. Thomas, T. Ullmann, M. Becker, A.-L. Boulesteix, D. Deng, and M. Lindauer. Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, pp. c1484, 2023.
L. Bouza, A. Bugeau, and L. Lannelongue. How estimate carbon footprint when training deep learning models? guide and review. Environmental Research Communications, 5(11):115014 2023.
H. Chen, Y. Wang, C. Xu, B. Shi, C. Xu, Q. Tian, and C. Xu. AdderNet: Do we really need multiplications in deep learning? In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Sealile, WA, USA, June 13-19, 2020, pp. 1465 1474, 2020a.
T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. Hinton. Big self-supervised models are strong semi-supervised learners. arXi::200. 10029y2 LC), 2020b.
M. Courbariaux, Y. Bengio, and David. Binaryconnect: Training deep neural networks with binary weights during propagations. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Proceedings of the 29th International Conference on Advonces in Neural Information Processing Systems (NeurIPS'15) volume 28. Curran Associates, 2015.
K. Deb. Multi-objective Optimization, pp. 403 449. 2014.
K. Eggensperger, Müller, N. Mallik, M. Feurer, R. Sass, A. Klein, N. Awad, M. Lindauer, and F. Hutter. HPOBench: collection of reproducible multi-fidelity benchmark problems for HPO. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Newral Information. Processing Systems Track Datasels and Benchmarks. Curran Associates, 2021.
M. Elhoushi, Z. Chen, F. Shafiq, Y. Henry Tian, and J. Yiwci Li. DeepShift: Towards multiplication-less neural networks. In IEEE Conference on Compuler Vision and Pallern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021, pp. 2359 2368, 2021.
T. Elsken, Metzen, and F. Hutter. Neural Architecture Search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
L. Fei- R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In IEEE Conference Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2004, Washington, DC, USA, June 27 - July 2, 2004, pp. 178, 2004.
P. Frazier, W. Powell, and S. Dayaník. The knowledge-gradient policy for correlated normal belicfs. Journal on Compuling, 21(4):599 613, 2009.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp- 770 778, 2016.
P. Hennig and C. Schuler. Entropy search for information-cffcccent global optimization. Journal of Machine Learning Research, 98888(1):1800) 1837, 2012.
J. Srnnandez-Lobaato M. Hoffman, and Z. Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger (eds.), Proceedings of the 28th International Conference on Advances in Neural Information Processing Systems (NeurIPS'A) 14). Curran Associates, 2014.
16

---

Published in Transactions on Machine Learning Research (07/2025)
A. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andrectto, and H. Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXi:: 2017.
F. Hutter, H. Hoos, and K. Leyton-Brown. Sequential mosel-based optimization for general algorithm configuration. In C. Cocllo (cd.), Proceedings of the Fifth International Conference on Learning and Intelligent Optimization (LION'11), volume 6683 of Lecture Notes in Computer Science, pp. 507-523. Springer, 2011.
F. Hutter, H. Hoos, and K. Leyton-Brown, An efficient approach for assessing hyperparameter importance. E. Xing and T. Jebara (eds.), Proceedings of the 31th International Conference on Machine Learning, (ICML'14) 11), pp. 754-762. Omnipress, 2014.
F. Hutter, L. Kottholf, and J. Vanschoren (eds.). Automated Machine Learning Methods, Systems, Challenges. 2019.
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In F. Bach and D. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning (ICML (1)) volume 37. Omnipress, 2015.
K. Jamicson and A. Talwalkar. Non-stochastic best arm identification and Hyperparameter Optimization. In
A. Gretton and C. Robert (eds.), Proceedings of the Seventeenth International Conference Artificial Intelligence and Statistics (AISTATS'66) volume 51. Proceedings of Machine Learning Research, 2016.
D. Jones, M. Schonlau, and W. Welch. Efficient global optimization of expensive black-box functions. J. Glob. Optim., 13(4):455-492 1998.
K. Kandasamy, G. Dasarathy, J. Oliva, J. Schneider, and B. Póczos. Multi-fidelity gaussian process bandit optimisation. J. Artif. Intell. Res., 66:151 196, 2019.
J. Knowles. ParEGO: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Trans. Evol. Compul.. 10(1):50 66, 2006.
A. Krizhevsky, G. Hinton, and Learning multiple layers of features from tiny images. University of Toronlo, 2009.
A. Lacoste, A. Luccioni, V. Schmidt, and T. Dandres. Quantifying the carbon emissions of machine learning. XxX::1910.099000 2019.
H. Li, K. Ota, and M. Dong. Learning IoT in edge: Deep learning for the internet of things with edge computing. IEEE Netw., 32(1):96 10161111111100010111 2018.
L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit- approach to hyperparameter optimization. J. Mach. Learn. Res., 18:185:1-185:55 2017.
M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp, D. Deng, C. Benjamins, T. Ruhkopf, R. Sass, and
F. Hutter. SMAC3: A versatile bayesian optimization package for Hyperparameter Optimization. Journal of Machine Learning Research, 23(54):1-9, 2022.
K. Lottick, S. Susai, S. Friedler, and J. Wilson. Energy usage reports: Environmental awareness as part of algorithmic accountability. arXiv:1911.0895, 2019.
J. Maurício, Domingues, and J. Bernardino. Comparing vision transformers and convolutional neural networks for image classification: A literature review. Applied Sciences, 13(9):5521, 2023.
A. Mernadez 1. Van Nieuwenhuyse, and S. Gonzalez. A survey on multi-objective hyperparameter optimization algorithms for machine learning. Artificial Intelligence Review, 56:8043 8093, 2022.
B. Alessandra Motetti, M. Risso, A. Burrello, E. Macii, M. Poncino, and D. Jahier Pagliari. Joint pruming and channel-wise mixed-preeision quant for efficient deep neural networks. 07700:2407.01054 2024.
17

---

Published in Transactions on Machine Learning Research (07/2025)
P. Probst, A. Boulesteix, and B. Bischl. Tunability: Importance of hyperparameters of machine learning algorithms. Journal of Machine Learning Research, 20(53):1-32, 2019.
C. Edward Rasmussen and C. Williams. Gaussian Processes for Machine Learning. 2006.
M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. Xnor-net: ImageNet classification using binary convolutional neural networks. 160: 05279, 2016.
B. Rokh, A. Azarpeyvand, and A. Khanteymoori. A comprehensive survey model quantization for deep neural networks in image classification. ACM Trans. Intell. Syst. Technol., 14(6):97:1-97:500 2023.
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and T. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the International Conference on Computer Vision and Pattern Recognition (VVPR'18) Computer Vision Foundation and IEEE Computer Society, IEEE, 2018.
R. Sass, E. Bergman, A. Biodenkapp, F. Hutter, and M. Lindauer. Deepcave: An interactive analysis tool for aulomated machine learning. In M. Mutny, I. Bogunovic, W. Neiswanger, S. Ermon, Y. Yue, and A. Krause (eds.), ICML Adaptive Experimentol Design and Active Leorning in the Real World (ReALML Workshop 2022), 2022.
M. Scherer, C. Cioflan, M. Magno, and L. Benini. Work in progress: Linear transformers for TinyML. In Design, Aulomation Test in Europe Conference & Exhibition, DATE 2024, Valencia, Spain, March 25-27, 2024, pp. 1-2, 2024.
R. Schmucker, M. Donini, V. Perrone, M. Zafar, and C. Archambeau. Multi-objective multi-fidelity hyper- parameter optimization with application to fairness. Tn R. Calandra, J. Clune, E. Grant, J. Schwarz, J. Vanschoren, F. Visin, and J. Wang (eds.), NewrIPS 2020 Workshop on Meta- Learnnn 2020.
R. Schwartz, J. Dodge, N. Smith, and O. Etzioni. Green AI. Comman. ACM, 63(12):54466) 2020.
B. Shahriari, K. Swersky, 7. Wang, R. Adams, and N. de Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148 175, 2016.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.155: fcs. CVJ, 2014.
V. Sze, Y. Chen, T. Yang, and J. Emer. Efficient processing of deep neural networks: A tutorial and survey. Proc. IEEE, (2))))) 2329, 2017.
C. Szegody, W. Liu, Y. Jia, P. Sermanct, S. Reod, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In IEEE Conference Compuler Vision and Pallern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 1-9, 2015.
M. Tan and Q Le. Efficient Smaller models and faster training. In M. Meila and T. Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139, pp. 10096 10106, 2021.
Y. Tian, D. Krishman, and P. Isola. Contrastive multiview coding. In A. Vedaldi, H. Bischof, T. Brox, and J. Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI, volume 12356, pp. 776-794, 2020.
T. Tornede, A. Tornede, J. Hanselle, F. Mohr, M. Wever, and E. Hüllermeier. Towards green automated machine learning: Status quo and future directions. Journal of Artificial Intelligence Research, 77:427-457, 2023.
T. Wada and Н. Hino. Bayesian optimization for multi-objective optimization and multi-point search. NrXiv.1955.0295 2019.
H. Wang, S. Ma, L. Dong, S. Huang, H. Wang, L. Ma, F. Yang, R. Wang, Y. Wu, and F. Wei. BitNet: Scaling 1-bit transformers for large language models. arXiv:2310.114533 2023.
18

---

Published in Transactions on Machine Learning Research (07/2025)
H. You, X. Chen, Y. Zhang, C. Li, S. Li, Z. Liu, Z. Wang, and Y. Lin. Shiftaddnet: A hardware-inspired deep network. Aduances in Neural Information Processing Systems, 33:2771- -2783, 2020.
Z. Zhou, X. Chen, E. Li, L. Zeng, K. Luo, and J. Zhang. Edge intelligence: Paving the last mile of artificial intelligence with edge computing. Proc. IEEE, 107(8):1738 1762, 2019.
19

---

Published in Transactions on Machine Learning Research (07/ (025)
A Appendix
A.1 Optimization Results
We provide tables that list the Pareto-optimal configurations obtained for RosNet20, MobileNet, GoogLeNet and Efficient V2 on CIFAR-10 and Caltech101 (Tables mean and standard deviation across multiple seeds. These results include both the Pareto-optimal solutions and the performance of the default configurations in terms of loss and emissions, offering comprehensive overview of the trade-offs achieved.
f to 10 For each model and dataset, we report the
A.2 Hyperparameter Importances
We further analyzed the hyperparameter imporlance for the additional architectures we examined, as shown in Figures (6a| [0b] Across all models, the DSNN-specific hyperparameters consistently remain among the most important ones, alongside weight decay and learning rate. This highlights the relevance of our approach: focusing on the DSNN-specific parameters and allocating computational resources toward tuning them optimally for the given task appears to be a promising and effective strategy.
A.3 Additional Baselines for Comparison
To demonstrate the effectiveness and relevance of our approach, compare against the same baselines the original DSNN paper by Elhoushi et al. 2002)). Since our work builds on multi-objective optimization framework targeting both accuracy and emissions, but the original DSNN paper provides only single-objective baselines (focused solely on accuracy), we restricted our evaluation to the accuracy objective for fair comparison. Specifically, we focus on the VGG19 architecture (Simonyan Zisserman 2014 as this is the network for which Elhoushi al. (222)) published baseline results.
From the Pareto front obtained via MOMF optimization of the VGG19 architecture, selected Pareto- optimal configuration that outperformod the default in accuracy while maintaining similar emissions. As shown in Table 2 the final accuracy of this configuration outperforms not only the original unquantized VGG19 but also the quantized network baselines such as the DeepShift-PS baseline from Elhoushi et al. (2021). AdderNet (Chen et al. 2020b). and AddNett You et al. 2020).
Model
Accuracy
AutoDSNN
93.45%
Original*
92%
VGG19 on CIFAR10
DeepShift-PS*
91.57%
AdderNet* (Chen et al. 2020b
93.02%
Ahift * You et al. 2020
90%
Table 2: Accuracy comparison of VGG19 model variants on CIFAR10. Baseline results from Elhoushi el. al. 2021 are marked with an asterisk.
Looking at the configuration details Table 3 we can see that the Pareto optimal configuration that beats the aforementioned baselines has 11 shift layers, meaning that little more than half of the VGG19 layers are quantized. This is in line with our analysis of optimal DSNN configurations in Section 5.2.3 Again, it is not optimal to simply quantize the whole architecture. More so, since with our MOMF approach can beat both the original unquantized architecture, the default DSNN, as well as both quantization baselines. These results highlight the relevance of our approach to discovering well-performing DeepShift configurations tailored to both accuracy and emissions. In particular, our optimal configuration, which outperforms all considered basclines, only partially quantizod, demonstrating that effective configurations are not straightforward or easily designed by hand. This underscores the importance of our method for systematically identifying efficient DSNNs beyond standard quantization heuristics, which are competitive with relevant baselines.
20

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Table 3: Configuration for VGG19 on CIFARIO, selected from the Pareto front. This configuration achieves
significant improvement the default unquantized VGG19 in terms of accuracy.
Hyperparameter
Config 32
Optimizer
Adam
Learning Rate
0.0107
Momentum
0.1474
Weight Decay
0.0032
Weight Bits
Act. Int. Bits
20
Act. Frac. Bits
11
Shift Depth
Shift Type
Q
Rounding
Stochastic
Table 4: Pareto optimal configurations and default DSNN initiation of ResNet 20 on CIFAR10. Pareto optimal solutions on the aggregated Pareto front of the ResNet20 architecture on CIFAR10 on three seeds, including the mean aggregated loss and emissions the default configuration.
Hyperpar.
Config 124
Config 116
Config
Config
Config
Config 28
Config 49
Default
Optimizer
Ranger
Ranger
Ranger
Ranger
Ranger
Adagrad
Adagrad
SGD
Learning
0.0130
0.0327
0.0150
0.0542
0.0129
0.0548
0.0929
0.1
Rate
Momentum
0.7489
0.7718
0.1529
0.4983
0.6838
0.6783
0.4825
0.9
Weight
Decay
0.0001
0.00005
0.0038
0.0001
0.0001
0.0002
0.0003
0.0001
Weight Bits
Act. Integer
22
24
16
Bits
Act. Frac-
32
30
32
11
27
11
8
16
tion Bits
Shift Depth
20
Shift Type
PS
PS
PS
PS
PS
Rounding
Det.
Det.
Det.
Stochastic
Det.
Stochastic
Stochastic
Det.
Loss
0.1127
0.1142
0.1161
0.1172
0.1176
0.1352
0.1443
0.3518
Emissions
0.000798
0.000792
0.000759
0.000708
0.000699
0.000682
0.000674
0.000751
21

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Table 5: Pareto optimal configurations and default DSNN instantiation of MobileNet V2 on CIFAR1O
Hyperparameter
Config 15
Config 68
Config 66
Config 21
Default
Optimizer
Adadelta
Adadelta
Adadclta
SGD
SGD
Learning Rate
0,182188
0.186665
0.197817
0.183219
0.1
Momentum
0.726835
0.689596
0.1837
0.76337
0.9
Weight Decay
0.002727
0.003048
0.00306
0.002277
0.0001
Weight Bits
5
Activation Integer
21
19
21
16
Bits
Activation Fraction
31
31
32
16
16
Bits
Shift Depth
14
1
53
Shift. Type
PS
PS
PS
PS
Rounding
Deterministic
Deterministic
Deterministic
Deterministic
Deterministic
Loss
0.1683
0.1756
0.1797
0.9016
0.3017
Emissions
0.000755
0.000655
0.000552
0.0005449
0.001656
Table 6: Pareto optimal configurations and default DSNN instantiation of GoogLeNet on CIFAR10
Hyperpar.
Config 65
Config 33
Config 28
Config 14
Config 32
Default
Optimizer
Adadelta
Adadelta
Ranger
Ranger
Adadelta
SGD
Learning Rate
0.028997
0.023838
0.020002
0.058610
0.115941
0.1
Momentum
0.209328
0.4944258
0.250184
0.673880
0.372339
0.9
Weight Docay
0.0084887
0.006737
0.006691
0.002313
0.009444
0.0001
Weight Bits
2
2
Activation Into-
21
24
29
26
29
16
ger Bits
Activation Frac-
4
4
8
7
16
tion Bits
Shift Depth
1
22
Shift Type
Q
PS
PS
PS
Rounding
Stochastic
Deterministic
Deterministic
Deterministic
Deterministic
Deterministic
Loss
0.1347
0.1409
0.1636
0.1748
0.1850
0.1810
Emissions
0.000916
0. 0.000912
0.000876
0.000840
0.000835
0.001388
22

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Table Pareto optimal configurations and default DSNN instantiation of ResNet20 on Caltech101
Hyperparameter
Config 26
Config 66
Config 32
Config 76
Default
Optimizer
Ranger
Ranger
Ranger
RMSProp
SGD
Learning Rate
0.023
0.039
0.076
0,015
0.1
Momentum
0.559
0.333
0.344
0.551
0.9
Weight Decay
0.0033
0.0027
0.0029
0.0069
0.0001
Weight Bits
5
Activation Integer
24
22
16
Bits
Activation Fraction
17
27
13
21
16
Bits
Shift Depth
2
2
1
20
Shift. Type
PS
PS
PS
PS
Rounding
Deterministic
Stochastic
Stochastic
Deterministic
Deterministic
Loss
0.456
0.532
0.636
0.874
0.679
Emissions
0.00046
0.00044
0.00044
0.00044
0.00109
Table 8: Pareto optimal configurations and default DSNN instantiation of MobileNet Caltech101
Hyperparameter
Config 9
Config 65
Config 33
Config 74
Default
Optimizer
Adadelta
Adaddelta
SGD
Adam
SGD
Learning Rate
0.192
0.199
0.004
0.006
0.1
Momentum
0.510
0.545
0.016
0.012
0.9
Weight Docay
0.009
0.004
0.004
0.004
0.0001
Weight Bits
5
5
Activation Integer
12
11
32
16
Bits
Activation Fraction
26
30
7
16
Bits
Shift Depth
53
Shift Type
PS
PS
Q
PS
Rounding
Deterministic
Deterministic
Deterministic
Stochastic
Deterministic
Loss
0.274
0.276
0.459
0.870
0.337
Emissions
0.00053
0.00049
0.00046
0.00046
0.00066
23

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Table 9: Pareto optimal configurations and default DSNN instantiation of GoogLeNet on Caltech101
Hyperpar.
Config
Config
Config
Config
Config
Config 21
Config 20
Default
Optimizer
Adadelta
Adadelta
Adadelta
Adagrad
Radam
Adagrad
Adam
SGD
Learning
0.058
0.059
0.052
0.13
0.199
0.187
0.017
0.1
Rate
Momentum
0.185
0.642
0.194
0.889
0.647
0.367
0.248
0.9
Weight
Decay
0.0004
0.0019
0.0029
0.0006
0.0027
0.0048
0.0059
0.0001
Weight Bits
3
5
Activation In-
30
16
teger Bits
Activation
19
20
30
16
Fraction Bits
Shift Depth
2
2
2
22
Shift Type
Q
Rounding
Det.
Det.
Det.
Det.
Det.
Stochastic
Stochastic
Det.
Loss
0.356
0.362
0.376
0.563
0.778
0.793
0.922
0.466
Emissions
0.0026
0.0026
0.0023
0.0022
0.0021
0.0021
0.0020
0.0027
Table 10: Pareto optimal configurations and default DSNN instantiation of Efficient CIFAR10
Hyperpar.
Config 61
Config
Config 31
Config 71
Config 39
Default
Optimizer
RMSprop
RMSprop
Adadelta
Adam
Ranger
SGD
Learning Rate
0.01737
0.03561
0.03508
0.17490
0.12583
Momentum
0.8861
0.6085
0.4682
0.3632
0.7668
0.9
Weight Docay
0.0000381
0.007015
0.003031
0.009879
0.007242
0.0001
Weight Bits
4
2
Activation Inte-
8
15
22
16
9
16
Bits
Activation Frac-
10
tion Bits
Shift Depth
28
42
342
Shift Type
Q
PS
PS
PS
PS
Rounding
Stochastic
Deterministic
Stochastic
Deterministic
Stochastic
Deterministic
Loss
0.1521
0.1539
0.1555
0.1653
0.1657
0.8283
Emissions
0.000973
0.000934
0.000926
0.000880
0.000870
0.002074
24

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Importance
0.06
0.04
0.02
weight, decay
shift, depth
momentum
15
activation fraction bits
activation Integer. bits
rounding
weight bits
shift type
(a) Hyperparameter importance with respect to loss.
Importance
0.4
0.3
0.2
shift depth
activation_ fraction bits
weight decay
activation integer. bits
momentum
rounding
weight bits
shift. type
(b) Hyperparameter importance with respect to emissions.
Figure 6: Hyperparameter importance according to IANOVA for MobileNet Caltech101. (a) Importance with respect to loss. (b) Importance with respect to emissions.
25

---

Published in Transactions on Machine Learning Research (07/202) 2025)
0.4
Importance
0.3
0.2
activation fraction_. bits
shift. depth
momentum
weight bits
weight. decay
shift. type
activation Integer bits
rounding
(a) Hyperparameter importance with respect to loss.
Importance
0.4
shift depth
activation_ fraction bits
activation Integer. bits
weight decay
momentum
weight bits
shift. type
rounding
(b) Hyperparameter importance with respect to emissions.
Figure 7: Hyperparameter importance according to FANOVA for GoogLeNet Caltech101. (a) Importance with respect to loss. (b) Importance with respect to emissions.
26

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Importance
0.03
0.02
0.01
weight, decay
activation. Integer bits
activation. Fraction. bits
shift depth
momentum
raunding
weight bits
shift type
(a) Hyperparameter importance with respect to loss.
Importance
activation_ fraction bits
weight, decay
shift depth
weight. bits
activation. integer.... bits
shift. type
momentum
rounding
batch size
(b) Hyperparameter importance with respect to emissions.
Figure 8: Hyperparameter importance according to IANOVA for ResNet20 Caltech101. (a) Importance with respect to loss. (b) Importance with respect to emissions.
27

---

Published in Transactions on Machine Learning Research (07/202) 2025)
0.2
0.15
0.1
0.05
0
weight. decay
shift depth
activation fraction bits
momentum
rounding
test batch. site
activation Integer .bits
shift type
weight bits
Hyperparameter importance with respect to loss.
0.8
Importance
0.2
shift. depth
activation. fraction. bits
momentum
activation, integer bits
rounding
weight. decay
Ir
test batch size
weight. bits
shift. type
(b) Hyperparameter importance with respect to emissions.
Figure 9: Hyperparameter importance according to fANOVA for MobileNet on CIFAR10. (a) Importance with respect to loss. (b) Importance with respect to emissions.
28

---

Published in Transactions on Machine Learning Research (07/202) 2025)
0.15
Importance
0.05
activation integer _bits
shift depth
weight decay
momentum
activation Fraction bits
weight. bits
shift. type
rounding
(a) Hyperparameter importance with respect to loss.
0.6
Importance
0.4
shift depth
activation fraction. bits
activation integer bits
weight, decay
momentum
weight bits
shift. type
rounding
(b) Hyperparameter importance with respect to emissions.
Figure 10: Hyperparameter importance according to IANOVA for GoogLeNet CIFARIO. (a) Importance with respect to loss. (b) Importance with respect to emissions.
activation
fraction
0.14
activation_intege Baatloo__nttte______
0.12
epochs
Importance
0.1
momentum
0.08
rounding
0.06
shift_depth
shift__ypp
0.04
weight. bits
0.02
weight_ decay
8
A.A
Weight for loss
Figure 11: MO analysis of hyperparameter importance of a ResNet20 on CIFAR10 w.r.t. loss and emissions using the IANOVA method. The x-axis shows wi, the weight. of the objective loss, ranging from to 1. The weight of the objective emissions thus w, wy. The y-axis shows the importance of each hyperparameter in the legend.
29

---

Published in Transactions on Machine Learning Research (07/2025)
Algorithm 1 Multi-Pidelity Optimization with Parego for DSNNs
Require:
Configuration space C, objectives [Clees Cemissions]; budget range [Bmin Bmax], number of trials N, DSNN architecture A.
Ensure: Pareto optimal configurations PN-
Initialize scenario S with C, L; (Bmin: Buax), and N.
Generate initial observation dataset Dinit by sampling k random configurations {1:) CC.
3: Define intensifier H. as Hyperband for budget allocation.
4: Initialize optimizer O using Parego and H.
5: for each trial E [I....,N} do Select a configuration Åe EC using Parego. Allocate budget b, E (Bmius Bmax) using H. Perform evaluation of λ with budget b;:
1.
Convert DSNN A to shift-based architecture A':
A' convert to (A, A (shift depthJ, [shiftt typel) w' round(w, Ae(rounding), Vw A',
where convert_to__st shiff replaces standard operations with shift operations, and round applics deterministic or stochastic rounding.
2. Train A' for b, epochs and compute the objective values:
Lloss (Ne) DDestt (2.g)
1
Σ ((fa(()))))
Lomissions (Nx) measure emissions(AA (A', (A',bb),) bb),
where Drest the test dataset, the cross-entropy loss, and measure emissions computes the energy consumption.
3. Update observation dataset:
D. + D U U {()e, (omisssoo))
Update the Pareto front:
P. ((), E D, AX D, L(X) (())),
where C(X)> > ((X) denotes that X' dominates X.
10: end for
11: return Pn
30


Reviews:
[{"review_id": "4pClTozksZ", "reviewer": "Reviewer_dPd2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- Overall, Deep shift neural networks are not as explored as other approaches, and this work's aim is to advance our knowledge in these architectures.\n- DSNNs are promising to reduce computation, and the overall motivation behind this work is valid.\n- The work is pointing out specific efficient configurations in the tested designs (this is also a weakness).\n- The work does a great job of reviewing the sota and providing the background.\n\n## Weaknesses\n- The overall technical contribution is weak. This work indicates specific configurations found in specific setups (a strength), but this is very limiting when claiming generality, given that the finding is limited to the specific setups presented in the work. The only possible technical novelty lies in (4.1), but I am wondering how that is not straightforward.\n- The experiments are very limited, the architectures are outdated, and the datasets are small-scale. I doubt it is possible to claim any generality from the observations taken from these architectures/datasets. Especially in a context of energy consumption analysis, more recent architectures (like MobileNetv3) or more effective (like the EfficientNet family, or NAS-generated ones) should be generated. Also, Transformers are becoming more and more present - since the approach is not limited to convolutional architectures, these should be benchmarked too.\n- The only analyzed task is image classification - is there any reason for that ? Showcasing applicability on efficient architectures deployed on different tasks, like YoLOv11 on object detection, would showcase the broad applicability.\n- There is a possible background pitfall in the whole work: this work is not really minimizing the CO2 emissions, but rather the *energy consumption*. Indeed, as the same authors write in Sec. 5.1, \"we use the CodeCarbon emissions tracker to track carbon emissions from\ncomputational processes, in our case inference, by monitoring energy use and regional energy mix in kgCo2eq\"; hence, the estimated energy consumption is multiplied by a factor to get the CO2 emitted in that specific region. If the algorithm minimizes CO2, it would also take into account *how* the energy is produced, *how far* the machine is from the production site, etc., which is not really what is done in this work. *This is easily fixable, just claiming that the algorithm aims to reduce power consumption, which also translates into CO2 emissions reduction*. \n\n\n### Minor Weaknesses\n- *Readability*. The authors should carefully check that all the definitions and abbreviations are correctly provided - for example, in the abstract, HPO is not defined (yes it is hyperparameter optimization, but it should be defined)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors take Deep Shift Neural Networks (DSNN), a class of deep models, and empirically analyze their competitiveness in terms of a trade-off between performance (accuracy) and energy consumption. Evidently, this requires hyper-parameters research, which is conducted through known multi-fidelity hyper-parameter optimization tools, adapted to handle two objectives. The authors benchmark ResNet20, MobileNetv2 and GoogLeNet on CIFAR-10 and Caltech101, finding in certain cases unexpected trade-offs.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Referring to the weaknesses section, please highlight better the technical contribution in case it is not simply limited to Sec. 4.2, and upgrade the experimental section following the indicated guidelines in the weaknesses.\n\nAre you going to release the code open-source in case of acceptance of the work?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W75Nw8PUst", "reviewer": "Reviewer_xxxD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths:__\nThe intention and potential impact of this work is clearly important in an age where energy usage of AI models in training and inference are major issues for sustainability. The approach also covers a reasonable amount of search space in attempting to find better hyperparameters for these models.\n\n__Weaknesses:__\nThe paper is rather hard to pin down in terms of how one might understand the outcomes. First, although it is often claimed that high accuracy is achieved, in fact the loss is the only metric described so the accuracy of the models tested are unclear. Furthermore, it is unclear how much the cost of the hyperparameter optimization was in order to achieve the final (performant) configurations or to even find the Pareto front.\n\nBeyond this, the hyperparameter importances seem to suggest that loss and emissions have rather unrelated hyperparameter importances. What to do with this is also unclear. Not to mention that it is unclear whether the same hyperparameters are important across models, therefore again: what is the take-away?\n\nFinally, and perhaps most importantly, it seems that there are no alternative baseline approaches (ourside of DSNNs) that the results are compared against. This means that the take-away, in terms of how well DSNNs can do in place of alternative quantization methods, distillation methods, or otherwise, remains unclear. For me (someone outside of the specific niche), this gives very little understanding of how to place the contribution of this work in general and how to transfer it's lessons outside of this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents an investigation into the use of AutoML approaches for hyperparameter optimisation applied to deep shift neural networks (DSNNs) in the pursuit of finding highly energy efficient networks and network parameter regimes. A novel combination of multi-fidelity hyper parameter optimization combined with a multi-objective optimization process attempts to find ideal Pareto tradeoffs for DSNN design.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "For a longer discussion on these, please see the weaknesses section above. To make things more concise:\n\n- To understand the results better, I would request that not only loss but also accuracies are provided\n- A more full explanation of how the hyperparameter importance changes across different models would be useful. Also to answer if there is a consistency across models.\n- The cost of the search for the Pareto front should also be considered (or at least discussed) so that one may take this into consideration when finding models.\n- Aside from the default DSNN models, no alternative approaches to DSNNs are compared against as baselines. Appropriate baselines should be used so that the significance of these contributions can be placed in context.\n- The datasets used are extremely small and limited, an analysis of how well these approaches work as we scale up closer to real world datasets would answer whether this methods scales or not. ImageNet scale would be appropriate for understanding scale for a sufficiently difficult task.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cTAQetDEvp", "reviewer": "Reviewer_8oNe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well-motivated as there is a further need for Green AutoML and applying AutoML techniques to DSNN is useful.\n* The related work is appropriately well-written and easy to follow. For the most part, the background and metholodogy are as well.\n\nWeaknesses: \n* The proposed methodology is not terribly novel as it is mostly an application of existing off-the-shelf techniques.\n* The paper does not really compare against baseline approaches in the literature.\n* Aspects of this work such as the Configuration Space arguably lie more in the realm of Neural Architecture Search (NAS) [1] than HPO, yet no discussion of this subfield is conducted. \n* The paper primarily conducts experiments on an Nvidia A100 GPU using very old models that severely under-utilize the GPU. For instance, the authors routinely mention their usage of ResNet-20 which is a very small and not even considered in the ResNet Strikes Back [2] paper.\n* The reviewer would question some of the provided insights, e.g., Figure 8 hyperparameter importance w.r.t. loss - why is epochs listed there at all? Carbon cost does scale directly with epochs for sure but it is irrelevant to inference once a model is trained.\n* The presentation is severely lacking in parts. Will elaborate in requested changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to apply Automated Machine Learning (AutoML) techniques such as Hyperparameter Optimization (HPO) to Deep Shifting Neural Networks (DSNN). DSNN operate using bit-shift operations as opposed to the more hardware-intensive multiply-add accumulate (MAC) operations. Specifically, the authors utilize Bayesian Optimization (BO) HPO techniques to this end, to optimize networks like ResNet-20 and MobileNetV2 on an Nvidia A100 GPU. Additionally, the authors provide insights into the most important DSNN design choices.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Experimental - the authors need to do at least one of two things:\n1) Use larger and more powerful models that fully take advantage of the A100 GPU and where optimizations would be more meaningful. For instance, aim for something that would not fit on the A100's predecessor, the 32GB V100. Using an A100 for ResNet-20 and MobileNetV2 is very much overkill and subtracts from the substance of the paper's findings. \n2) Experiment with smaller GPUs, potentially NPUs, ASICS, etc. for mobile or battery powered devices where energy savings are a bigger concern and where larger models cannot run. \n\nAdditionally, there should be some comparison with existing methods in the literature in these scenarios. The premise of AutoML is that an automated procedure can find better neural network designs and training schema than manually-designed methods. \n\nPresentation:\n* Itemize section 4.1 or de-itemize introduction contributions\n* Table 1 is a mess. Notice how \"Hyperparameters\" and \"Search Space\" overlap with each other? \n* Provide more clarification on the items in Table 1. For instance, are bit options a range? If not, why are the choices for weight bits 2 and 8, yet the default is 5?\n* Revise writing for Section 5.1 talking about the A100, as even though it is a very powerful server GPU it is older now.\n* Section 5.2.1 => Figures should be introduced in text in the order they appear in the paper. \n* Page 8: Generally avoid having whole pages devoted to floats. Additionally, these floats contain a lot of empty white space and can be optimized by shrinking their height, increasing marker size, and making the color of \"Uncertainty Bounds\" stand out more. The reviewer comments that the font size of these figures is adequate. \n* Figure 10 same issue with white space. Potentially consider a logarithmic plot.", "broader_impact_concerns": "N/A\n\nReferences:\n\n[1] Xiao, Muhan, et al. \"RetNAS: Reinforce Evolutionary Trend in Lightweight Neural Architecture Search.\" 2024 IEEE Smart World Congress (SWC). IEEE, 2024.\n\n[2] Wightman, Ross, Hugo Touvron, and Hervé Jégou. \"Resnet strikes back: An improved training procedure in timm.\" arXiv preprint arXiv:2110.00476 (2021).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)