∇QDARTS: Quantization as an Elastic Dimension to Differentiable NAS
Authors: Payman Behnam
Uday Kamal
Sanjana Vijay Ganesh
Zhaoyi Li
Michael Andrew Jurado
Alind Khare
Igor Fedorov
Gaowen Liu
Alexey Tumanov
Venue: Accepted by TMLR ()

Abstract: Differentiable Neural Architecture Search methods efficiently find high-accuracy architectures using gradient-based optimization in a continuous domain, saving computational resources. Mixed-precision search helps optimize precision within a fixed architecture. However, applying it to a NAS-generated network does not assure optimal performance as the optimized quantized architecture may not emerge from a standalone NAS method. In light of these considerations, this paper introduces ∇QDARTS, a novel approach that combines differentiable NAS with mixed-precision search for both weight and activation. ∇QDARTS aims to identify the optimal mixed-precision neural architecture capable of achieving remarkable accuracy while operating with minimal computational requirements in a single-shot, end-to-end differentiable framework, obviating the need for pretraining and proxy methods. Compared to fp32, ∇QDARTS shows impressive performance on CIFAR10 with (2,4) bit precision, reducing bit operations by 160× with a slight 1.57% accuracy drop. Increasing the capacity enables ∇QDARTS to match fp32 accuracy while reducing bit operations by 18×. For the ImageNet dataset, with just (2,4) bit precision, ∇QDARTS outperforms state-of-the-art methods such as APQ, SPOS, OQA, and MNAS by 2.3%, 2.9%, 0.3%, and 2.7% in terms of accuracy. By incorporating (2,4,8) bit precision, ∇QDARTS further minimizes the accuracy drop to 1% compared to fp32, alongside a substantial reduction of 17× in required bit operations and 2.6× in memory footprint. In terms of bit-operation (memory footprint) ∇QDARTS excels over APQ, SPOS, OQA, and MNAS with similar accuracy by 2.3× (12×), 2.4× (3×), 13%
(6.2×), 3.4× (37%), for bit-operation (memory footprint), respectively. ∇QDARTS enhances the overall search and training efficiency, achieving a 3.1× and 1.54× improvement over APQ and OQA, respectively.

Published in Transactions on Machine Learning Research (04/2025)
VQDARTS: Quantization as an Elastic Dimension to Differ- entiable NAS
Payman Behnam*i, Uday Kamal*, Sanjana Vijay Ganesh, Zhaoyi Li, Michael Andrew Jurado, Alind Khare
Georgia Institute of Technology
Igor Fedorov Mela
Gaowen Liu
Cisco Research
Alexey Tumanov
Georgia Instilute of Technology
Reviewed on OpenReview: https: openreview. net/ forum? id= ubrOSwyTS8
Abstract
Differentiable Neural Architecture Search methods efficiently find aighacccrrcyy architectures using gradient-based optimization a continuous domain, saving computational resources. Mixed-precision search helps optimize precision within fixed architecture. However, applying it to NAS-generated network does not assure optimal performance the optimized quantized architecture may not emerge from standalone NAS method. In light of these considerations, this paper introduces VQDARTS, a novel approach that combines differentiable NAS with mixed-precision search for both weight and activation. VQDARTS aims to identify the optimal muxxed-precisse neural architecture capable achieving remarkable accuracy while operating with minimal computational requirements in a single-shot, end-to-end differentiable framework, obviating the need for pretraining and proxy methods. Compared to fp32, VQDARTS shows impressive performance on CIFAR10 with (2,4) bit precision, reducing bit operations by x60x with slight 1.57% accuracy drop. Increasing the capacity enables VQDARTS to match fp32 accuracy while reducing bit operations by 18x. For the ImageNet dataset, with just (2,4) bit precision, VQDARTS outperforms state-of-the-arl methods such as APQ, SPOS, OQA, and MNAS by 2.3%, 2.9%, 0.3%, and 2.7% in terms of accuracy. By incorporating (2,4,8) bit precision, VODARTS further minimizes the accuracy drop to 1% compared to fp32, alongside a substantial reduction of 17x in required bit operations and 2.6x in memory footprint. In terms of bit-operation (memory footprint), VQDARTS excels over APQ, SPOS, OQA, and MNAS with similar accuracy by 2.3x (12x), 2.4x (3x), 13% (6.2x), 3.4x (37%), for bil-operation (ineorr footprint), respectively. VQDARTS euhances the overall search and training efficiency, achieving a 3.1x and 1.54x improvement over APQ and OQA, respectively.
1 Introduction
Neural Architecture Search (NAS) has emerged as a powerful method that reduces the manual and human effort find optimum network architectures for the target applications. However, traditional NAS methods require complete training of every candidate network, which is computationally prohibitive due to the enormous search space (Kyriakides & Margaritis 2020 To alleviate this problem, several bodies of work
"Equal contribution.
i "oorreppondigg authorrppymaabbbbelmmmoooaoaa www.bennm@@attc.h.eeii....

---

Published in Transactions on Machine Learning Research (04/2025)
suggest different techniques to reduce the NAS search space (Negrinho & Gordon (017)) Liu et al. (2018a)
201 7a); Kandasamy et al. (2018)). Evolutionary search, Monte-carlo tree search (MCTS) Negrinho & Gordon (2017) ) sequential model-based optimization (SMBO) (Liu et al. (2018a) 2017a)), and Bayesian optimization Kandasamy et al. (2018)) are examples of these approaches. However, even these approaches suffer from
inherent search inefficiency, because they treat architecture search as a black-box optimization problem over a
discrete domain, which leads to a large number of architecture evaluations. These methods are also criticized
for their high time complexity, energy consumption, and not being scalable (Elsken et al. (001
Liu et al.
(2018)))
To address these challenges, weight-sharing methods have been proposed (Ullrich et al. (2017); ; Cal et al. (2019); Sahni et al. (002)); Liu et al. (2018b); Xu et al. (2019)). These methods provide the ad advantage of sharing weights across varied architectures, resulting in a considerable reduction in the compute and memory
resources needed to conduct the search. As a result, they have recently attracted significant attention
Ullrich et al.] (2017) : Cai et al. (2019); Sahni al. (2021) Liu ct al. (0188);; ; Xu et al. (2019); Sahni et al. (2021): Yu et al. 22020a bb; Banbury el al. [2021]
Among the weight-sharing methods, DARTS (Liu et al. (018b) addresses the challenge of scalability of the architecture search through a differentiable search formulation. It relies on continuous relaration of the architectural representation by employing efficient search using gradient descent compared with conventional approaches that use evolution, reinforcement learning, sampling, and random searches over discrete and non-differentiable search spaces.
DARTS has introduced a novel avenue of research in the NAS domain by circumventing the need to traverse
discrete and large scarch spaces ( [uu ct al. (2019); Wan et al. 2020); Zela ct al. (2019
; Chu ct al. 2020 (2019); Chen & Hsieh [0000]; Wang el al. 20021 15 Zhang el al. (2021); Yang et al. 2022
Liang et al.
Zhou et al.
2020); Liu et al. (2021); Zhang et al. (2020a]: ; Chen et al. (2019)) Among DARTS methods,
PC-DARTS
Xu et al. (2019)) further reduces the search cost by introducing partial channel sampling during
the search operation. As a result of this strategic implementation, the framework has improved memory
efficiency and accelerates processing speeds (Elsken al. (2019))
Although existing NAS methods can find a suitable architocture for a target device and application, they suffer from computationally expensive floating point operations. These operations are nost-prohibitive, especially for resource-constrained devices. Existing research work demonstrates that lower-bit quantiization policy improves the performance of the network in terms of memory footprint, latency, computational cost, bandwidth efficiency, power, and energy consumption ( (Krishnamoorthi (018 : Choukroun et al. (2019); ;
Wang et al. (2019); Gholami et al. (202T)). However, applying quantization is not straightlorward in a
NAS setting. First, due to the search space explosion, is non-trivial to incorporate quantization during
architecture//eeiee search (Wang et al. 2019 2020); [ai et al. (2021)).
Few methods have been proposed to incorporate quantization into NAS. These methods usually exploit proxies (e.g., pretrained neural networks) to estimate the performance of the sampled rrchitecture/quantizni search (Wann et al. (2020)) Proxy-based methods usually lack accurate estimation. In addition, proxies need to be treated carefully to ensure that quantized architectures are ranked fairly. Besides, evaluation using a proxy is a time-consuming procedure (Bai et al. (2021)).
Second, it is difficult to predict, in advance, exactly how quantization will affect a neural network found by NAS method. The reason is that quantization efficiency depends the neural architecture itself. For instance, Shen et al. show that the accuracy of MobileNetv2 with full precision (fp32) is better than ResNet18. However, with the 2-bit precision, the ResNet18 accuracy is higher than MobileNetv2 (Shen et al. (2021)).
Third, it is unclear how quantization should be applied within NAS architectures. As an example, we can apply quantization to a regiment of potential SubNets within a DARTS network that results in a multitude of quantized architectures. However, there is no guarantee that the resultant SuperNet, which is formed by combining these SubNets, will have the optimal quantized architecture.
In this paper, we introduce VQDARTS, novel framework that integrates architecture and weight optimization along with a comprehensive mixed- search for both weights and activations. The framework is achieved in a single shot, end-to-end, differentiable optimization, eliminating the need
2

---

Published in Transactions on Machine Learning Research (04/2025)
for any pretraining proxy. Furthermore, VQDARTS leverage the complexity-aware training to favor the discovery of lower computational complexity (i.e., lower-bit precision) models during the search phase. In this manner, the model produced by the search stage is optimized for a computational budget determined by a user-specified hyperparameter where no grid search is requirod. In addition, VQDARTS enables a trade-off between bit-operation (BilOps) and accuracy to make a suitable fit. for the larget edge application with relatively low GPU hours.
In summary, the contributions of the paper are follows:
We propose VQDARTS, which provides a single-shot, end-to-end, differentiable framework by integrating architecture and weight search with a mixed -eexed-peecsion quantization policy for both weights and activations, without any need for pretraining and proxies.
VQDARTS provides framework for ficxible navigation in accuracy-Bite -irtappp trade-off space, where the produced model by the search stage is optimized for complexity budget determined by user without any grid-search requirement.
VQDARTS enables highly efficient and accurate architecture discovery even when ultra low bit-precision is considered (2 and 4) with negligible accuracy loss compared to the full-precision (fp32) networks.
2 Background and Challenges
2.1
Differentiable NAS Formulation
The key objective of any NAS framework to find both the optimal architecture, a', and its corresponding weights, w", which is bi-level optimization problem formulated by the following equations:
minCval(w (a), w" w"(a)= argminCurainus
(1)
where (...) and (... denote the training and validation loss, respectively. The architecture parameters are optimized over the validation dataset to avoid overfitting (Liu et al. (2018b)). To ensure differentiabilitty discrete architecture search spaces are converted to continuous ones (Wu et al. (2019a); Zhang et al. 2220bb;; Zela et al.] [2019); Cai et al. 2018)) ). In continuous relaxation, the architecture search space A is relaxed to A(D), where 0 is the continuous parameter that presents the distribution of architectures (A C A())). In this way, we can employ gradient-based methods to find the best weight and architecture
(uoo et al. (2020))
To solve these equations, we can use either single-level or bi-level optimizations. In single-level optimization, a and w are updated simultaneously in each step. While this computationally less expensive, they suffer from poor generalization due to overfitting. Bi-level optimization, on the other hand, updates and w in an alternative fashion.
2.2 Quantization
The quantization can be fixed-bit, meaning the entire network has a single weight and/or activation precision. Because different layers, filters, and channels have various impacts on the accuracy, a model's accuracy can be significantly degraded when it is uniformly quantized to ultra-low precision. This problem can be addressed with siixed-precision quantization schemes (Dong al. 2019); Habi et al. (2020); Hu et al. (021 Zhao et al. (2 In muieed-precision each layer, filter, channel, and/or activation have separate precision. It has been shown that converting floating-por point values to low-precision fixed integer values in four bits or loss can reduce memory foolprint. and latency significantly GGollmii et al. (2021); Dong el. al. (2019) Deng el al. (2020)). ). For instance, ResNet50 inference with INT4 precision can produce a speedup of 50%-60% compared to INT8 inference, emphasizing the importance of using lower-bit precision for optimal performance (Salvator et al. (201)). However, when the bit precision goes below 8-bit, reaching high accuracy, especially for large datasets, is challenging task (un et al. (2020 2022
3

---

Published in Transactions on Machine Learning Research (04/2025)
Table 1: Comparing VQDARTS with state-of-thes the-art quantization methods. "Weight-sharing" means whether the mothod is utilizing weight-sharing solution. "Singlo-shot mixed-precision" quantization policy refers to using QAT to train the SuperNet with arbitrary mixed-precision quantization policies. "Mixed-precision"" means that the method supports different precisions. "No training during search" means during the search phase there is no need for re-training the sampled network candidate. As VQDARTS does not need any sampling mechanism, we do not need to train and evaluate the selected SubNet. The rescarch community has set standard for evaluating NAS methods, using "DARTS Search Space" that has been widely used. It. has larger search space compared the MobileNet search space (Mehta al. (2022); [Zhang & Ding (2023) "Proxies Avoidance" refers to the fact that the employed technique avoids employing trained neural network to estimate the accuracy of sampled architecture. "Pretraining Avoidance" means the method can avoid
any required pretraining.
Property
DNAS
HAQ
EdMIPS
APQ
OQA
BatchQuant
SPOS
AutoNBA
MINAS
UDC
VODARTS
2018
2019
2020
2020
2021
2021
2020
2021
2019
2022
2025
Joint DNN & Quan Search
Weight: Sharing
4
4
v
Mixed Precision We. Quan
Mixed Precision Act. Quan
Single-shot Mixed Precision
Difterentiable Quan
DARTS Search Space
No Training During Search
Proxies Avoidance
Pretraining Avoidance
v
Relatively Low Train Time
2.3
Incorporating Quantization into NAS
Few studies try to incorporate quantization into NAS or employ the idea of the differentiable search for efficient quantization of a fixed architccture.
DNAS (Wu et al. (2018)) proposes a differentiable mixed-precision solution per layer that takes a fixed architecture, like ResNet, and searches over the possible quantization bit precision per layer. The method assigns probability weight to the quantization 'path' per layer such that the entire network is fully differentiable. DNAS is not. like NAS architecture, rather, it employs NAS formula for fixed architecture to find a proper mixed-precisio solution.
HAQ (Wagg et al. (2019) proposes a Hessian-aware mixed-precision quantization technique that finds the optimum bit-precision for each output channel within every layer. The importance and bit precision of each output channel are recognized basod on the Hessian values of the channel. To make this method work, is necessary have an already trained, fixed architecture network.
EdMIPS Cai & Vasconcelos 2020 opts for mixed-precision search strategies for the different filters in fixed architecture and achieves high efficiency by sharing weights for all mixed-precision operations. It employs the idea of DARTS for finding the best precision of a predefined archilecture. EdMIPS employs a block that a weighted sum of all quantized weights for a single convolution. With different precisions, this reduces the complexity of the forward and backward passes by nearly factor of N. Similar DNAS, it does not perform an architecture search on the layer types and just optimizes mixed-precision for fixed architecture.
4

---

Published in Transactions on Machine Learning Research (04/2025)
APQ (Wang et al. (2020) applics joint search for NAS, pruming, and quantization policies. This technique enables mixed-preeiision architecture search but relies proxy to decide the best precision and pruning scheme. It employs an already trained 32-bit precision predictor as a proxy. For training the accuracy predictor, APQ performs QAT on 5000 architectures and quantization policies for 0.2 GPU hours cach. Producing this predictor expensive terms of time and resource consumption.
OQA (Shen et al. (2021) develops Once Quantized for All, which achieves training efficiency by gradually shrinking the bit-width of the supernetwork after it trains via bit-inheritance to enable efficient searching for different quantization bit widths. However, this approach lacks the support of mixed-precistee search.
BatchQuant (Bai et al. 2021)) proposes a mixed quantization for OFA-like architecture and not differentiable architectures). However, it needs pretraining the supernet at the beginning, otherwise, the accuracy will drop. BatchQuant first trains SuperNet with bit width options 2, 3, 4, 32 for the first 65 epochs and then continucs training the SuperNet with 2, 3, bit width options for the rest of the training reach high accuracy. Besides, due to retraining, the total GPU hours are significantly high. Unlike VQDARTS, it proposed for the MobileNet search space.
SPOS (Guo et al. 22200) employs uniform path sampling for supernet training, conducting random sampling of block and weight bit widths. The evolutionary step determines the final values. However, under SPOS, only channel searches are permitted, reducing the architecture search space size.SPOS targets fixed architecture like ResNet18 and NesNe
UDC (Fedorov enables lower bit quantization, UDC utilizes higher bit precision like 8- and 32-bit for the weight quantization.
(0222) does not support activation quantization. In addition, unlkke VQDARTS that
MNAS (Gong et al. 2019 combines NAS and quantization over the MobileNet search space. MNAS relies an 8-bit pretrained model and cannot reach high accuracy.
Except for the Batchquant, OQA, and UDC, all of the baselines rely on a proxy estimator. The proxy is usually a trained neural network that predicts the accuracy of the selected archilecture. Using proxy increases search space exploration time.
Table 1 shows various features of different techniques. It illustrates how VQDARTS is unique compared to the her existing methods. Here, different components of the proposed solutions are explained. As seen, VQDARTS provides single-shot, end-to-end differentiable NAS joint with search for both weight and activation without any need for pretraining or proxy.
quantization
3 Proposed VQDARTS
3.1 VODARTS Overview
We propose VQDARTS, a differentiable NAS framework that jointly finds optimal architecture, weight, and bit- bit-precision of both weight and activation in a single-shot, end-to-end manner. VQDARTS extends the existing DARTS (Liu et al. (2018b); Xu et al. (2019) framework by adding differentiable precision search. Unlike many existing stagg methods where architecture search and optimal quantization policy search are explored independently, joint optimization in VQDARTS allows for exploring more efficient architecture while preserving its accuracy even in a lower precision regime.
Given target task, we first follow the fixed architectural macroblocks (or cells) similar to [Liu et al. (201b)) where each cell has a set of architectural components (i.e., convolution with different kernels, pooling, etc.) o(.) E O., The output of each cell, cor is a weighted sunmation of the candidate operations:
Σ dgo(x)
(2)
OEO
where 0 is a set of learnable weights defined as:
exp (a:)
a Eo,co exp (ao.)
(3)
5

---

Published in Transactions on Machine Learning Research (04/2025)
Output
Pool 3x3
0
Normal Call
XN
Identity
Input
Output
Reduction Cell
Conv 5x5
0x
Conv 3x3
Normal Cell
XN
Reduction Cell
2
1-bit
1-bit
v
vi
Normal Cell
XN
2-b
2-bit
Input
Y2
Y2
Output
3
Input
yx
Y
Activation
n-bit
n-bit
Cell
Operation
Bit-precision
Weight
Search
Search
Search
Figure 1: VQDARTS brings quantization into DARTS. Each cell consists the operation search of DARTS, while the innermost block shows the mixed precision search candidates. Crucially, the entire SuperNet is fully differentiable. Hence, the architecture and mixed precision search can be performed simultaneously in an end-to-end single-shot fashion.
Thus the architecture search involves learning this set of continuous variables that define the relative weights of each operation over the same input. Next, consider specific operation with w weight and a activation with na and no possible bit precisions for activation and weight, respectively. The output of this operation can be formulated as:
no y
Σ Egj (x)
1
(4)
i=1
s.t.
Σ x 1, Σ
74 1, r [0.
Here of and 7/ are the weighting hyperparameters for mixed- operations (with {p. q} as the set of learnable parameters) for the i-th and j-th precision path of activation and weight, respectively, computed through the following equations:
exp (pi)
exp (q.)
55)
Σn exp (pa)'
exp (ga)
Similar to the architectural search, we leverage continuous relaxation of the bit- bit-precision search space. While the most straightforward way is to combine search over architecture and quantization bit precision space in a multi-stage fashion (i.e., find the optimal architecture in full precision, then search over the bit-precisions), we have experimentally shown in Section how such naive integration results suboptimal performance. The key challenge in incorporating such mixed-preeision search into the NAS framework that simultaneous exploration of different quantization bit-precisions per candidate operations can make the scarch space compulationally intractable. To address this challenge, we extend the originally proposed bi-level optimization in DARTS (Liu et al. (2018b)) to combine both weight and bit-precision in the following Subsection). Furthermore, to constrain the search space, we leverage the chamnl-masking mechanism proposed in PC-DARTS (Xu et al. (209 to reduce the search cost by 1/K, where K is the number of channels for each convolution operation (K>3 <>>>] 1).
unified lower variable (details
Another challenge to scarch over both architecture and prccision parameters that optimizing them over the same objective function (i.e., the validation loss) would always favor higher precision. Therefore, to penalize the optimization for high compute cost, additionally add model complexity regularizing term to the

---

Published in Transactions on Machine Learning Research (04/2025)
loss function to incentivize the discovery of model architectures with lower computational complexity during the search phase.
Figure shows the overall perspective of the VQDARTS. The mixed-precision quantization in VQDARTS considers normal and reduction cells in DARTS search space, where different normal cells and/or reduction cells can have different bit precision for both weight and activation.
3.2 VQDARTS Optimizations
Our formulation idcally involves optimization over three levels of variables: architecture parameters (a), weights (w), and bit-precision (7). While it can be desired to add as another level of inner variable to make it a tri-level optimization problem, this would increase the optimization steps exponentially. Instead, we propose an extension bi-level optimization where we consider architecture parameters as the upper variable while weight and the associated bit-precision as a joint set of lower variables. has been shown that single update both weight and quantization parameters can preserve similar performance while enjoying half the complexity of their alternating (i.e., update the weights first, then the next step involves updating bit precision parameters while keeping the weights fixed, thus doubling the update steps) updates (Cai & Vasconcelos (2020)). ).
We reformulate Eq. to introduce the mixed-precision search weighting paramcter, 7 as follows:
min L (w" (a), (a).
s.t. w (a), (a) ) argmin Lerain v,
The exact solution to Eq. 6 involves expensive inner optimization of (w, 7) over the whole training dataset. This can be avoided by approximating the gradient over a single training step (instead of the whole dataset) as follows Liu et al. (2018b)):
V VaLvallw* 2 VaLual (w - V. (w, 7, a), 7 7- v, (w, or), (x)
(7)
where Ew and Ey denote the learning rate of inner step optimization for weight and bit dit-precision, respectively. Effectively, this implics that updates on a and {w, ) occur in an alternating manner.
3.3 Complexity-wware Loss
Another challenge in joint architecture, weight, and precision search is that the existing methods predominantly rely on validation loss as its search parameter. This would always lead the VQDARTS mixed-preciste search to converge to a uniform bil-precision (i.e., the highest available bit-precision in the search space) for all the operations.
To overcome this, VODARTS introduces a pivotal enhancement: the intogration of a complexity-aware loss function denoted in Eq. g This augmentation allows for more nuanced and refined search, considering not. just validation loss but also the intricate complexities inherent in the network operations. As result, the optimization process becomes more capable of discerning varying levels of precision for different operations, leading to a more efficient search solution:
[[F] Rg(F) + vR)(F)
where [F is categorical cross-entropy loss and [F] represents the total computational cost of the given architecture. (FFF) is computed as follows:
Rc(F) = Σ OPs BitA BaW
(9)
7

---

Published in Transactions on Machine Learning Research (04/2025)
BitA = = BitW = =7,w;
(10)
OPs = InChannel x OutChannel x (kerSize)*
JnW x InH
(11)
( stride?
The hyperparameter y determines the complexity of the final architecture. A higher v will result in an architecture that favors lower mixed-precision architectures.
3.4 Increasing Model Capacity
In PC-DARTS, the training stage nceds an increase in employed cells compared to the scarch stage to increase the capacity of the architecture to reach better accuracy. Usually, the same normal/reducting cells discovered during the search stage are stacked together. However, the straightforward stacking approach will not work in VQDARTS due to variations in bit-precision among the discovered cells. This brings a challenge in selecting the appropriate cells for stacking, thereby increasing the capacity of the final architecture. Addressing this challenge necessitates a smart approach identify cells that ensure optimal performance expanding the architecture's 's capacity.
Through exploring various methods, realized technique to enhance the architecture of VQDARTS during the training phase. By duplicating the first cells and placing them at the top of the found cells, we effectively increase the capacity of the underlying architecture. This approach is motivated by the recognition that initial layers often possess weights and activations that are more sensitive to quantization and play a significant role in the final accuracy. Leveraging this understanding, VQDARTS strategically employs the first cell and stacks multiple copics of it on top of other discovered cells. As a result, the architocture's capability is expanded, leading improved accuracy. We explored alternatives such as selecting cells from the last, the middle, and employing random selection, but it the replication of the first cells that yielded the most promising results.
4
Experimental Setup and Evaluation
4.1 Setup
We have released the source code on GitHut We perform VQDARTS experiments CIFARIO and ImageNet, as the two most popular datasets for evaluating the efficiency and scalability of the NAS algorithm.
We employ HWGQ CCai et al. (201)) as a quantization schome for 2- and 4-bit and channel-wise min-max (Zmora et al. (2019); Gholami el al. (202T)) for 8-bit quantization. We used (Tev-based accelerator data
loading {Leclere al. (202 ) to reduce the training time of all basclines and VQDARTS.
Unless explicitly mention it, we use the same sctting of PC-DARTS for the DARTS based basclines and VQDARTS. For APQ (Wang et al. (2020)), OQA (Shen et al. (021 SPOS (Guo et al. (020 ), and MNAS (Gong et al. (2019) baselines, we use the best setting introduced in the manuscripts.
The experiments are carried out on 8 (1) A40 GPUs on our internal clusters for ImageNet (CIFAR10) datasets. The training batch size 2048(256) for ImageNet (CIFAR10) datasets. We consider 500 epochs for finetuning the scarched architecture for all our experiments.
During the search phase, we use different optimizers for the w, a, and 7. For the w, an SGD optimizer with learning rate of 0.5 (0.1 for CIFAR10), momentum 0.9, and weight decay of 3 x 10-4 is used. For the a, Adam optimizer with learning rate 6 x 10-3 (6 10-4 for CIFAR10), betal 0.5, beta2 0.999, and weight decay of 10-3 is used. Finally, for y, an SGD optimizer with a learning rate of 0.01, momentum of 0.9, and weight decay of 10-3 is used.
4.2 Baselines
Unfortunately, there is no dircct SOTA work that is comparable to our work mentioned in Table [ However, we show how much VQDARTS can reduce the BitOps compared to the full-precision PC-DARTS
8

---

Published in Transactions on Machine Learning Research (04/2025)
Full Precision DARTS SuperNet Search
Train Full Precision DARTS SubNet
PTQ (Fixed Bit)
Total Hours= 78.9
PTQ
Full Precision DARTS SuperNet Search
Quantization Aware Training (QAT)
Total
FBQAT
(Fixed Bit)
Hours= 104.3
QSubNet
Full Precision DARTS SuperNet Search
Quantization Search on DARTS SubNet (Mixed Bit: 2, 4)
Quantization Aware Training (QAT) of DARTS SubNet (Mixed Bit: 2,
Total Hours 123.9
AQDARTS
Joint Precision and DARTS SuperNet Search
Quantization Aware Training (QAT) DARTS SubNet (Mixed Bit: 2, 4)
Total Hours= 96.6
Figure 2:
Comparing the total cost. (both search and Sinetune/retraa rain time) of various baselines against. VQDARTS cost to evaluate the effectiveness VQDARTS. We show the total cost in terms of GPU hours for the ImageNet dataset.
(32-bit) at the cost of negligible accuracy drop. In addition, we consider assorted basclines that can be employed to be compared against VQDARTS shown in Figure
Post Training Quantization (PTQ): A simple alternative to VQDARTS is applying post-training quantiza- tion (PTQ) on the final trained architecture with fp32 precision (i.e., SubNet) discovered by PC-DARTS. This requires no additional fine-tuning and therefore is highly computationally efficient. We employ asymmetric linear post-training quantization for this purpose (Gholami al. (2021)
Fixed-bit Quantized Aware Training (FBQAT): The high accuracy drop of lower-bit quantization is usually alleviated with quantization-aware training (QAT) (Shen et al. (2021)). It is well known that quantization-aware training outperforms post-training quantization, especially in low-bit regimes (Gholami let al. (202I)). Therefore, we apply QAT to the PC-DARTS full precision searched architecture using the HWGQ (Cai et al. (2017)) technique with a uniform, fixed-bit quantization policy (e.g., three experiments
with 2-, 4, and 8-bit).
Quantization on SubNet (QSubNet): To understand if VQDARTS better than a multi-staged optimization that involves NAS followed by mixed precision architecture search, apply differentiable mixed-precision quantization technique (i.e., EdMIPS) QuubNet. After 50 epochs of EdMIPS search on the architecture produced by PC-DARTS, we apply 450 epochs of fine-tuning so that it is a fair comparison against. VQDARTS.
the output of the search stage of PC-DARTS. We call this baseline
Existing Baselines: From the literature, we compare against APQ (Wang et al. 2020 ), OQA Shen et al. (2021) ) SPOS (Guo et al.] (2020)), and MNAS (Gong et al. (2019) as the closest work to VODARTS. We also
compare VODARTS with state-of-tle of-th-rrt non-NAS methods such as EMQ (Dong et al. (2023) ), HAWQ Yao
et al. 2021 OMPQ (Ma al. (2023)
4.3
Footprint Results
Table 2 and Table 3 compare accuracy, required bit operations (BitOps), and memory footprint of different basclines with VODARTS. The first column shows the employed method. The number of employed cells in the search and training stages is shown in the second and third columns. The fourth column displays bit-precision, which pertains to both weight and activation, unless explicitly stated otherwise. The accuracy results (Top-1 for ImageNet) are shown in the fifth column. The sixth and seventh columns show the improvement in terms of bit operations normalized to the PC-DARTS values. The last column indicates the amount of improvement in memory footprint normalized to PC-DARTS. The absolute values of PC-DARTS are shown in pesentheses in the first row (i.e., in terms of Gigabits for the required bit operations and megabytes (MB) for memory footprint). The difference between the two versions of VQDARTS (2,4) is the complexity-awan loss values. By changing this parameter (u in Eqn[8. we push for higher accuracy at the cost of higher BitOps.
4.4 CIFAR10 Results
As demonstrated in Table 2 the effectiveness of VQDARTS shines for CIFAR10 dataset. With a mere utilization of 2- and 4-bit operations, VQDARTS manages to achieve remarkable reductions in required bit operations by
9

---

Published in Transactions on Machine Learning Research (04/2025)
Table 2: Comparing the impact of different methods on Accuracy, BitOps, and Memory Footprint over
CIFARIO dataset
Method #Cells-search
# Cells--raii
Precision
Accuracy
BitOps (G)
Memory Footprint (MB)
PC-DARTS
8
20
32
96.25%
(634.712)
(218.977)
PTQ
20
95.59%
0.0625
0.250
PTQ
20
19.19%
0.0150
0.125
PTQ
8
20
9.99%
0.0039
0.063
FBQAT
20
95. 86%
0.0611
0.236
FEQAT
8
20
62.68%
0.0157
0.125
FBQAT
8
20
38, 45%
0.0039
0.063
QetbNet
20
2. 4
70.05%
0.0039
0.063
VGDARTT
93.43%
0,0056
0.047
VQDARTS
2.4
94.68%
0.0062
0.062
VQDARTS
2. 2,4.8
95.95%
0.0185
0.081
VQDARTS
8
200
2, 4,
8
96.20%
0.0556
0.198
staggering 160x, along with substantial savings memory footprint of 16x while the accuracy drop 1.57%.
VQDARTS showcases its capacity to attain the same level of accuracy as PC-DARTS while simultaneously roducing the required operations by 18x and the memory footprint by 5x.
As can be observed, VQDARTS achieves remarkable accuracy in CTFAR1O without having to increase training capacity as compared to search capacity. (i.e,, merely 0.3% lower accuracy than fp32 with PC-DARTS). This exemplifies the efficiency of VQDARTS in determining the optimal architecture and weight configurations, which enables us to reach high accuracy with less capacity. The last row in Table 2 shows that VQDARTS reaches the same accuracy as PC-DARTS with 18x less BitOps and 5x less memory footprint.
Comparing against the best outcomes achieved by other baselines (i.e., QSubNed), it is worth noting that VQDARTS excels. With 1.58x the bit operations required by QSubNet, VQDARTS improves the accuracy by improssive 24.63% while outperforming QSubNet in terms of consumed GPU hours by 28%. This further highlights the superiority of VQDARTS in optimizing accuracy while maximizing efficiency in comparison to other baselines.
4.5 ImageNet Results
As shown in Table 3. for the ImageNet dataset, we observe signíficant accuracy degradation (i.e., 1%) for the QSubNet when utilizing 2- and 4-bit operations compared PC-DARTS. In contrast, VQDARTS, with 1.5x less required BitOps, reaches 21% higher accuracy than QSudNet.. Although VQDARTS with only 2- and 4-bit precision for both activation and weights experiences an accuracy drop of 1.7% compared to PC-DARTS, it reduces the required bit operations by an outstanding 65.36x and the memory footprint by 5x.
Moreover, by incorporating 2-, 4-, and 8-bit operations, VQDARTS further mitigates the accuracy drop to a mere 1%. Particularly, this is achieved by reducing the required bit operations to a notable 17x, underscoring the superior performance of VQDARTS in optimizing accuracy and reducing computational requirements on the ImageNet dataset.
4.5.1 Compare with SOTA
[Wang et al. (2020)), ), MNAS (Gong et al. (2019 ). OQA (Seen et al. (202T)) )). and SPOS (Guo et al. (2020) as the closest. state-of-the-art baselines to VQDARTS as shown Table APQ, SPOS, OQA, and MNAS converge at the accuracy of 72.1%, 71.5%, 74.1%, and 71.77% while APQ and MNAS methods use even 8-bit reight/activatte
We also compare VODARTS with APQ
In contrast, by only using 2- and 4-bit, VQDARTS outperforms APQ, SPOS, OQA, and MNAS by 2.3%, 2.9%, 0.3%, and 2.7% in terms of accuracy. By involving 8-bit precision, VQDARTS outclassos APQ, SPOS, OQA, and MNAS by 3%, 3.6%, 1%, and 3.4%, respectively. In terms of BitOps (memory footprint), by considering almost the same accuracy, VQDARTS outperforms APQ, SPOS, OQA, and MNAS by 2.3x (12x), 23x) 13%(6.)x),, 337%)
Figure 3 illustrates the accuracy-BitOp trado-off for different versions of VQDARTS compared to other state-of-the-art methods. As shown, VQDARTS effectively pushes the frontier toward the top-left, achieving higher accuracy with fewer BitOps.
10

---

Published in Transactions on Machine Learning Research (04/2025)
Table 3: Comparing the impact of different methods on Accuracy, BitOps, and Memory Footprint over
ImageNet datasct
Method
#Colls-scarch
# Cells-train
Precision
Accuracy
BitOps (G)
Memory Footprint (MB)
PC-DARTS
14
76.1%
[636.052)
((172.102)
PTQ
14
67.47%
0.0625
0.254
PTQ
8
14
4
1.0%
0.0157
0.127
FEQAT
8
14
8
74.01%
0.0622
0.244
FBQAT
8
14
50.27%
0.0156
0.127
FIOOT
33.57%
0.0039
0.063
QeabNel
50.70%
0.0156
0.127
APQ
21 blocks)
(21 blocks)
4,6.8
72.1%
0.0239
1.284
MNAS
(22 blackss)
(22 hlocks)
2,4,6.8
71.77%
0.0347
0.147
OQA
(21 blocks)
(21 blocks)
74.1%
0.0173
1.245
SPOS
(20 blocks)
(20 blocks)
1,2.3.4
71.5%
0.0244
0.321
VQDARTS
8
2,4
71.70%
0.107
VODARTS
2,4
74.41%
0.0153
0.199
VQDARTS
8
14
2,4,8
75.10%
0.0587
0.385
Accuracy vs BitOps(G)
76
SQDARTS-2b4b(2)
VODARTS-2b440 2b4b8bbbbbb FBQAT-8b
V0DARTS-2b4D(JPOAAB 4b APQ-4b6b8b
MNAS-2b4b668b
72
SPOS-1b263b4b
68
PTQ-8b
60
Accuracy (%)
56
52
Qsubnet-2b4b
FBQAT-4b
48
40
FBQAT-2b
32
0.01
0.02
0.03
0.04
0.05
0.06
BitOps (G)-Normalized to PC-DARTS
Figure 3: Comparing accuracy-BitOps trade-off of differentiae version of VQDARTS and SOTA methods over ImageNet. VQDARTS outperforms existing approaches by providing higher accuracy and/or lower BitOps.
4.6 Total Cost
Figure 2 and Table offer comprehensive depiction of the total needed time (i.e., search and training), measured in GPU hours, for various methods the ImageNet dataset 8 A40 GPUs. Upon observation, it is evident that VQDARTS takes 28% (8%) shorter runtime compared to QSubNet (FBQAT), and longer runtime compared to PTQ. PTQ lakes less time since it assumes that an expensive, fully trained network is available to be quantized with fixed-bit bit and no extra fine-tuning is required. Compared with state-of-the-an art methods VQDARTS improves the total cost significantly. VQDARTS improves the total scarch and training cost by 3.1x, 1.54x, compared with APQ, OQA, respectively. SPOS has less GPU hours at the cost of low accuracy, high BitOps, and large memory footprint.
11

---

Published in Transactions on Machine Learning Research (04/2025)
Table 4: Comparing the total GPU hours (8,A40) of VQDARTS and other methods on ImageNet dataset
Method
APQ Wang (2020
OQA Shen et al. (2021
SPOS Guo al. (2020) VODARTS
GPU Hours
302.6
149.4
4.7 Compare with Non-NAS Quantization Methods
We compare VQDARTS with non-NAS state-of-the-ar quantization schemes in Table 5 The numbers are the highest accuracy that these methods can reach with their corresponding BitO normalized PC-DARTS). Among them, only HAWQ supports mixed-precision for both weight and activation.
In terms of accuracy, OMPQ, EMQ, and HAQW3 ResNet50 have 1.18% (1.87%), 1.6% (2.3%), and 0.29% (0.89%) higher accuracy compared to two versions of VQDARTS with (2,4) bit setting. However, their BitOps are 495% (1900%), 460% (1800%), and 489% (1870%) higher than VQDARTS (2,4). Notably, PCDARTS in FP32 demonstrates lower accuracy compared to quantized ResNet50 (76.1% vs. 76.7%). This indicates that the higher accuracy of these methods attributed to the inherent capacity of ResNet50 relative to PCDARTS. However, by increasing the number of cells PCDARTS, it can reach higher accuracy, at the expense of increased BitOps. Within the same BitOps budget, VQDARTS outperforms all these methods in terms of accuracy.
Table 5: Comparing VQDARTS with non-NAS quantization schemes on ImageNet dataset
Method
W/A
Accuracy
BitOps(
4-8/8
72.30
0.1808
MMPQQReeNNele
4-8/6
72.08
0.1398
OMPQ-ResNenr
4-8/5
76.28
0.2907
HAWQv33Ressye
4,8/4,8
70.22
0.1342
SeAWQv3-ReeNeeee
4,8/4,8
71.56
0.2162
H'WQv3-Reshee
4,8/4,8
75.39
0.2870
EMQ ResNet18
2,3,4/8
72.31
0.1715
EMQ ResNet118
2,3,4/6
72.28
0. 1313
EMQ- Resset50
2,3,4/5
76.70
0.2758
4.8
Ablation Study
In this subsection, we study different aspects of the proposed VQDARTS and perform different experiments to show the impact various parameters on the proposed methods.
4.8.1 Alternative Update Methods
the DARTS framework, bi-level optimization concurrently optimizes network weights and architecture parameters using gradient descent. The upper-level optimization targets architect parameters, whereas the lower level focuses on network weights. Conversely, one-level optimization merges these stages, simplifying the process at the potential expense of architecture quality. In VQDARTS, the complexity escalates it involves three optimization parameters: archilecture parameters (a), weights (w), and bit-precision (7).
As discussed in the main paper, incorporating 7 introduces tri-level optimization challenge, substantially complicating the optimization process. Thus, we propose an efficient bi-level optimization variant in VQDARTS.
Below, we outline various update mechanisms and their effects on accuracy and BitOps.
VODARTS (Efficient Bi-level Optimization): In VQDARTS, the sequence involves forward propagation with validation dataset, backpropagation a, a forward pass with the training dataset, and subsequent backpropagation for and 7.
min
Lval(w (a),""))))
(12)
s.L. ('(a),7"(a) = argmin w.y
Lirain (w,
12

---

Published in Transactions on Machine Learning Research (04/2025)
Bi-level-other Optimization: This optimization method (Eq. 13 starts with forward pass using validation dataset, followed by backpropagation for and 7. Then, forward pass with the training dataset. precedes backpropagation for w.
min Lval(w (a; 7,
(13)
s.t. (a, argmin Ctrain 7.
Tri-level Optimization: This optimization scenario (Eq. 14p begins with a forward pass using a vali- dation dataset and backpropagation for a. Next, forward pass with the training datasct followed by backpropagation for Finally, another forward pass with the training set leads to backpropagation for 7. min Lval(w" (a, 7,))
s.t. (a, argmin Crrin (w, 7 (w))
(14)
s.L. ("(w) argmin Lurain(w, (,,
One-level Optimization: This optimization procedure (i.e., Eq. 15 eliminates the need for a validation
dataset, conducting a forward pass with the training dataset and then backpropagating for w, a, and 7.
min (uraia(w,
(15)
Figure illustrates the variations in accuracy and BitOPs across different optimization methods where all of them use the same hyperparameter. Each label indicates optimization method, BitOps, and accuracy. Figure shows that while alternative methods reduce the accuracy, VQDARTS maintains higher accuracy with fewer BitOps. Table 6 compares the search time of optimization methods. The One-level needs only forward pass with the training dataset mini-batch, per epoch. Hence, it gives the lowest search time and lowest accuracy. Compared to Tri-Level and Bi-Level-other, VQDARTS improve the search time by 41.8% and 6.6%, respectively. Surely, for the larger datsets, we will see more improvement. For example, for the ImageNet dataset, we observe a timeout issue for Tri-Level optimization (5 days).
93.6
93.4
Accuracy(%)
93.2
93.0
QDARTS
92.8
Tri-Level
92.6
Bi-Level-Other
92.4
One-Level
92.2
2.8
2.9
3.0
3.1
3.2
3.3
3.4
3.5
BitOps(G)
Figure 4: Comparing the impact of various optimization methods accuracy and BitOps.
Table 6: Comparing scarch time (minutes) of various optimization methods over CIFAR10 datasct
VQDARTS
Bi-Level-other
LeveL
One-Level
372.28
398.83
640.02
272.08
13

---

Published in Transactions on Machine Learning Research (04/2025)
4.8.2 Validation Accuracy
Figure illustrates how the validation accuracy increases through the first 250 training epochs over the D
ImageNet dataset. As it has shown, QSubNet and FBQAT (4-bit) achieve almost similar performance and convergence rale throughout the training period and they significantly better than FBQAT (2-bit). VQDARTS outperforms all the baselines extensively from the beginning of the training.
80
70
60
Accuracy
50
40
30
13
19
25
31
37
43
49
55
61
67
73
79
85
91
97
103
109
121
133
139
145
151
157
163
169
175
181
167
193
199
205
211
217
223
229
235
241
247
#Epoches
VQDARTS(2,4)
FBQQA(44btt -2BQAT(2bit)
QSubNet(2,4)
Figure Validation accuracy for different techniques throughout the first. 250 training epochs (ImageNet).
4.8.3 Required Bit Operations Per Cell
6 compares the required bit operations of different cells of VQDARTS that have 2-,4-bit, and also VQDARTS with 2-, 4,, 8-bit against PC-DARTS. The Coll#5 and Ccll#8 are reduction cells and the rest are normal cells.
Figure
The nuanced dissimilarities observed among different normal cells in PC-DARTS can be attributed to fundamental distinction between cells positioned directly after a normal cell and those following reduction cell. Notably, the cells immediately succeeding reduction cell lead to variations in the computational requirements across normal cells in PC-DARTS. This complex architectural style highlights the detailed design decisions made to improve performance.
the case VQDARTS, the efficacy is exemplified by the substantial reduction in required operations for each cell. Specifically, VQDARTS achieves significant reduction in bit operations of up to 65.5x for the (2,4) bit configuration and 17x for the (2,4,8) bit configuration. These reductions further emphasize the effective computational optimization achieved by VQDARTS.
1000000.0
236727.3
242309.1
247858.2
247858.2
323630.5
244029.1
247858.2
323568.0
100000.0
Bitops(log)
10000.0
833.2
3482.6
3280.3
2312.1
2225.4
5447.9
3352.6
1372.3
1459.5
1546.2
1546.2
1486.4
1146.2
1000.0
606.9
606.9
100.0
10.0
1.0
1
3
6
8
Cell Index
VQDARTS (2,4,8) VQDARTS (2,4)
PC-DARTS(32)
Figure 6: Comparing required bit operations of different cells for PC-DARTS and VQDARTS (ImageNet).
14

---

Published in Transactions on Machine Learning Research (04/2025)
4.8.4 Sensitivity Analysis of Complexity Decay Parameter (v)
In this analysis, we explore the effects of altering the complexity decay parameter (v) within the context of complexity-aware loss on the resulting accuracy. Previously, in Table 2 and Table 3 of the main paper, we present diverse accuracy /BttOpp metrics for the same bit precision levels within the VQDARTS framework, highlighting the sole variable the complexity decay parameter (V). Here we extend this experiment. As shown in Table 7 altering the complexity decay parameter by factor of 10 results in a 0.58% change in accuracy the CIFAR10 dataset. However, its impact on the ImageNet dataset is significant. By changing this hyperparameter from 10-4 to 10-9, the accuracy varies by 6.81%.
Table 7: Comparing the impact of different values of complexity decay (v) on accuracy over CIFAR10 and
ImageNet datascts
Method
# seells-eaarch
#Cells-train
Precision
Complexity decay
Accuracy
Dataset
VQDARTS
8
2,4
10-3
94.28 %
CIFARIO
VQDARTS
8
8
2,4
10
94.43%
CIFARIO
VQDARTS
8
2,4
10 10-9
94.49%
CIFARIO
VQDARTS
8
8
2,4
10
94.86%
CIFARIO
VQDARTS
8
2,4
10
64.45%
ImageNet
VQDARTS
8
14 2,4
10-9
71.26%
ImageNet
Employing Discovered Precision of the first/last Cell for Stacking Cells During Retraining Phase: As we mentioned in Section 3.4, increasing the model capacity in VQDARTS is not straightforward as different cells have different precision. Table g shows how stacking the cells using the discovered cells will impact the accuracy. As seen, by replicating the first cells and positioning them at the top of existing cells, we cnhance the capacity the underlying archilecture compared to using the cells discovered the end.
Table 8: Study the impact of employing the discovered precision of the first cell and the last cell for stacking cells during the retraining phase on accuracy over ImageNet dataset
Method #Cells-search #Cells-train Precision
Position
Accuracy
QQAARTS
2,4,8
last
74.608%
V QDARTS
8
14
2,4,8
first 75.04 %
In the appendices, we have provided more ablation studies to show how the bit-precision parameter (1) evolved during the search phase. Furthermore, we study the impact of the complexity decay parameter, having additional cells in the training phase.
5 Conclusion
The VQDARTS method represents significant. advancement in architedture-wighll-reeess joint search, skillfully integrating weight values, architecture, and bit-precision (for both weight and activation) search into a single-shot and differentiable framework. VODARTS approach not only eliminates the need for proxy and pretraining but also demonstrates its effectiveness through a notable enhancement in accuracy, reducing required bit operations and memory footprint. VODARTS achieves remarkable results on CTFAR10 using (2,4) bit precision compared to fp32, cutting bit operations by 160x while only decreasing accuracy by 1.57%. Enhancing the capacity allows VQDARTS to equal fp32 accuracy, decreasing bit operations by 18x. Using (2,4,8) bit precision, VQDARTS reduces the accuracy drop to 1% relative to fp32, while significantly cutting bit operations by 17x x and memory footprint by 2.6x. Regarding bit operation (memory footprint), VQDARTS surpasses APQ, SPOS, OQA, and MNAS with comparable accuracy by APQ, SPOS, OQA, and MNAS with similar accuracy by 2.3x (12x), 2.4x (3x), 13% (6.2x), 3.4x (37%), respectively. VQDARTS improves the total search and training cost by 3.1x, 1.54x, compared with APQ. OQA, respectively. These pieces of evidence show that VQDARTS holds promise for efficient and powerful network design, showcasing the potential of joint quantization with NAS in optimizing computational resources while maintaining high accuracy.
15

---

Published in Transactions on Machine Learning Research (04/2025)
References
AMD. Amd alveo u250 accelerator card, 2023. URL www.:aam..comii.. http://wwww....oc//////////////s
alveo/uu50/=-u20-aagg pq- html Product webpage.
Haoping Bai, Meng Cao, Ping Huang, and Jiulong Shan. Batchquant: Quantized-for-all architecture search with robust quantizer. Advances in Neural Information Processing Systems, 34:1074-1085, 2021.
Rajeev Balasubramonian, Andrew Kahng, Naveen Muralimanohar, Ali Shafiee, and Vaishnav Srinivas. Cacti 7: New tools for interconnect exploration in innovative off-chip memories. ACM Transactions on Architecture and Code Optimization (TACO), 14(2):1-25) 2017.
Colby Banbury, Chuteng Zhou, Igor Fedorov, Ramon Matas, Urmish Thakker, Dibakar Gope, Vijay Janapa Reddi, Matthew Mattina, and Paul Whatmough. Micronets: Neural network architectures for deploying tinyml applications on commodity microcontrollers. In A. Smola, A. Dimakis, and Stoica (ods.), Proceedings of Machine Learning and Systems, volume 3, pp. 517-532, 532, 2021. URL https://rrcroeeeinngg pdf
Llays.org/peee http:/pppr_ffleee//eeeaeeee
Payman Behnam, Jianming Tong, Yangyu Chen, Yue Pan, Pranav Godikar, Abhimanyu Bambhaniya, Tushar Krishna, and Alexey Tumanov. Subgraph stationary hardware-software inference co-design. In Proceedings of 6th Conference on Machine Learning and Systems (MLSys 23), 2023.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and 2018.
hardware. Xiv preprint
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for elficient deployment. arXiv preprinl arXiv:1908 09791, 2019.
Zhaowei Cai and Nuno Vasconcelos. Rethinking differentiable search for mixed- neural networks. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 2338,, 2020.
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. In Proceedings of the IEEE conference computer vision and pottern recognition, pp. 5918 1126669666611 2017.
Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IREE/CVP international conference on computer vision, pp. 12270-12288 2021.
Thomas Chih Chen. Eaploiting Sparsity, Compression and In-Memory Compute in Designing Data-Intensing Accelerators. PhD thesis, University of Michigan, 2019. URL https://deepblue.. .ibb. umi edu/ Eatatreaa/handdee///
Xiangning Chen and Cho-Jui Hsieh. Stabilizing differentiable architecture search via perturbattion-bssed regularization. In International conference on machine learning, pp. 1554 1565. PMLR, 2020.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the TEFF/CV international conference on computer vision, pp. 1294 1294-1303, 2019.
Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3009 3018. IEEE, 2019.
Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair darts: Eliminating unfair advantages in differentiable architecture search. In Compuler Vision-ECCCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 28, 2020, Proceedings, Part XV, pp. 465-480. Springer, 2020.
Google Coral. Coral ai datasheet. Online, 2020. URL https://coral.
Accessed: 2024-03-11.
16

---

Published in Transactions on Machine Learning Research (04/2025)
DeepBlue. Energy costs in memory hierarchy. University of Michigon Publications, 2023. Accessed: 2024-03-11.
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware accelcration for neural networks: A comprehensive survey. Proceedings of the IEEE,
2020.
Peijie Dong, Lujum Li, Zimian Wei, Xin Niu, Zhiliang Tian, and Hengyue Pan. Emq: Evolving training free proxies for automated mixed precision quantization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17076-17086, 2023.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with Conference Computer Vision, pp. 293-302, 2019.
In Proceedings of the IEEE/CVF International
Quentin Ducasse, Pascal Cotret, Loic Lagadec, and Robert Stewart. Benchmarking quantized neural networks on fpgas with finn. arXiv preprint arxiv:2102.0 01341, 2021.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: survey. The Journal of Machine Learning Research, 2011:1197- 2019.
Igor Fedorov, Ramon Matas, Hokchhay Tann, Chuteng Zhou, Matthew Mattina, and Paul Whatmough. Ude: Unified dnas for compressible tinyml models for neural processing units. Advances in Neural Information Processing Systems, 35:18456-18471, 2022.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. arXiv preprint ar arXiv:2103. 13630, 2021.
Chengyue Gong, Zixuan Jiang, Dilin Wang, Yibo Lin, Qiang Liu, and David Pan. Mixed precision neural architecture search for energy efficient deep learning. In 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 1-7. IEEE, 2019.
Google Coral. Coral m.2 dual edge tpu datasheet, 2024. URL https://ooral.aa/ssttit////
pdf Accessed: 2024-11-13.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pp. 460. Springer, 2020.
Hai Victor Habi, Roy H Jemnings, and Arnon Netzer. Hmq: Hardware friendly mixed precision quantization block for cnns. In European Conference on Computer Vision, pp. 448-463 Springer, 2020.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William Dally. Eie: Efficient inference engine on compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3):243 24,2 2016.
Peng Hu, Xi Peng, Hongyuan Zhu, Mohamed M Sabry Aly, and Jie Lin. Opq: Compressing deep neural networks with one-shot pruning-quantzation. In Proceedings of the AAAI Conference on Artificial Intelligence, volune 35, pp. 7780 7788, 2021.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schncider, Barnabas Poczos, and Eric P Xing. Noural architecture search with bayesian optimisation and optimal transport. Advances in neural information processing systems, 31, 2018.
Alind Khare, Animesh Agrawal, Aditya Annavajjala, Payman Behnam, Myungjin Lee, Hugo Latapie, and Alexcy Tumanov. Superfednas: Cost-cfficient federated neural architectura scarch for on-device inference. In European Conference on Compuler Vision, pp. 161 179. Springer, 2024.
Raghuranan Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. Xiv preprint ar 1ix:18806 08340 2018.
17

---

Published in Transactions on Machine Learning Research (04/2025)
George Kyriakides and Konstantinos Margaritis. An introduction to neural architecture search for convolutional networks. arXiv preprint arXi::200.11 10774, 2020.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. FFCV: Accolerating training by removing data bottlenecks. htttps://github www.etttu.....mmmm/emmmmme womm//bffc//ffc/// 2022. commit XXXXXXX.
Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping. arXiv preprint or Xiv: 909.06035, 2019.
Ji Lin, Ligeng Zhu, Wei-Ming Chen, Wei-Chen Wang, Chuang Gan, and Song Han. On-device training under 256kb memory. Advances in Neural Information Processing Systems, 35:22941 22954, 2022.
Aoming Liu, Zehao Huang, Zhiwu Huang, and Naiyan Wang. Direct differentiable augmentation search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12219 12228, 2021.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European conference computer vision (ECCV), pp. 19-34, 2018a.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukeuoglu. Hierarchical representations for efficient architecture search. arXiv preprint arXiv:1711.00436, 2017a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint MXix:180600005: 2018b.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEFE international conference computer vision, pp. 2736 2744, 2017b.
Yucxiao Ma, Taisong Jin, Xiawu Zheng, Yan Wang, Huixia Li, Yongjian Wu, Guannan Jiang, Wei Zhang, and Rongrong Ji. Ompq: Orthogonal mixed precision quantization. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp.
2023.
Medium.com. Overview of edge tpu memory, Online, 2020. URL httth:://mediumm com Accessed: 2024-03-11 11.
Yash Mehta, Colin White, Arber Zela, Arjun Krishnakumar, Guri Zabergia, Shakiba Moradian, Mahmoud Safari, Kaicheng Yu, and Frank Hutter. Nas-bench-suite: Nas evaluation is (now) surprisingly easy. Proceedings of the 39th International Conference Machine Learning, 2022.
Renato Negrinho and Geoff Gordon. Deeparchitect: Automatically designing and training deep architectures.
arXiv preprint 1eXxx:1704.08777 2017.
Angshuman Parashar, Priyanka Raina, Yakun Sophia Shao, Yu-Hsin Chen, Victor A Ying, Anurag Mukkara, Rangharajan Venkatesan, Brucek Khailany, Stephen W Keckler, and Joel Emerr. Timeloop: A systemalic approach dnn accelerator evaluation. In 2019 IEEE international symposium on performance analysis of systems and software (ISPASS), pp. 304-315. IEEE, 2019.
J. Thomas Pawlowski. Prospects for memory, 2019. URL httpp:///as..a.ag.ttii.///ororrr Minbemtttaiion//MCCPCCC Pawlowski keynote. (fff Keynote presentation at the Workshop on Memory- Centric High-Performance Computing (MCHPC).
Ravi Rao. ASIC vs FPGA: comprehensive comparison. Wevolver, 2023. URL https://www.wevoolve com/artiicle/asic-va--ra
Manas Sahni, Shreya Varshini, Alind Khare, and Alexey Tumanov, Compofa: Compound onco-for-all networks for faster multi-platform deployment. arXiv preprint ar Xrxi:22104. 12642, 2021.
18

---

Published in Transactions on Machine Learning Research (04/2025)
Dave Salvator, Hao Wu, Milind Kulkarni, and Niall Emmart. Int4 precision for ai inference:. hllps://deseloper: wtti:///deeelleer..iiiaaaa mavida. wwwadia.ccom/aa com/bloog /llgg/inl/ for---nfeeennce,, 2019.
Mingzhu Shen, Feng Liang, Ruihao Gong, Yuhang Li, Chuming Li, Chen Lin, Fengwei Yu, Junjie Yan, and Wanli Ouyang. Once quantzation-awwree training: High performance extremely low-bit architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5340-5349, 2021.
Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi Viji Srinivasan, and Kailash Gopalakrishnan. Ultra-low precision 4-bit training of deep neural networks. Advances in Neural Informalion Processing Systems, 33:1796 1807, 2020.
Zhenhong Sun, Ce Ge, Junyan Wang, Ming Lin, Hesen Chen, Hao Li, and Xiuyu Sun. Entropy-driven mixed-precision quantization for deep network design. Advances in Neural Information Processing Systems, 35:21508 21520, 2022.
Karen Ultrich, Edward Meeds, and Max Welling, Soft weight-sharing for neural network compression. arv preprint arxi (tte::77020444 04008, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Shreyas Kolala Venkataramaniah. Energy efficient asic/fpga neural network accelerators, 2022.
Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pallern Recognition, pp. 12965 12974, 2020,
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8612-8620. 2019.
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking architecture selection in differentiable nas. Xiv preprinl orXiv:2108.044 04392, 2021.
Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang, Yujun Lin, and Song Han. Apq: Joint search for network architecture, pruning and quantization policy. In Proceedings of the IEEE/CVF 2020.
Conference on Computer Vision and Pattern Recognition, pp.
Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search. arXiv preprint 1xki::8122000099 2018.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardwaro-aware efficient convnct design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 10734 10742, 2019a.
Yannan Nellie Wu, Joel S Emer, and Vivienne Sze. Accelergy: An architecture-level energy estimation methodology for accelerator designs. In 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 1 8. IEEE, 2019b.
XiLD Finn: framework for fast, scalable quantized neural network inference on fpgas- documentation, 2023. URL httpa://sillixx.iittmee www.tt..it/ffiiii...o
Online documentation.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc-darts: Partial channel connections for memory-effeieent architecture search. arXiv preprint arXiv:1907.05m2 2019.
19

---

Published in Transactions on Machine Learning Research (04/2025)
Yibo Yang, Shan You, Hongyang Li, Fei Wang, Chen Qian, and Zhouchen Lin. Towards improving the consistency, efficiency, and flexibility of differentiable neural architecture search. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6667 6676, 2021.
Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qijing Huang, Yida Wang, Michael Mahoney, et al. Hawq-v3: Dyadic neural network quantization. In International Conference on Machine Learning, pp. 11875 11886. PMLR, 2021.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabricl Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architecture search with big single-stage models. In Computer Vision- FFCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part VII 16, pp. 702 717. Springer, 2020a.
Kaicheng Yu, Rene Ranftl, and Mathicu Salzmann. How train your super-net: An analysis of training heuristics in weight-sharing nas. arXio preprinl urXiv:2003.042: 04276, 2020b.
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Under- standing and robustifying differentiable architecture search. arXiv preprint arXiv:1909.088 09066, 2019.
Jiuling Zhang and Zhiming Ding, Rethink darts search space and renovale new benchmark. Proceedings of the 40th International Conference on Machine Learning, PMLR 2023, 2023.
Miao Zhang, Huiqi Li, Shirui Pan, Xiaojun Chang, Zongyuan Ge, and Steven Su. Differentiable neural architecture search in equivalent space with exploration enhancement. Advances in Neural Information Processing Systems, 33:13341 13351, 2020a.
Miao Zhang, Steven W Su, Shirui Pan, Xiaojun Chang, Ehsan Abbasnejad, and Reza Haffari. idarts: Differentiable architecture search with stochastic implicit gradients. In International Conference on Machine Learning, pp. 12557-12560 PMLR, 2021.
Xinbang Zhang, Zehao Huang, Naiyan Wang, Shiming Xiang, and Chunhong Pan. You only search once: Single shot neural architecture search via direct sparse optimization. IEFE Transactions on Pattern Analysis and Machine Intelligence, 43(9):2891-298) 2904, 2020b.
Zhengyuan Zhang, Dong Zhao, Renhao Liu, Kuo Tian, Yuxing Yao, YuanChun Li, and Huadong Ma. Camonet: On-device neural network adaptation with zero interaction and unlabeled data for diverse edge environments. IEEE Transactions on Mobile Compuling, 2024.
Liang Zhao, Kumming Shao, Fengshi Tian, Tim Kwang-Ting Cheng, Chi-Ying Tsui, and Yi Zou. A flexible precision scaling deep neural network accelerator with efficient weight combination. ai preprint wwXhh:2502.00657 2025.
Sijie Zhao, Tao Yue, and Xuemei Hu. Distribution--waree adaptive multi-bit quantization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9281 9290, 2021.
Pan Zhou, Caiming Xiong, Richard Socher, and Steven Chu Hong Hoi. Theory-inspired path-rogularized differential network architecture search. Advances in Neural Information Processing Systems, 33:8296 8307, 2020.
Neta Zmora, Guy Jacob, Lev Zlotnik, Bar Elharar, and Gal Novik. Neural network distiller: python package for dnn compression research. https://arriv.. oeg.aioorg////oo// org/ans/////aaae/oo/ooo htt//////1910... 1..00.22294 12232, October 2019.
20

---

Published in Transactions on Machine Learning Research (04/2025)
A Appendix
A.1
Visual Analysis of Bit Precision and Architecture Identified by VQDARTS
The plots Fig. 9 showcase the structure of the discovered cells within VQDARTS when employing (2,4,8) bit configuration for the ImageNet dataset, juxtaposed against PC-DARTS (i.e., first row). A noteworthy observation within the visualization is the strategic selection of bit precision for various convolution blocks. Notably, 2-bit precision is predominantly chosen for many of the 5x5 convolution blocks, while a higher precision of either or 8 bits is judiciously employed for the 3x3 convolution blocks. This representation demonstrates VQDARTS' ability to uncover computationall efficient operations for both weight and activation within each cell, attesting to its optimization capabilities. Figures and show detailed analysis of which operations tend to which precision for the discovered architecture in Figure for both weight and activation, respectively. In the label of each bar, the first number shows the cell number and the second number shows the layer number. As shown, significant portion of layers (about 54%) employ 2-bit weights, indicating heavy use of ultra-low precision weights in the network. Roughly, 32% of layers use 4-bit weights, and 14% use 8-bit weights. For the activation, we observe the same trends. The contributions of 2,4, and bits are 52%, 26%, and 22%. Usually 2-bit precision is chosen for bigger kernels, and 8-bit precision is selected for skip connections. is noteworthy that the other layers cells 2 and are pooling layers, which we do not quantize them.
0-1200 pon, 545
02 red not
Sa6
06s comooooo oot
08:00 cony Sa5
0.d 565
comnec
1-1000 pon, 545
5as
1-Sarp curmes
1:40 cone Sa5
commed
5-140 cone Se5
porn 363
S-orkp curmer
5--mm cane 515
sawepp corwed
Kcheetty 345
4-4d pony 363
1-bocip, comess
Sa5
4--5ip pesiipsomees comved
6-1ee com 340
62e0ip cornect
340
6-6-1 poon 30
6-6 d- Sad
67-1 Sa0
6e-si commeed ees
7-1 top comn S4G
T-bokip.coo comee Sag
7- poon 3x3
7 G Sas
18:00 commect
1:7 SkG
7 commers eot
2.4 con Sx5
03300 soin
porn
1:340
1:74
5-400
Call-layer-perratoo
Figure 7: Operation-wise precision analysis of weights for discovered architecture in Figure
7
(chap cany Se
o24io cornec
(ddd) comy 30
ssablo cornect
odemm oae Sa
Gavvii poree
cin Se
1-dd) com, 30
1osalo.co connect
Idee own Sa
gon 30
1iaaio cornec
3-6d1 s-dd_oom pomm 300
seainn s connec
seaalo.co connect
a-dd Leddoom__e 30
s conner
4. comu Sh
4:dd Low Dd
se conned
6-1 torp wwwppomm_eeee Ind
e- se-kiocoomeee connect
2ap.pp... somy SN
s-d Lomm 3x0
a connect
669229 200.0.o.mmm. somu SNG
6.7d Loom 56
connect
7:1 100 wispppccoo comm ENd
7:3 conv 5v6
7441 Lomms 3x0
cornect
782299 www..mm........... SN
7:74 Lowy 566
re reakio.ormed cornect
240000 comi SuB
sesioo seallo.ooo
3-24iio,
5-0000
Figure 8: Operation-wias precision analysis of activation for discovered architecture in Figure
21

---

Published in Transactions on Machine Learning Research (04/2025)
PC-DARTS Normal Cell
PC-DARTS Reduction Cell
3x
cony 3x3
[k-2]
Sel
S49 con Ex5
skig cornet
dl Al.
dkt cornect
1ep_oxvv_
c.0k)
skip_comeet
3d
CI
(.-3))
sep rave 3c3
opp
dil Lony 3x5
C 1--1
AQDARTS Normal Cell
AQDARTS Normal Cell
5x
sep_coon
skip_ skip_comnect connect aed ard
skip wet
conv
pnd
(k-2)
sep_corv_Sxt
sep
Lrp_ooov..C conv__xs 5x5
(1-2)
cony, 5x5
und and
CL
skip_coneett
ConY 5
rimaget
skip conv
and
(_(1]
AQDARTS Normal Cell
AQDARTS Normal Cell
5x5
sep. cony 5x5 w55,
skip__cnnect wet and
skip connect
dil conv 18.. ped
di conny
[kk2
sep_conv__5a 5x5
Sepp JOba 5x5
Exdi
9C' CODV Sx55
c_(k)
saip Comnert
skip conneeL
diLy 5xB
Ckip set an
convy 5 M-3,317
C [k-1
AQDARTS Normal Cell
AQDARTS Normal Cell
scp_cons: 5x5 23.82 skip ConLeed
sep_poov _53 and,
dil comN
skip connect
dil cocv Jes A-d au
sep_eomy_aee facs
ael.und
Sep 200Y
c_ rep_conv_stt
seg. CONN 5x5
C_IN)
skip Comnert
skip connect..... 2
dil_comy 5ee5 w
skrp
skip__come connees
ax2
2014
[K-11
AQDARTS Reduction Cell 1
AQDARTS Reduction Cell
22 Ba
VEL
rot 3ev
RUNUA
Figure 9: Comparing the discovered cell structures with bit precision of all normal and reduction cells (iee. row 2-5) with PC-DARTS (i.c., row 1) for the ImageNet datasct. Due to mixed-precision, unlike PC-DARTS, the cells VODARTS are not similar.
22

---

Published in Transactions on Machine Learning Research (04/2025)
A.2 y Evolution During Search
The plots in Figure 10 show the evolution of the bit precision (7) parameter over 50 epochs during the search phase for activation (first six plots) and weights (second six plots) on randomly selected cells, using the CIFAR-10 dataset with bit precisions (2, 4, 8). The Y-axis shows the probability of selecting each bit. precision. The results demonstrate that the proposed framework tends to favor higher precision to enhance network capacity and improve accuracy.
Call layer aap_comy____ Acthationn
layer wwwppooow______oeeeee.eee Rww. Weights
0.5
0.55
-03
4335
Cell byer diooom__Sst 5xd Acthvations
layer di__oow__see
0.31
630
415
layer pep_conv_503
Cell layer O- 10--pppporr 545 Activrrions
8.55
4.5
44
42
byer dil_oomy___
layer 2- didd_cooov__a BXS Weights
2.h
580
-200
011
Coll 5- layer 0 hat_seeoo___s_eeeeeeeeeeeee 5xS Activations
Col 5- layer sesp_eepppree _at__oov__ss__ Weights
8.64
0.5
4.00
-61
2.5
0115
Cell 5- layer 2- dill www.coovvveeee Sx5 Activations
Cel - 2 dil
5x5 Weights
6.8
006
400
10
4000
35
416
4.66
4008
Figure 10: Evolution of the bit precision parameter.
23

---

Published in Transactions on Machine Learning Research (04/2025)
A.3 Latency and Energy Measurements
Most of the existing work that study quantization implement their methods using ML frameworks (e.g.. PyTorch) and show the efficiency of their methods with simulation or proxy metrics rather than demonstrating actual inference on real hardware. Nevertheless, we employ two approaches to approximate how VQDARTS impact latency and energy on hardware.
First Approach: Simulation & Analysis
For the simulation-based results, we employed Timeloop (i.e, DNN accelerator mapper/simulator) ct al. (2019) and Accolergy (i.c., energy estimator) (Wu et al. (2019b)) to evaluate latency/cncne /energy of the VQDARTS networks and compare them with the PC-DARTS baseline fp32 precision.
Parashar
In our analysis, we use Google's Coral Edge TPU- a mobile edge Al accelerator with publicly available specifications (Coral (2020); Google Coral (2024); Medium.com (2020)) ) The Edge TPU was chosen for its representative mobile-edge performance and its well-documented specifications, making it suitable to configure Timeloop/ Accelergy tools.
Simulation Configuration: We set the clock frequency to 500 MHz and configure the Processing Elements (PE) array to match the Edge TPU's compute capabilities. We assume 64 x64 MAC units (4096 MACs total). At 500 MHz, 4096 MACs can perform almost. 4 TOP/s. We set scratchpad memory data width to 1 byte per element (since it stores 8-bit data primarily). Reading data from an 8MB on-chip SRAM costs around pJ per 8-bit word (Balasubranonian et. al. (2017); Chen (2019); Pawlowski (2019) Since model footprints exceed 8MB, we include DRAM level for main memory. We assume 320 pJ per 8-bit access for DRAM (Balasubramonian et al. (2017) | Chen (0199: Pawlowski (2019)). We ensure the on-chip SRAM and PEs have sufficient bandwidth. E.g., the 8MB may be divided into banks.
MB and the
The gramularity of the precision per layer, as mentioned in Figure 9 and we already know which operations work at what precision. We also use 0.3 pJ. 0.125 pJ. 0.03 pJ, pJ. and pJ for MAC-8bit, MAC-4bit, MAC-2bit, MAC-16bit, and MAC-32bit, respectively (Han et al. 2016 Zhao et al. (2025)).
Second Approach: FPGA Implementation
For FPGA implementation, we use AMD Xilinx Alveo U250 FPGA (MD 2023
that supports
FINN framework for low-bit neural networks via LUT-based logic for binary/ dinary/ternary ternary ops (Kilix--AMD 2023 To keep comparisons fair with the simulation, we assume the same technology node and frequency and similar memory hierarchy constraints as the Timeloop/ Sacelooo//cceerryyyy
Accelergy study.
FPGA Configuration: FINN (illnx-AMM (2023) yields pipelined design where each layer processes data in streaming fashion. The FINN accelerator would be structured to read weights from DDR, buffer them chip, and stream the input through the pipeline. Each weight is reused for all necessary MAC' (e.g., in convolutions) before loading new weights, minimizing redundant transfers.
Because FINN allows increasing parallelism when quantization reduces resource usago, the low www-prccision models can be significantly unrolled, reducing cycles. In contrast, the FP32 model demands more resources per operation, forcing more pipeline stages, and higher cycle count, demanding more weights/featun maps stream from off-chip DDR4. Power is estimated based on device utilization and toggling activity. Lower-bit operations switch fewer transistors per compute. Quantized circuits consume less dynamic power per operation (Ducasse et al. (202 ). FINN's post-implemmntation reports can provide an estimate of dynamic power given the switching rates for LUTs, DSPs, and clocks. Then, energy is computed as Power x Latency (Rao 2023
Venkalaramaniah 2022 DeepBlue 20233
Comparing FPGA and Timeloop/Accelerge (ASIC) Results:
Table 9 summarizes the latency, and energy results for the three DNN models on the FPGA (FINN) and the ASTC (Timeloop/ (Timeloop/Acceeergy)) Accelergye. All results for single inference (batch=1).
24

---

Published in Transactions on Machine Learning Research (04/2025)
Table 9: Estimated Latency and Energy Inference for Edge FPGA and ASIC under VQDARTS
Model (Gbit-ope, Memory)
Platform Latency (ms)
Compute Energy (wJ)
Memory Energy (mJ)
Total Energy (mJ)
FP32 Model (BitOps(C) 836.6)
FPGA
80.7
8.1
28.3
36.4
(Accuracy: 76.1%)
ASIC
12.9
Mixed 2/4/8-bit Model (BisOpe(G) 31.3)
FPGA
15.6
1.3
5.1
6.4
(Aocuracy: 75.1%)
ASIC
10.1
0.5
2.0
2.5
Mixed 2/4-bi Model (BiOOpe(G) 5.42)
FPGA
2.0
(Aocuracy: 71.7%)
ASIC
5.2
0.15
0.6
0.75
The FP32 design uses large bit-widths and drives off-chip DDR, approaching a high power draw, whereas the quantized designs use fewer bits, fewer bit toggles, and skip most off-chip memory, leading to lower power and latency. The mixed-precision models are far less demanding. The smallest 2/4-bit network executes in only few milliseconds, with the ASIC drawing only a few hundred milliwatts (energy 0.75 mJ). This aligns with expoctations that VQDARTS quantization shrinks the workload by an order of magnitude, yiclding lower total latency and energy versus the FP32 baseline.
Please note that the degree of VQDARTS improvement can vary based on the specific hardware device selected and its associated configuration parameters. Nevertheless, adopting VQDARTS consistently provides substantial advanlages in lerms of reduced energy consumplion and lower lateney, highlighting their polential for impact across various deployment scenarios.
A.4 Approximate Complexity Decay Parameter Tuning by Data & Model
In all datasets, by adjusting the value of the complexity decay parameter (v), we can either achieve the highest accuracy (at v 0) or minimize BitOps (at = 1). Based on our experimental observations, we derive fitting functions to estimate accuracy and BitOps for each decay value of on CIFAR10 and ImageNet datasets. We note that the behavior of these functions near zero differs from their behavior other intervals. Specifically, for CIFAR10, both accuracy and BitOps can be accurately estimated using different quadratic polynomial functions in both intervals of v. In contrast, for ImageNet, while a quadratic polynomial remains effective near zero, a logarithmic function provides a better fit for accuracy when lies within the range of 10 to 1. For CIFAR10, the accuracy and BitOps as functions of are defined piecewise as:
Accuracy(v) 2.52 x 10 2.52 x 10 + 95.95, 0 10 10-3 <ル
(16)
52.97 v2 70.10 v + 91.55,
6.599 x 10 6.599 x 10 + 9.926, 9.135 10.863 v + 1.805,
0 0 10 <V5L.
BitOps(v))
(17)
For ImageNet, the corresponding piecewise functions are given as:
9.966 10 1.0 x 109v + 75.1,
0gu510-0
Accuracy(v)
(18)
0.332 (log10 2)2 5.780 log10 a + 50.1, 10-6 <いड1,
2.326 10 2.328 x 10 + 31.49,
0 S 10-6
BitOps(v)
(19)
0.166 (log10 2²² + 0.285 (log10 v) + 0.146,
<v51
It is noteworthy that other research studies also have such hyperparameter that need to be adjusted to get the final results. For instance, ProxylessNAS (Cai et al. (2018) uses an w that controls the tradc-off between accuracy and latency. Network Slimming (Liu et al. (2017b)) and others that prune via a penalty on batch-norm scales often try a few values and pick the one that yields the desired sparsity. In Network Slimming [Liu et al. (2017b)), for CIFAR10 and SVHN datasots, the sparsity-controlinng hyperparameter X (balancing empirical loss and sparsity) is selected via grid search over {10-3 10-4, 10-53. For ImageNet, in case of VGG, it sets X = 10
25

---

Published in Transactions on Machine Learning Research (04/2025)
A.5 Applicability of DNNs for Widely Used Applications
DNNs particularly compact CNN-based models remain widely used for inference in deployment settings because they offer favorable balance of accuracy, latency, and energy. In latency- and energy-constraiie applications (mobile, smart home devices, wearable devices, AR/VR, etc), simpler DNN architectures are preferred over extremely large models due to their lower computational and memory roquirements, which translate to faster inference and lower power consumption. Numerous recent work across different domains also underscore the continued dominance of optimized DNNs in deployment (Khare et al. (024)): Zhang et al. (2044
Lin al. 2022); Behnam et al. (2023); Sahni et al. (2021)
A.6 Potential Joint Differentiable UAS-quantization Search for Transformers
Traditional DARTS-based NAS operates on CNN architectures e.g., choosing conv vs. pooling operations, cell connectivity patterns, etc. whereas Transformer-bused NAS focuses on different macro-parammeters and layer structures. In this architecture, critical design dimensions include the network depth (number of self-attention layors), the embodding dimension of tokens, and the number of attention heads per layer, among others. These factors have large impact on model accuracy and efficiency. Recent NAS approaches aim to automate this. For instance, AutoFormer (Chen et al. (0021) introduces a one-shot NAS Transformer, treating depth, embedded size, and heads as search parameters rather than fixed values. It uses weight- sharing across Transformer blocks to train many architecture variants simultancously. This illustrates that transformer-based NAS requires handling a struct ured, layer-wise search space (with choices like how many layers or heads), which a different scenario than the cell-based micro-search DARTS. Nonetheless, the diffcrentiable search philosophy can be extended to the Transformer. For example, we can assign continuous relaxation variables to Transformer choices (e.g., a parameter that selects between smaller or larger embedding sizes), similar to how DARTS selects between candidate operations.
We can potentially formulate the joint NAS and quantization search as bi-level optimization problem. Let a denote the architecture parameters (e.g., network depth, embedding dimension, number of heads, etc.), and let denote the quantization parameters (e.g., bit-precision for model weights). We use we notation for the network weights. The goal is to find the optimal architecture (including quantization policy) that minimizes validation loss Cval: while the weights w are trained to minimize training loss Lirain. This can be expressed as:
min
(val("(a,))) a,7), o s.t. w((a,)) arg min Lirain a, 7,7))
(20)
(21
Equation equation 20 is the outer loop (architecture/preet) optimization) and equation 21 is the inner loop (weight training for a given architecture). Solving this bi-level problem directly challenging because a and 7 are involved in the training process of w.
Alternatively, we can use the more optimized equations:
min
(a),
(22)
s.t. (a) aagmmin w.y
Lirain (, a)
Following the VQDARTS approach, we relax the architecture search space be continuous so that a and y be optimized with gradient-based methods alongside
26

---

Published in Transactions on Machine Learning Research (04/2025)
Continuous Relaxation of the Search Space (Architecture & Quantization)
Transformer-basee architectures have several discrete parameters such as the number of Transformer layers L, the embedding (hidden) dimension d, the number of attention heads h, and the quantization precision 7 We incorporate these parameters into the search.
To make these choices differentiable, we represent each as a weighted softmax over possible options, parame- terized by a or 7. This analogous to the differentiable relaxation VQDARTS, but extended to non-operator choices and quantization. We detail each below.
Depth (L): We let the search consider a set of possible layer counts I2....... hm} (e.g., (2,4, 63 in the search space). We assign architecture parameters aL for each candidate T.. A softmax over these yields selection probabilities for each depth:
exp(a
PT
> m j=1 (
i= 1....,,m m.
(23)
We construct SuperNet that can accommodate the maximum depth Luuax max(L.). During forward pass, we can blend the oulputs from different depths according to pr. For example, let be the oulput of the network truncated to l layers. We form the final output as weighted sum of intermediate outputs: Zfinal = E Pi. 24,- This effectively creates continuous mixture over architectures of different depths. During the search, az is learned such that it increasingly favors the optimal depth. In the end, we select the discrete depth L L, with the highest probability PL
(6)
Embedding Dimension (d): We consider a set possible hidden dimensions .h..d,...,ddd (for instance d E (128, 216,) Similarly, we introduce architecture parameters of for each candidate dj and form softmax:
exp(a (1)) exp(a (1)) j j- 1.......
Pd E
(24)
To implement a mixture dimensions, one can build the model with the largest dimension dmax and use a masking or gating mechanism. For example, each lincar layer in the Transformer can be designed to have dmax dimension, and smaller d, configurations are obtained by only using the first d, neurons (the remaining dmax dj neurons can be masked out simply not activated). In the continuous relaxation, we weight, the contribution of each dimension corresponding to a given dj by ()) Conceptually, this means the effective embedding dimension during a forward pass is an average of all candidate dimensions, weighted by pa- The architecture gradients adjust aa to favor the dimension that best balances accuracy and complexity. After searching, we discretize by choosing the dimension d d, with the highest probability.
Number of Heads (h). We also search over the number of attention heads. Let the candidate head counts be (h1, .2,..., h1) (e.g., h E {4, 16) heads). We assign parameters (r) for each h, and obtain:
exp(a
(r) Ph
F T= T=
(25)
exp(a (e)
Including different head counts in one SuperNel is achievable by computing multi-head allention with a maximum number of heads fmax then treating some heads as inactive for smaller h. In practice, if hmax is the largest option, an attention operation with fewer heads (say hy < hmax) can be represented by grouping or masking heads in the head attention. In the relaxed formulation, we can compute attention outputs for all Amax heads and then weieht//slecct the appropriate combination corresponding to each candidate hy.
27

---

Published in Transactions on Machine Learning Research (04/2025)
(r) then mix these outcomes, and ultimately we pick the discrete h that has the highest The probabilities Ph weight.
Quantization Precision (7). For weight quantization, we consider set of bit-widths (precisions) (71 72......77. 7t We introduce a set of quantization architecture parameters 7(1) (overloading notation to also denote the set of softmax weights for precision choices) for each candidate bit-width Yu- The softmax relaxation is:
exp(>/u)) Σi=1 exp()(11))
W= u=
(26)
To combine different precisions in one model, we leverage the idea of mired-precision SuperNet in VQDARTS. During a forward pass, we can simulate low-precision weights by quantizing the underlying high-precision weights w to the required bit-width. Let wi, denote the network weights quantized to %a bits. We then compute the network output for each precision candidate, 2%,- = f(2; wyr) The overall output is a weighted sum of these precision-specific outputs: 2final = E P7 Sn.- This guides y towards the bit-width that best trades accuracy for compression. After search, we select the bit-width with the highest probability (or sometimes the smallest bit-width that maintains accuracy, depending on the target criteria).
(a)
Through the above continuous relaxation, the architecture parameters a and quantization parameters y now define a soft architecture that is differentiable. The effective choices of L, d, h, and bit-width are implicitly given by the probability weights (6) (3) (r) _. Initially, these weights are typically uniform. :Pd -Ph During training (solving the bilevel problem), a and y are updated (in the outer loop) using gradients from the validation loss, while w is optimized on the training loss (inner loop). Over time, the softmax weights concentrate around the best-perrorming architectural configuration. Once training converges, we obtain a discrete architecture by picking the argmax option for each hyperparameter (or by sampling according to p and retraining, done in VQDARTS). This yields Transformer architecture with a specific depth L, embedding dimension d, number of heads h, and weight, precision y bits, all chosen to minimize validation loss.
Complexity-Awen Aware Loss Function
Simply optimizing for validation loss may lead to choosing the largest model and highest precision (to maximize accuracy). To steer the scarch towards efficient models, we incorporate complexity penalty in the objective, as outlined in the original VQDARTS methodology. We quantify the model's complexity in terms of estimated parameter count and computational cost, and add this to the validation loss with tunable weight.
First,, estimate the number of parameters Nparams (L, (L,d) d) for Transformer with L layers and embedding dimension d (assuming the feed-forward network dimension is f. and keeping other aspects constant). A single Transformer layer consists of the multi-head self-attention and the feed-forward sub-layer. The attention part coutributes roughly 4d parameters (for the query, key, value, and output projection matrices). and the feed- ford-forward part contributes about 2d f (for the two linear Transformations) (Chen al. (2021): Vaswani
et al. 2017)). Thus, the total parameter count scales as:
N Nparams (L,d) I(Ad + 2df).
(27)
where f is typically multiple of d (for example f = 4d in many Transformer designs). This formula provides rough but useful estimate of model size given L and d.
Next, we define complexity cost C(a,n) that incorporates both model size and compute cost, taking into account quantization. We have Iwo sources of complexity to consider: (1) model memory ccomputation due to the number of parameters (which is reduced by lower precision), and (2) the runtime operations for attention and ford-forward computations per layer. We denote by Oaten (d, h) the cost (c.g., number of operations) of the attention mechanism in a single layer with dimension d and h heads, and by Ona(d, f) the cost of the feed-forwal layer. We combine these as follows:
28

---

Published in Transactions on Machine Learning Research (04/2025)
c(a,7) 1.44 + 2df) 32
+ n (0rttnn (d,tt((,hh) (d, + Om)(d,
(28)
In this expression, the first term L(4d + 2df) 32 represents the effective model size scaled by the precision lactor 7/32. Here y (in bits) is normalized by 32, since full-precision model is 32-bit; Lower y directly reduces this term, favoring low-bit quantized models. The second term vy L (Oattn + Om.) captures the estimated computation cost per layer (which could be measured in FLOPs or a proxy for latency). The coefficient yj is weighting factor to balance the importance of operation count relative parameter count; can be set to reflect hardware characteristies (for instance, if memory is more constrained than compute, "/ can be set. smaller, and vice versa). Both components multiplied by L because a deeper network repeats the cost at each layer. Note that in the continuous relaxation setting, L, d, h, and y in equation equation 28 are the soft values (e.g., expected layer count, expected dimension, etc., under the current distribution).
Finally, the above complexity torm is added to the validation loss to form the augmented objective for the outer optimization. The combined objective (to be minimized with respect to a,7) is:
(va (a,7); y); a,7) + xC(a,7)
(29)
where À is a regularization hyperparameter that controls the trade-off between accuracy and complexity. A higher À places more emphasis on the complexity penalty, pushing the search towards architectures that are smaller or computationally cheaper (e.g., fewer layers, smaller d, fewer heads, lower precision), potentially at some loss accuracy. lower makes the search focus more on accuracy (the validation loss term) with less regard for model size or speed. By tuning X, we can obtain a family of architectures along the Pareto frontier of accuracy vs. efficiency.
It is noteworthy that the implementation and getting results for the Transformers are left for future work. Through the implementation phase and getting results, some of the equations might need some modifications.
29


Reviews:
[{"review_id": "HrbKMWjYqZ", "reviewer": "Reviewer_g4pC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths \n1. **Novel extension of the DARTS Framework**: The paper extends DARTS to include quantization as a differentiable search dimension. This is a non-trivial extension that requires careful reformulation of the objective function and search space to accommodate bit-precision parameters.\n    * The authors cleverly formulate bit-precision selection as a differentiable process by applying continuous relaxation similar to how DARTS handles architectural operations.\n\n2. **Single-Shot End-to-End Framework**: Unlike multi-stage approaches like APQ or OQA that require pre-training or proxy networks, QDARTS performs joint architecture-weight-precision optimization in a single shot, which is useful in resource-constrained settings.\n\n3. The complexity-aware loss function is interesting. The hyperparameter $\\nu$ allows for flexible navigation of the accuracy-complexity trade-off space without requiring grid search.\n\n4. **Comprehensive Experimental Validation**: The paper includes extensive comparisons against various baselines, including post-training quantization (PTQ), fixed-bit quantization-aware training (FBQAT), and SOTA methods. QDARTS has consistent advantages across different datasets and metrics.\n    * The search and training efficiency is also better, achieving 3.1x and 1.54x improvements over APQ and OQA respectively.\n\n\n# Weaknesses\n1. **Sensitivity to Complexity Decay Parameter**: As shown in Table 7, the complexity decay parameter ($\\nu$) has a substantial impact on the results, particularly for ImageNet (6.81% accuracy variation). The paper does not provide clear guidelines for selecting this parameter for new datasets.\n2. **Limited Architecture Search Space**: The paper builds upon the DARTS search space, which primarily focuses on cell-based architectures. Exploring how QDARTS performs with other search spaces (e.g., MobileNet-like search spaces) would strengthen the generalizability claims.\n3. **Task Diversity**: The evaluation is limited to image classification tasks. Testing on other computer vision tasks (object detection, segmentation) or even other domains would better demonstrate the versatility of the approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "(***NOTE***: *in my review, I will use the text \"QDARTS\"  to denote the approach of this paper, i.e. without using the gradient symbol in the acronym.*)\n\n1. This paper introduces QDARTS, an extension to the Differentiable Architecture Search (DARTS) framework that incorporates mixed-precision quantization as an additional search dimension. While DARTS and its variants (like PC-DARTS) have proven effective for neural architecture search through gradient-based optimization in a continuous domain, they operate in full precision and require separate quantization steps afterward, leading to suboptimal results. The key innovation in QDARTS is the seamless integration of mixed-precision search for both weights and activations directly into the differentiable architecture search process.\n\n2. The authors formulate this as an extension of the bi-level optimization used in DARTS, where architecture parameters remain upper-level variables while weight and bit-precision parameters are jointly optimized as lower-level variables.\n\n3. Experiments on CIFAR10 and ImageNet show that QDARTS can discover architectures that achieve high reductions in bit operations (up to 160x on CIFAR10 with 2-4 bit precision) while maintaining accuracy within 1-1.57% of full-precision DARTS models.\n\n4. The method outperforms other quantization-aware NAS approaches like APQ, SPOS, OQA, and MNAS in terms of accuracy-efficiency trade-offs and search efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Recommended changes:\n1. Evaluate QDARTS on at least one additional computer vision task beyond classification, such as object detection or semantic segmentation, to demonstrate its versatility across different vision tasks.\n2. Include measurements of actual inference latency and energy consumption on real hardware platforms, particularly edge devices. This would provide more direct evidence of the practical benefits of the reduced computational requirements.\n3. I am not sure if this is possible but it would be interesting to see ablation studies on different components of the bi-level optimization approach, such as comparing against a tri-level optimization where bit-precision is a separate level from weights or against a one-level approach where all parameters are optimized together.\n4. Visualize and analyze the distribution of bit-precision selections across different layers and operations in the discovered architectures. While Figure 6 shows the cell structures, a more detailed analysis of which operations tend to use which precision would be insightful.\n5. Include clearer guidelines for setting the complexity decay parameter ($\\nu$) based on dataset characteristics, model requirements, or other factors. This is crucial for practitioners who want to apply QDARTS to new problems.", "broader_impact_concerns": "The paper does not include a broader impact statement, which would be valuable given the potential applications of efficient neural networks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XuZ8e5TMaH", "reviewer": "Reviewer_gXNN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The sections of the paper are well organized.\n\n2. The paper presents a new method that combines differentiable NAS with mixed-precision search for both weight and activation.\n\n3. The method is experimentally evalauted.\n\nWeaknesses:\n\n1. Some parts of the paper are not well written and it is difficult to follow. \n\n2. The paper needs some clarifications in order to be techincally sound.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a framework that integrates architecture and weight search with a mixed-precision quantization policy for both weights and activations. The proposal is evaluated experimentally.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is not clear why applying quantization is not straightforward in a NAS setting. We could (1) use the same bitwidth across all the layers of a model and then (2) do any standard NAS method. Please explain what are the issues of doing this.\n\n2. \"PTQ takes less time since it assumes that an expensive fully trained network is available to be quantized with fixed-bit and no extra fine-tuning is required\". is this possible to be done for QDARTS? If yes, pleas present the results. If not, please explain with arguments why this is not possible. \n\n3. Some of the works mentioned in section 2.3 are not included in the experimental section (for example DNAS). Moreover ,the reader will be able to understand better the SOTA compariosn if the works are grouped in the tables and maybe sort them based on accurcy or cost. \n\n4. Please add the accuracy-BitOps trade-off plot for the proposal and on top of that include the SOTA works in this plot. \n\nMinor Comments:\n\nTable 3 is not referred in section 4.5. Please make sure that the tables and the figures are properly referred in the manuscript.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QHRYcsWAYi", "reviewer": "Reviewer_mDKj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n1. This paper is well-written and very easy to follow\n2. Detailed discussion on related work is provided, making the proposed method well-motivated and well-positioned\n3. Extensive experiments and ablation studies are conducted, including comparison with both NAS baselines and other non-NAS quantization methods.\n4. Experimental results shows the proposed method is effective in searching for efficient mixed-precision architectures \n\n## Weakness\n\nThe main weakness of the paper is that the exploration is solely based on the DARTS search space, which is old in today's standard and is limiting on the performance that it can achieve. The impact of this work can be further strengthened by showing the extension of the proposed training method on more advanced differentiable NAS search space, especially those for Transformers. One possible reference is the AutoFormer search space proposed in https://arxiv.org/abs/2107.00651", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes novel methods to include mixed-precision quantization into the differentiable NAS algorithm on the DARTS search space. The proposed framework enables a flexible navigation in accuracy-BitOps trade-off space by including a complexity regularizer into the search process. Highly efficient arhchitectures are discovered from the search, outperforming both previous NAS methods and manually-designed quantized models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please discuss the extension of the proposed method on transformer search space, such as the one proposed in AutoFormer: Searching Transformers for Visual Recognition. https://arxiv.org/abs/2107.00651", "broader_impact_concerns": "There's no concern on the broader impact of this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a framework that combines architectures search and weight optimization, along with a mixed-precision search for both weighs and activations in convolution networks. The method is evaluated on CIFAR10 and ImageNet datasets, demonstrating improved performance and searching efficiency compared to previously reported methods. 

The paper is well-written and has received favorable reviews from all reviewers. The reasons for acceptance are:

1). The method is reasonably novel, particularly in its extension of DARTS frameworks to support efficient mix-precision search and the introduction of complexity-aware loss functions.

2). The evaluation and ablation studies are comprehensive. The results clearly show improvement over baseline methods.

3). During the rebuttal, the authors provide detailed clarification and additional results, including the hardware performance and the extension the framework to transformer models.

In summary, this paper is technically solid and potentially useful for deploying models on edge devices. Hence, the paper is recommended for acceptance.
Decision: Accept (poster)