UDC: Unified DNAS for Compressible TinyML Models for Neural Processing Units
Authors: Igor Fedorov
Ramon Matas
Hokchhay Tann
Chuteng Zhou
Matthew Mattina
Paul Whatmough
Venue: NeurIPS 2022 Accept (2022)
Keywords: NAS
compression
NPU

Abstract: Deploying TinyML models on low-cost IoT hardware is very challenging, due to limited device memory capacity. Neural processing unit (NPU) hardware address the memory challenge by using model compression to exploit weight quantization and sparsity to fit more parameters in the same footprint. However, designing compressible neural networks (NNs) is challenging, as it expands the design space across which we must make balanced trade-offs. This paper demonstrates Unified DNAS for Compressible (UDC) NNs, which explores a large search space to generate state-of-the-art compressible NNs for NPU. ImageNet results show UDC networks are up to 3.35x smaller (iso-accuracy) or 6.25% more accurate (iso-model size) than previous work.

UDC: Unified DNAS for Compressible TinyML Models for Neural Processing Units
Igor Fedorov" Meta Al Menlo Park, CA 94025 fedorov. uof nofi@gmail.ccmm com
Ramon Matas" NVIDIA Santa Clara, CA 95050
Hokchhay Tann" Tenstorrent Boston, MA 01721 ctannotenstorrent com
Camonm@nvidias com
Chuteng Zhou
Matthew Mattina" Tenstorrent Boston, MA 01721
Paul N. Whatmough"
Arm Inc. Boston, MA 01721
Qualcomm Al Research
Cambridge, MA 02140 pwhatmouEqti. qual com
chu. mhuuzrhuaarm.. com
mmattina@tenstorrent com
Abstract
Deploying TinyMI, models on low-cost ToT hardware is very challenging, due limited device memory capacity. Neural processing unit (NPU) hardware address the memory challenge by using model compression to exploit weight quantization and sparsity to fit more parameters in the same footprint. However, designing compressible neural networks (NNs) is challenging, as it expands the design space across which we must make balanced trade-offs. This paper demonstrates Unified DNAS for Compressible (UDC) NNs, which explores a large search space to generate saate-of-theeart compressible NNs for NPU. ImageNett results show UDC networks are to 3.35x smaller (iso-accuracy) or 6.25% more accurate (iso-model size) than previous work.
Introduction
IoT applications demand TingML models that fit highly-constrained hardware (HW), with limited memory and compute power [14, 25, 30, 42). Canonical TinyML tasks include visual wakewords, audio keywords, anomaly detection, speech enhancement, and image classification [14, 13, 26, 9, 48). Traditional microcontroller units (MCUs) are not well suited to meet the memory and compute challenges of MinyML, HW vendors offer specialized processors for neural network (NN) inference, called neural processing units (NPUs) [2, 1).
MCC inference runtimes [7, 8) do not implement model compression, which is slow in software. Hence MCUs do not benefit from sub 8-bit quantization or unstructured pruning [31]. In contrast, NPUs with HW model compression [2, 1] do benefit from both optimizations [3]. For example, the Arm Ethos-U5: NPU compiler (Vela [3]) encodes weight tensors using two components: a binary mask marking non-zero elements (run-length and Golomb-Rie compressed), and 2) the non-zero values (Golomb-Rice compressod). A model aggressively pruned to have many zeros and quantized to low bitwidths can be deployed in very little device memory. AL inference time, weights are then decompressed by the NPU on demand.
NPU compression is enormous advantage for ML, where meager Flash memory limils model size and therefore accuracy [25, 48, 30, 42). A smaller memory footprint also reduces memory access energy, critical for battery powered IoT devices [45, 11, 31, 5). Fig. shows
Work done while at Arm Inc., Boston, MA
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

MobilNNetV2 compressed using Velo
Compresibie Model
Model Weights
Deploy
Deployed Moded (ha8)
Ouantication
Unstonetand prannng
UDC: Unified
Compressibie Models
Compleer (Vela)
Bath
HV
Prusting
Zern grg compression
TFLRe
Comprrssed
Model
2-
Search
Ruoent oation
Quantized Pruned
Compresoed
(Plash Mcmond
Layer Witch
hon Zero Weight Compression
De Compressson
Training
Arm USS
Aiponthm
Openitions
[Golomb
B-it
4-b8t
6-hit
4-bit
1-27
1-b8t
Frune >
75%
90%
95%
Figure 1: Left: NPU weight compression allows significant deployed model size reduction for ML. Compressed MB size scales with increasingly aggressive quantization and pruning choices. Right: UDC designs compressible models, tailored to NPU deployment.
how model size scales with quantization, pruming, and both, when compressing MBNet [35] for deployment using Vela, down to a 9.3x reduction in model size for 1-bit quantization and 1% non-zero weights in the extreme.
To fully exploit NPU model compression, we define our problem statement as follows:
Automatically design compressible NNs with the smallest model size.
(PI)
Typically, the model architecture itself is developed first, either through manual trial-and- error or by neural architecture search (NAS) [47, 21, 68, 33, 54, 67, 26, 62, 65). Then, model conditioning techniques, e.g., quantization [61, 69, 64, 66, 63) and unstructured pruning [18, 43, 32, 31, 50], are applied before deploying to the target HW. Furthermore, the generated model must also be small enough to deploy on the NPU device; merely regularizing size does not guarantce that the models found will fit into device Flash memory. It is essential that:
Generated NNs do not exceed a hard constraint on (compressed) model size. (P2)
This paper describes UDC (Fig. 1), which merges the model design and conditioning steps by conducting a joint search over NN architecture, weight bitwidths, and sparsity rates. UDC builds upon differentiable NAS (DNAS), which exploits efficient weight sharing to solve (P1), while addressing key challenges like how to explore the design space while still addressing (P2) [51, 56). The search space is ww--copression-aae -warre, including only model conditioning techniques supported by the real-world Vela NPU compiler. We explicitly exclude low-rank matrix factorization [38] and non- non-uniform meating-point quantization [66, 63), which are not supported. The contributions of this paper are further summarized below.
Joint network architecture and conditioning search We extend the DNAS formalism to learn layer-wise weight sparsity levels. We present a method for searching for sparsity levels in conjunction with layer bitwidths. We show how to maximize weight sharing while jointly searching over sparsity, bitwidth, and layer width, as well as provide a differentiable and easily computable measure of compressibility for the DNAS algorithm to optimize.
Novel search algorithm Previous work (e.g. [22, 21)) fails to effectively trade-off accuracy with model size (P2) in our search space (Table 3, first row). UDC addresses this, with the following improvements: 1) guarantees that the search yields a model which satisfies specified (Sec. 4.1), and 3)
HW constraints (Sec. 4), provides control over exploration-exploita. avoids over-rogulaiizttion from biased Gumbel-softmax approximation (Soc. 4, 4.2).
Novel sparse, low-bitwidth representation and training algorithm UDC yields com- pressible models that are difficult to train. We identify the root cause of the problem and propose solution using novel weight representation (Sec. 5).
SOTA NPU-deployable TinyML models Using UDC, we demonstrate compressible NNs Pareto dominant over prior work on CIFAR100, ImageNet, and super resolution (SR) tasks (Fig. 4) [41, 57, 10, 37]. Compression gains are validated by deploying to Ethos- U55 NPU,
2

---

2
Related work
Table 1 contrasts UDC with relevant pre- vious work. MCUnet [48] is the closest to UDC, as it targets the ML HW form-factor with severely limited mem- ory and includes results on large scale datasets like TetgeNet. However, UDC is fundamentally different to MCUNet: 1) unlike MCUnet, UDC also performs per- layer bitwidths and sparsity rate search to exploit HW compression, 2) UDC DNAS algorithm whereas MCUnet uses cvolutionary search of a pre-trained once-for-all (OPA) supernet [17]. In par- ticular, the UDC search space largels compressible models and much more diverse than that of MCUNet; training single OFA supernet for our search space would be infeasible (Sec. 6.2).
Table 1: UDC features related work.
Ublich et al. [o1]
MCUnet (ay]
Yang et al, [M]]
Gong on al. 277
Chai [is
FBNe (n2)
Unc (Ours)
APQ 64
HAQ
Width:/Opman Depth Biteidth/Sppartt HW Constraint Guaraaee Deployable w/ w/lntoger Math TinyLL (< 1.25BB)
Yang et al. [66] also demonstrate learning of per-layer bitwidth and sparsity rate, but otherwise differ significantly in that they: 1) do not include storage of the (required) pruning mask in the reported model size, which can even dominate at high sparsity rates [16]: 2) only consider model conditioning, without accompanying NN width, depth, or operator search; 3) employ floatingppoin point non-uniform quantization, not deployable on TinyML MCU and NPU platforms which only support inleger operations.
APQ proposes a multi-stage algorithm to search over layer width and bitwidth [64]. Compared to APQ, UDC: 1) optimizes over sparsity rates whereas APQ does not, and 2) produces models 3 9x smaller than APQ. Other relevant works include Gong et al. [27], which searches over width and bitwidth but not sparsity, Choi et al. [18], which searches over bitwidth and sparsity but not layer widths and without addressing (P2), and Uhlich et al. [61] which scarches over only bitwidths.
3 Modelling design decisions
We use bold-face to denote pectors/tensors and z[k] to mean the k'th element of vector For a layer with input and parameters 0, denote its output by f (x, 0). We say K follows a categorical distribution parameterized by ग, i.e. z ~~ Cat (T), if p(=[k] = 1) T [kk].
Width selection Layer widths are modelled fax 0) @ w, ||l/ /|wl p, where is binary mask to toggle any given channel, p is the fraction of non-zero channels, @ is element- wise multiplication, |/w]]] ln is the En pseudo-norm which counts the nunber of non-zeros in w, and |w]] is the number of elements in w. We adopt the convention of setting the first fraction of the channels of w [62). Different choices (p of layer width can be modelled by the random variable (RV)
K
(T))))))
Zm Wk Zn ~ Cal (Tm)), | loello//ee /lunal = Pk.
(1)
We refer to Zw as decision variable and to ps as an option.
Sparsity Sparse weight tensors are expressed as m, |m /mml s where m is binary mask and is the fraction of non-zcros. We set the non-zeros of m to correspond to the largest magnitude elements of 0 [50, 31, 28). We model different choices of amongst {si.".., [ with the RV
K.
m (TT) =
Σ
2g [k] [k)m,, 2 ~ Cat (TTx), l|mmlo //mml = Sk-
(2)
k=
3

---

Quantization Uniformly quantized tensors are given by
b> 1 Q(0,b,r) d round (dlp((a r> 0, d d
(3)
sign(() (0)
b=1
where r is the quantization range and b the bitwidth. We model different choices of b E {b,.... _bK, as the RV q (Tg) Ekn 1 ky (0, 1k), Cat (Tg)), where each bitwidth by is parameterized by its own range Tk.
Sparsity and quantization Sparse, quantized tensors modeled Q(0,b, b, r) find setting the non-zero values of m based on 0 superior to using Q(0, b, r), which discards in- formation about the relative magnitude of weights during the quantization process. To model lensors whose spansity level and bitwidth must be chosen, we define the RV g(TT,) o m(TT).
We
Operator selection We frame the choice over different layer operators as the RV Eks 2 fi (2x 0x), 2f Cat (Tς), where each operator has (possibly) different func- tional form f-(-) parameterized by its own weights 4g. By including identity as a can- didate operation, can also model varying NN depth.
With
Irpur
Figure 2: UDC search space overview.
Fig. 2 shows the layer-level search space, excluding operator selection for brevity. Weight sharing is maximized by using the same in the sparsity, bitwidth, and width decisions. The output of the layer in Fig. is modelled the RV f (æx ((T,)) O m(T.)) + w(T.).
3.1 Computing layer storage size
Satisfying (P1)-(P2) necessitates quantitative measure of storage size, accounting for data compression. For a given layer, the storage size achievable by prefix- free compression is by the weight (empirical) entropy, H+), times the number of weight elements [19). The entropy bound can, in turn, be bounded by (Appendix C):
(4) The advantage of the right hand side (RHS) of (4) over the left hand side (LHS) is that it can be computed cheaply without processing 0. Using the LHS requires computing the empirical entropy, which expensive for large
H (Q(0, r) @ m) ||w/|o I (b (6++ (-8 log, 8 - (1 - 8) log, s))) |ollo-
Morcover, combining the LHS with gradient-based optimizer requires gradient approximation, since the empirical entropy is not difforentiable [60]. We refer to the RHS of (4) as the compressod tensor size and use as our measure of layer size. Sec. 6.2 confirms that it is achievable with both arithmetic encoder and Vela. Let e(s,b, p) be given layer's storage size (the RHS of (4)) as a function of sparsity 8, bitwidth b, and non-zero channel fraction p. When (s, b, p) must be chosen, let
[K K2 [k[ks Σk zy zy[k]bb, Σk k= zw. [k]pp) be the storage size. For the entire NN, we sum the storage size of all layers and denote the result (fz)), where {z} short-hand for the set of all decision variables.
Proposed DNAS algorithm
Our optimization objective is:
arg min E(x), [L (fz, 0) D)] s.t. & (f+(T))) = =², ((T) = onchot arr max T(k] (m.)m..))
(5)
where I(.) is a task loss, e.g. cross-entropy, D the training data, and e* is the target model size. The constraint in (5) stems from the two-stage process typical in DNAS: 1) Optimize {T, 0} and extract the most likely configuration {T(T)), 2) Train and deploy the result [51, 21, 22, 40, 62, 65). While prior works use the constraint [21, 22, 40, 62, 27]
E(x) [8 (fz)) =e"
(6)

---

we constrain the most likely configuration since this is what is actually deployed. Even if (6) satisfied, it is not guaranteed that {y(T)), the deployed model, satisfies the constraint.
Gradient-based optimization Solving (5) using stochastic gradient descent (SGD) requires:
1) ensuring that the constraint is met, 2) differentiating with respect to {0, )). To deal with the constraint and avoid dealing with the non-differentiable 7(T), we modify (5) to
arg min E(=),D (1(, +A (E) [| ({z}) - = (n,0,
mig
(7)
Cusk
While L has different form than the constraint in (5), it actually represents much stronger constraint and its minimization implies the constraint is met.
Lemma 4.1. If - O, then for any sample of {z}, denoted {z*}, with non-zero probability, E ((2*)) = and & ((7(T)))
e" e""
43 differs from (6) because it penalizes any configuration which violates the constraint, whereas (6) penalizes { only if the expected storage size violates the constraint.
The derivative of (7) with respect to (w.r.t.) {0} can be approximated using a Monte- Carlo (MC) approximation of the expectation and applying standard automatic differentiation. The derivative w.r.t. {τ) } is more complex since the expectations in (7) depend on {π]. A popular solution is to use biased but differentiable approximation of z, given by the Gumbel--oftmax
distribution: 2 = softmax log log [[[]] Gumbel (0, 1), > 0 [39, 21, 22). As + 0, &
approaches in distrriutiin,, while the variance of any gradient estimator which uses increases ((55) and Fig. 5d). As a result, the common practice is to anneal from a high to a low value throughout the search. Using& can lead to co-adaptation of search space options, which is undesirable but solvable in practice (Appendix D).
and regularization We observe two issues with anncaling т: 1) increased gradient variance at low coupled with a complex search space, causes issues for SGD, 2) when we replace with & in Ce, with the result C the regularizer becomes artificially inflated. To understand the impact of on CJ we evaluate C for different values of r, setting π such that & (fT(T))) = =". The results are presented in Table 2, col. and show that increasing increases Lg, i.e. the relative impact of E on (7) depends on T. As such, we seek to keep t low, while minimizing gradient variance. Our solution is to multiple samples of { in the MC approximation of (7). To maintain the same computational cost as the single MC sample case, we divide the number of optimization steps by the number of samples. As well as reducing gradient variance, our strategy has two additional practical benefits: 1) trivial extension to multi-GPU systems, since each GPU can run its computing the gradient are amortized across the MC samples. We observe a 5.3x speed-up when going from MC sample to 32 (Table 3, col. 6, row vs. row 8.)
Gumbel-softmax
over-
Table 2: Values of L for different values of T, E(see (8)), (Sec. 4.2). The search space is based on MB (Sec. 6) and & ({)(T))) e* in all cases.
Vanilla
Projection Rejection Sampling DNAS { (=0.5,
Projection
0.S,
MC sample and gradient computation, 2) the overheads of
4.1 Exploration--xploitaaii
When solving (7), the goal is to explore as many configurations as possible (exploration), while still training each configuration for meaningful number of steps (exploitation). Ideally, the search algorithm should gradually move from exploration to exploitation. We propose to explicitly control the exploration-exploitatii trade-off by projecting onto the set
{ π max x[k] < 1 55
S S= s
(8)
#
after each SGD step, where g is the upperbound on π at step Setting gt con- strains to parameterize uniform distribution and represents maximal exploration.
5

---

Setting 5 = 1 1//T| removes the constraint on allowing the optimizer to enter full exploitation. We define the projection operator as Ps( = softmax (log π "T",, T = magg Σ |1 max (T (1// + 5"), 0), which we solve numerically. We choose this form for (T( because of its simplicity and because the relative ordering of options between π and (T) does not change. While we find it necessary to enforce exploration by projecting π onto S, f implicitly promoles exploitation.
Lemma 4.2. Let C = 0 and zj be the lll decision variable. Let there be no decision for which two of its options have the same cost, i.e. for two configurations {2"} and { xx") such
that 23 X zj and z vK vk + j, we have C + Le Then each Ty must be one-hot.
The assumption in Lemma 4.2 that no decision has two options with the same storage cost is satisfied for a typical compressible model search space.
4.2 Combating over-regularization with rejection sampling
Projecting π onto S enables explicit control over the exploration-explliaation dynamics, but also inflates f.. Table 2, col. 2 shows that setting 5 = 0.5 increase L3 dramatically. By forcing π to be closer to uniform, the number of configurations {2} with non-zero probability increases, such that the probability of a randomly drawn configuration violating the constraint also increases. Inflating C3 forces the optimizer to focus less on Lraskr leading to solutions which meet the constraint but perform poorly on the target task. C increases when the properties of {т change because it depends on all possible configurations {2) instead of the most likely one, i.e. {. To motivate the remedy, observe that while not all samples of 2, denoted é*, satisfy 7(8*) = ? some do. Indeed, C3 would be it was evaluated over those samples that satisfy 7(2)) = =()), assuming & ((T(T))) = e* (and к 1 in Appendix D (12)). We refer to samples generated in this manner as ø* and they correspond to a RV whose distribution is different from 2, but still depends on π and can therefore be used to generate gradients to æ from C. Alg. shows how to generate z². Replacing & with ž for all decisions negates the effects of controlling π through projection onto S. Therefore, we use & for a given decision with probability & and & otherwise.
Algorithm 1 Rejection Sampling
Algorithm 2 Complete UDC Algorithm
1: Sample 2º p()), 1 15 S
1: for S S mmx do
2: 8= 1, k- = -arg max, [k]
2: for 1 < 8 S S do
3: for $ do
3:
Generate MC sample s for deci-
4: ==argman max, [*[
sion j using 2, w.p. & and ž₁ else
5: if k= 1.' then
4: Update {m} } if modulo 16 = 0
6:
=== 2,, 3+1
5: Take SGD step on (7)
7:
1 == 5-1
Average
6:
π + Ps(TT)
> Projection
8: return 2
7:
return {T]
Using E has two major benefits. First, even when exploration is enforced by projecting ा onto S, C is not inflated. Table 2, col.s 3-4 show how 0 > 0 brings down C3 for the same underlying (T). Second, mixing 2 and ž gives the flexibility of being in exploration for some decisions and exploitation for others. By randomly choosing which decisions use & and which use &, we prevent greedy behavior whereby a given decision enters exploitation and never returns to exploration. The complete UDC algorithm is summarized in Alg. 2.
5 Training sparse, quantized models
DNAS is typically a two-stage process: a search to find the model architocture is followed by finetuning to find optimal weights. However, we find that the second stage yields poor results when training sparse, quantized models from scratch [71]. UDC employs a three- stage fineluning process: Stage 1: initialize (0) and train with quantization enabled but unstructured pruning disabled, Stage 2: enable unstructured pruning gradually, Stage 3:
train with both quantization and unstructured pruning enabled. To counter the training

---

challenges induced by quantizing and pruning 0 we employ several known techniques, with one slight modification. For quantization, we use variant of [59] where each weight is quantized with probability a during the forward pass, such that the weights used for training are Q(0, b, r) o + clip (0, (0, 3 (1 h), hij,, J. Gin, Cout ~ Bernoulli (a). Unlike UDC, [59] uses 0 in the second term. We observe that the range of 0 can differ significantly from that of Q(0, r), especially when unstructured pruning is applied during the learning process, so we clip 0 to the same range as Q(0, b, r). For unstructured pruning, pruning rate from 0% to 100% of the target rate during stage 2 [72, 50].
gradually anneal the
Weight numerical representation To understand the challenge of training sparse, quan- tized NNs, consider a single layer of pretrained weights, with histogram in Fig. 3a. The quantization bins (b = 4) are shown in red and the pruning boundary in purple, with everything between the purple lines mapped to O. The sparse quantization problem clearly apparent here: 8/14 of the non-zero quantization bins are unused because they fall inside the pruning boundary. If we now train with weight sparsity and quantization constraints, the weight distribution, quantization bins, and pruning zone adjust (Fig. 3b). The optimizer flattens the weight distribution to use more quantization bins, resulting in increased weight range. To quantify the weight growth, we report the norm of the NN weights before and after training in Fig. 3, showing an increase of over 4.6x. The rate at which NNs can be trained, known the effective learning rate, is inversely proportional to the weight norm [12]. Therefore, the interaction of sparsity and quantization cause weight norm inflation, which decreases the effective learning rate, reducing NN performance. We propose different weight representation and only quantize the range beyond the pruning boundary (Fig. 3c), using the quantization operator Q b, 5)= = Q(e - sign (0) b, r) + sig for b > 1, E IR We set B be the largest pruned value of O. Training the sparse, quantized NN with the proposed number representation leads to much smaller weight norm (only 2.2x growth over the pretrained weights), which makes training easier and accuracy higher (Fig. 5c).
Deployment with integer math NPU HW platforms typically only support integer operations, which are cheaper than floating point. There are at least two ways of deploying NNs quantized using the proposed approach on such HW. Firstly, convolution can be decomposed into fxx Q(0 (0 r)) + ff (O)). Both terms can be calculated using only integers, but since sign(() is a 1-bit temsor, the second term does not require any multiplications. Secondly, the NN can be trained using the proposed number format, then deployed with @ (Q, ,r):, such that the deployed NN is uniformly quantized,
with 6- = 8 to match the datatype supported in MCUs/NPUs [7, 8, 2]. The advantage of the latter approach
that the training benefits from the expressivity of the proposed number format, while its deployment uses standard data type. We find that this approach with 6" = 8 does not incur an accuracy loss on ImageNet (Fig. 4a).
6 Results
We compare UDC with SOTA mothods on model size vs. accuracy basis. All reported model sizes, including related works, use compressed size (RHS of (4)), except Choi et al. [18]. who use bzip2 to compress weights and we use their reported sizes.
CIFAR100 is an image classification task with 50k training / 10k Lest images, and 100 classes. Our search space is based on the wide residual network with depth 20 and width multiplier 10 (70, 52). We search over layer width (increments of 10% of the orignal), bitwidth (1, 4, 8, 32), and sparsity (1- 100% non-zeros, increments of 10%). We use UDC settings in Appendix E) to find and train models at two sizes: 0.55MB and 0.7MB. Fig. 4b shows that UDC generates Pareto-dominant models compared with SOTA methods [27, 52, 65, 63, 58].
ImageNet is an image classification task with 1.28M training / 50k test images, and 1k classes. Our scarch space is based on M2,ett2, with the same options as for CIFAR100, and we target 0.5, 1, and 1.25 MB models (Appendix F). For the 1. 1.25MB experiment, we replace 3x3 kernels with 5x5 and make all layers 1.5x wider in the baseline architecture.
In practice, the model size constraints are determined by the Flash memory size of the deployment HW platform. To be sure, 0.5 1.25 MB Flash sizes are fairly common for

---

30
30
30
Wrights Prune boundary
Weights
Weights
Hizogram count
Quant. bins
Hizogran couns
Quant:
20
20
Prune boundary
20
Prune boundary
25
Histegran
15-
15-
10
20-
Weight
Weight
(a) Baseline, retraining
(b) Baseline, retrained
(c) Proposed, retrained
Figure 3: Histogram of weights from a single pruned and quantized layer in the ImageNet experiment under different conditions. Weight norm ||0|| (3a) 1.7c3, (3b) 7.9c3, (3c) 3e3.
0.50
-207X 1.550%
1J8K
70
1.29%
20.79
233x
ka
0.78
66
1000
APS
0.77
USC Gre
-433K
103.7
1.35a.1
10.3
0.76
0.4
0.8
1.2
1.5
16
Mode sizs (MB)
Model Now (M8)
(a) ImageNet classification
(b) CIFAR100 classification
(c) set super resolution
Figure 4: UDC finds pareto-dominant models on image tasks.
commodity HW platforms [6] and are often used in research targeting deployment on constrained HW platforms [48, 14]. As such, we targeted this range because represents reasonable, but extremely challenging deployment scenario.
Fig. 4a shows that UDC generates pareto-dominant NNs the SOTA [48, 18, 27, 49, 65, 62, 64). Next, we inchude depth/operators in the search space, considering two alternatives to the inverted bottleneck: 1) regular 3x3 convolution, 2) identity. We target 0.5MB model. UDC chooses inverted bottleneck blocks everywhere, such that the model found by including depth/ope /operators in the search is identical to the one found when they are excluded. We compare UDC with non-uniform quantization approaches (Appendix H), and find that UDC models are pareto-dominan even though approaches like HAQ [63] and [66] employ more expressive, non-unifon con-uriform quantization which cannot be deployed on commodity MCUs and NPUs with integer math operations. Finally, comparing UDC to SOTA unstructured pruning algorithm (Appendix I) shows that UDC finds considerably more accurate models.
SR The purpose of the SR experiment is to: 1) show that UDC be applied to regression problems, 2) demonstrate UDC in a setup which constrains the computational complexity (MAC) operations, instead of
of the NÑ, measured in number of multiply-and-uccumut model size. Note that 1 MAC = floating point operations (FLOPs), but authors often conflate the two terms. This experiment excludes bitwidth and sparsity from the search space, which do not impact the inference compute cost. The topology of our search space inspired by FSRCNN [20], with search over depth, width, and kernel size. We do all search/traing on div2k and report results on s4t (see Appendix G for results on div2k and sct5). We target 4x upscaling and report MACs for an input patch of 64x64. We train all NNs using the same settings. Fig. 4c shows that UDC finds Pareto-dominan NNs compared with SOTA efficient SR methods SESR, FSRCNN, and TPSR. [15, 20, 44].
6.1 Comparison with random search
Random search is a strong baseline that can compete with DNAS in some settings [46]. We are not able add early stopping [46], since sparsity is only applied during stages 2-3, when it is too late to save time from stopping. Fig. 5a compares UDC to randomly generated
8

---

$2.9
50.00
70
Durs Unitorm
camples
10
70.00
20.0
UDC
$5.000
10
condom
0.0
Model
30.00
kumber
50
150
07
32
0.2
1.0
4.0
(a)
(b)
(c)
(d)
Figure 5: 5a-5b: Models found by random search. 5c: Benefit novel number format on Tet.eNet. 5d: Maximum gradient variance over elements of {अ) for CIFAR100 experiment.
Table 3: Ablation UDC components for experiment targeting 0.5MB models. *-Constraint not met, so accuracy not reported. f-GPUD not reported as hyperparameter search for À is difficult to account for. For ImageNet, GPUT is normalized to the typical training cost of MB (460 images / /) [4)) to allow for comparison to other works. Dashes indicate the
experiment is prohibitively expensive to run because it requires hyperparameter search for A.
Rejection
Multiple
No
CIFAR100
ImageNet
Sampling
MC Samples Projection Tuning
Acc. (%)
GPUD Acc. (%) GPUD
Vanilla DNAS
Partial UDC
3.2
Partial UDC
79.14
1.17
3.2
Partial UDC
79.06
6.21
Partial UDC
0.83
3.2
Partial UDC
76.48
Partial UDC
Full UDC
79.24
1.17
64.09
3.2
NNs constrained to model sizes 0.55MB for CIFAR100, with the search space used in Fig. 4b. The gap between the best UDC models and random search is 4.77%. The best accuracy achieved by random search after given number of trials is shown in Fig 5b. To integrate out randomness from the the ordering of trials, we permute the order of the random scarch results 100x, averaging over the permutations. Since UDC search requires ~2x longer find NN than training baseline NN, plot the cost of UDC 3 trials (2 to search and to finetune). These results confirm that UDC finds better NNs faster than random search.
DNAS alternatives like evolutionary search require training many NNs or proxies to relax the compute burden [24, 23, 17, 48, 49). However, using proxies assumes a search space dense with performant NNs, which is not true in our case (Fig. 5a).
6.2 Validation of algorithm and system components
Compression We verified our the
model size approximation, RHS of (4) with pruming mask com- pressed to the entropy limit, by compressing m using arithmetic coding [53] for all UDC ImageNet models (Fig. 4a). We observed that theoretical and practical com- pressed model sizes within two decimal places (Table 4). Table 4 shows that Vela can achieve com-
Table Compressed model size (MB). Vela size repre- sents Flash usage when deployed to NPU. MBNet listed for reference.
Original
ItHS of
IHHS of Compressed
Vela (Ratio)
Model
2.32
0.19
0.19
0.44 (5.27x)
Model
2.22
0.9
0.9
0.83 (2.67x)
Model
(4.55x)
MBNetV2
3.31
3.31
3.31
2.87 (1.15x)
pression ratios even larger than those predicted by UDC for NPU deployment, which stems from the fact that non-zero weights can often be compressod to less than bits per weight by Golomb-Rot Rice compression.
A Ablations Table gives an ablation UDC components (Sec. 4) on CIFARI00 and ImageNet, which we evaluate on final accuracy, hyperparameler search over requirement, and runtime in GPU days (GPUD) normalized to typical training setup. Table 3 shows that disabling

---

Table 5: Algorithm runtime comparison on ImageNet. System is defined as software implementation and HW platform. Algorithms which begin with a pretrained model are assumed to pretrain the model for 200 epochs [4) and are marked by f.
Images /
Search epochs
GPUD (systom and alg. specific]
GPUD pormalized to 400 im. / (algg specifie)
(systom & alg. speciifie)
(elg. specifiie
UBC
230
100
FBNet (S1
16.8 (11 Sed 216 hour)
(on 1/10 ImageNett clase))
0.3
FBNNe
14.8 (11.Ge6
216 hous)
1/10 IazggeScct classs))
MCUNEt (18)
450
MCUNN
450
Choi [18
Uh et [61]
2507
any of the UDC components leads to significantly worse results on both datasets. Vanilla DÑAS [21] fails to meet the HW constraint. Fig. 5c shows the isolated benefit of the proposed number format over the baseline (3).
Runtime An important aspect in evaluating NAS algorithms is their runtime, or the time required to yield results. Part of the challenge in comparing algorithm runtimes is that the runtime generally depends on: 1) software implementation quality and HW platform, which jointly form the "system" that the algorithm runs on, 2) the algorithm itself, i.e. what the algorithm is actually doing during the search, 3) the number of epochs (or amount of data processed) for which the soarch is run. The challenge is that only 2-3 are algorithm dependent, whereas 1) depends on the code quality and the resources of the experimenter (i.e. higher grade GPUs exhibit higher throughput compared to low-end GPUs).
In order to compare with other works, we now attempt to disentangle the components that make up the runtime in Table 5. We list the system and algorithm specific search speed measured in images per second, when available from the reference, the number of search epochs, the search cost in GPUD under the system used in the reference, and the search cost. GPUD under a common system assumed to be running at 460 images / s [4]. For several references, the algorithm begins with pretrained model, which we assume is trained for a standard 200 epochs [4]. Table shows that in absolute terms (i.c. the system specific cost), UDC is 1.4x faster than FBNet and FBNetV2. When comparing approaches based on a normalized system running at 460 images / s, UDC is faster than all of the competing approaches other than FBNet and FBNet Our hope is that Table 5 gives rough sense of the relative search cost of UDC and the competing methods.
Societal impact & limitations Developing UDC used hundreds of mnergy-consume GPU hours. However, this can be amortized by increasing the energy efficiency of billions of ToT devices. A limitation of UDC is that we retrain (0} for every constraint e", whereas approaches based on OFA can amortize the cost of training (0} across multiple constraints.
Conclusion
Emerging NPU HW platforms specialized for TinyML support model compression, whereby quantized and pruned NNs can be stored in reduced memory footprint. While compression is highly desirable, it increases the complexity of the NN design process, as the space of candidate NNs increasod by adding quantization and pruning on top of the conventional NN architecture choices. To enable TinyML practitioners to fully exploit HW model compression in NPUs, we describe a unified DÑAS framework to search both architecture choices and aggressive per-layer quantization and pruning. We describe number of improvements on top of DNAS, allowing us to demonstrate SÔTA Tiny models that fully exploit model compression, as well as comparison with random sampling and extensive ablations.
10

---

References
[1] Alif Semiconductor: Introducing the Ensemble and Crescendo families of fusion proces- sors and microcontrollers. https://alifaemi.coon wttp://mmmmoooooooccctcseecsc//e Accessed: 2019-05-02.
2 Arm Ethos- U55 Micro Neural Processing Unit (uNPU). https://www arm. com/
Accessed: 2021-10-26.
3 Arm Vela Tool for Ethos-U Micro Neural Processing Unit (uNPU). https://pypi.org
project/ethos-u-velll u- Accessed: 2021-10-26.
[]] Reproduction of MobileNetV2 using MXNet
https://github.. www.tttuuu...mmmmaanni.ti
mxnet-mobine monet-mobilene-v22 Accessed: 2022-10-2.
55] Practical application of tinyML in battery powered anomaly sensors for predictive maintenance of industrial assets. https://cms. tinyml.org/up Taaks2200/tiiyMM__allsss.s Talkss
org/vp- cowpep--ootnnttpllaaaaeee
2000a1200011_.11 pdf. Accessod: 2021-10-09.
6 STM32 Hardware Specification, Wikipedia. https://en http:s///ee.wikkieaaa..oooo http://www.........////////// Accessed: 2019-05-02.
TensorFlow Lite for Microcontrollers,
Accessed: 2019-05-02.
u'Tensor. http://utensor.. ai/. Accessed: 2019-05-02.
99] Visual docs http:///c/ccccaa////////////aaad _45xjclmr /rg.aa/aaa.aa.....aaaaaaaaaaaaon cessed: 2019-03-02.
Wake
Words
Challenge, CVPR 2019.
URL https://
Ac-
[10] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super- resolution: Dataset and study, In IEEE/CVF CVPR workshops, pages 126 135, 2017. 126-135,
[11] Hazoor Ahmad, Tabasher Arif, Muhammad Abdullah Hanif, Rehan Hafiz, and Muham- mad Shafique. Superslash: A unified design space exploration and model compression methodology for design of deep learning accelerators with reduced off-chip memory access volume. IEEE Transactions on Computer-Aiie Aíded Design of Integrated Circuits and Systems, 39(11):4191 1491-444,, 2020.
[12] Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical analysis of auto rate-tuning by batch normalization. In International Conference on Learning Representations, 2019.
URL https://openreviier hetp://eeetttooornnd/d--------
[13] Colby Banbury, Vijay Janapa Reddi, Peter Torelli, Nat Jeffries, Csaba Kiraly, Jeremy Holleman, Pietro Montino, David Kanter, Pete Warden, Danilo Pau, Urmish Thakker, antonio torrini, jay cordaro, Giuseppe Di Guglielmo, Javier Duarte, Honson Tran, Nhan Tran, niu wenxu, and xu xuesong. MLPerf tiny benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmorks Track (Round 1), 2021. URL https://openreviert htt.:///ww...ooomiiaaaaocatttt
[14] Colby Banbury, Chuteng Zhou, Igor Fedorov, Ramon Matas, Urmish Thakker, Dibakar Gope, Vijay Janapa Reddi, Matthew Mattina, and Paul Whatmough. MicroNets: Neural network architectures for deploying tinyml applications on commodity microcontrollors. Proceedings of Machine Learning and Systems, 3, 2021.
[15] Kartikeya Bhardwaj, Liam O'Neil, Dibakar Gope, Ramon Matas, Alex Chalfin, Naveen Suda, Lingchuan Meng, Danny Loh, and Milos Milosavljevic. Collapsible linear blocks for super-efficient super resolution. Proceedings of Machine Learning and Systems, 4, 2022.
11

---

[16] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag, What is the state of neural network pruming? In I. Dhillon, D. Papailiopou- los, and V. Sze, editors, Proceedings of Machine Learning and Systems, volume 2, pages 129-146, 2020. URL https://proceeddngg .alpss..ssss........m wwlsy..rr//pp........oo... arg/paperr/220//f1ee 22dde118006666c622e (eteeee3dddee337555P5555- Paper. Pwt.pp--......... pdf.
[17] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for- all: Train one network and specialize it for efficient deployment. Tn International Conference on Learning Representations, 2020. URL //openrviieee net/forum? id=HyIxEIHKsS.
[18] Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Learning sparse low-precision neural networks with learnable regularization. IEEE Access, 2020.
[19] Thomas M Cover. Elements of informalion theory. John Wiley & Sons, 1999.
[20] Chao Dong, Chen Change Loy, and Xiaoou Tang, Accelerating the super-resolution convolutional neural network. In European conference on computer vision, pages 391 407. Springer, 2016.
[21] Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. In Advances in Newral Information Processing Systems, pages 759 770, 2019.
[22] Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In IEEE/CVE CVPR, pages 1761 1770, 2019.
[23] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. In International Conference on Learning Representations, 2019.
[24] Thomas Flsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-211 2019.
[25] Igor Fedorov, Ryan P Adams, Matthew Mattina, and Paul N Whatmough. SpArSe: Sparse architecture search for CNNs on cnoource-oonstraneed microcontrollers. Advances in Neural Information Processing Systems, 32, 2019.
[26] Igor Fedorov, Marko Stamenovie, Carl Jensen, Li-Chia Yang, Ari Mandell, Yiming Gan, Matthew Mattina, and Paul N Whatmough. TinyLSTMs: Efficient Neural Speech Enhancement for Hearing Aids. INTERSPEECH, 2020.
[27] Chengyue Gong, Zixuan Jiang, Dilin Wang, Yibo Lin, Qiang Liu, and David Z Pan. Mixed precision neural architecture search for energy efficient deep learning. In 2019 BEEE/ACM International Conference on Computer-Aidee Aided Design (ICCAD), pages 1-7. IEEE, 2019.
[28] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 1387-1395, 2016.
[29] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European Conference on Computer Vision, pages 544 560. Springer, 2020.
[30] Chirag Gupta, Arun Sai Suggala, Ankit Goyal, Harsha Vardhan Simhadri, Bhargavi Paranjape, Ashish Kumar, Saurabh Goyal, Raghavendra Udupa, Manik Varma, and Prateek Jain. Protonn: Compressed and accurate knn for resource-scarce devices. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pages 1331-1340. JMLR. org, 2017.
[31] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems, 28, 2015.
12

---

[32] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, International Conference on Learning Representations (TCLR), 2016.
[33] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Compuler Vision, pages 1389-1397, 2017.
[34] Weijun Hong, Guilin Li, Weinan Zhang, Ruiming Tang, Yunhe Wang, Zhenguo Li, and Yong Yu. Dropnas: Grouped operation dropout for differentiable architecture search. In IJCA,, pages 2326-2332 2020.
(35] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convo- lutional neural networks for mobile vision applications. CoRR, 008/1704.04861, 2017.
URL http://arxiv. http//axxv...oo/aaa// /asp/bbb//1004....///
(36) Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin. Dsnas: Direct neural architecture search without parameter retraining. In IEEE/CVF CVPR, pages 12084 12092, 2020.
[37] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-oxemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5197 5206, 2015.
[38] Yerlan Idelbayev and Miguel A Carreira-Perpinán. Low-rank compression of neural nets: Learning the rank of each layer. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pages 8049 8059, 2020.
[39] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparamcterization with gumbel- softmax. International Conference on Learning Representations, 2017.
[40] Xiaojic Jin, Jiang Wang, Joshua Slocum, Ming-Hsuan Yang, Shengyang Dai, Shuicheng Yan, and Jiashi Feng. Re-darts: Resource constrained differentiable architecture search. arviv preprint ar Xic::19912 12814, 2019.
[41] Alex Krizhevsky, Gooffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[42] Ashish Kumar, Saurabh Goyal, and Manik Varma. Resourco-officient machine learning in 2 kb ram for the internet of things. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1935 1944. JMLR. org, 2017.
[43] Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Pratock Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference Machine Leurning, pages 5544 5555. PMLR, 2020.
(44) Royson Lee, Lukasz Dudziak, Mohamed Abdelfattah, Stylianos I Venieris, Hyeji Kim, Hougkai Wen, and Nicholas D Lane. Journey towards tiny perceptual super-resolution. In European Conference on Compuler Vision, pages 85-102. 102. Springer, 2020.
(45) Haitong Li, Mudit Bhargav, Paul N Whatmough, and H-S Philip Wong. On-chip memory technology design space explorations for mobile deep neural network accelerators. In 2019 56th ACM/IEEE design automation conference (DAC), pages 1-6. IEEE, 2019.
[46] Liam Li and Ameet. Talwalkar. Random search and reproducibility for neural architecture search. Tn Uncertainty in artificial intelligence, pages 367 377. PMLRR 2020.
[47] Edgar Liberis, Lukasz Dudziak, and Nicholas D Lane. pmas: Constrained neural architecture search for microcontrollers. Tn Proceedings of the 1st Workshop on Machine Learning and Systems, pages 70-79, 2021.
13

---

[48] Ji Lin, Wei-Ming Chen, Yujun Lin, Chuang Gan, Song Han, et al. Mcunet Tiny deep learning on devices. Advances in Neural Information Processing Systems, 33: 11711-11722, 2020.
[49] Ji Lin, Wei-Ming Chen, Han Cai, Chuang Gan, and Song Han. Mcunetv2: Memory- efficient patch-based inference for tiny deep learning. arXiv preprinl arxi2:2100. 12111011133 15352, 2021.
[50] Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference Leurning Regresenlations, 2020. URL
httpe/viiwwieeeffnnnneeeeeeeeeee rett/forum?dddSJemBBBB5.
[51] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019.
[52] Mark D. McDonnell. Training wide residual networks for deployment using a single bit for each weight. In International Conference on Learning Representations, 2018. URL ottps://openreett
net//forumiddrryMMMIAAA
[53] Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool. Practical full resolution learned lossless image compression. In IEEE/CVP CVPR, 2019.
[54] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Prun- ing convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.064400 2016.
[55] Max Paulus, Chris J Maddison, and Andreas Krause. Rao-blackwellizing the straight- through gumbel-softmax gradient estimator. arXiv preprint Xrx:::2010.044 048388 2020.
56) Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient, neural architecture search via parameters sharing. In International Conference on Machine Learning, pages 4095 4104. PMLR, 2018
[57] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Jour- nal of Computer Vision (IJCV), 115(3):211-252, 2015. doi:
[58] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In IEEE/CVF CVPR, pages 4510-4520, 2018.
[59] Pierre Stock, Angela Fan, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. Training with quantization noise for extreme model compression. In International Conference on Learning Representations, 2021. URL https://openrevier net http://paareeeeeeeeeeeerrara//ae
[60] Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. arXiv preprint arxio 700.003955 2017.
[61] Stefan Uhlich, Lukas Mauch, Fabien Cardinaux, Kazuki Yoshiyama, Javier Alonso Garcia, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Mixed precision dnns: All you need is good parametrization. Tn International Conference on Learning Representattons, 2020. URL /
[62] Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. Tn SEEE/CVE CVPR, pages 12965 12974, 2020.
633 Kuan Wang, Zhijian Lüu, Yujun Lin, Ji Lin, and Song Han. HAQ: hard warewware automated quantization. In IEEE/CVP CVPR, 2019.
14

---

[64] Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang, Yujun Lin, and Song Han. Apq: Joint search for network architecture, pruning and quantization policy. In IEEE/CVE CVPR, June 2020.
[65] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Kentzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In IEEE/CVF CVPR, pages 10734-10742, 2019.
[66] Haichuan Yang, Shupeng Gui, Yuhao Zhu, and Ji Liu. Automatic neural network compression by sparsily-quantizatie joint learning: A constrained oplinizalion-based approach. In IEEE/CVF CVPR, pages 2178-2188 2020.
[67] Ticn-Ju Yang, Yu-Hsin Chen, and Vivienne Szc. Designing energy-efficient convolutional neural networks using energy-aware pruning. In IEEE/CVE CVPR, pages 5687 5695, 2017.
[68] Jianbo Ye, Xin Lu, Zhe Lin, and James Z. Wang. Rethinking the smaller-norm- cossinformmttie assumption in channel pruning of convolution layers. In International Conference Learning Representations, 2018. URL https://openreviae wwt.:/eee/..rrr. id=HJ94fqAph.
[69] Haibao Yu, Qi Han, Jianbo Li, Jianping Shi, Guangliang Cheng, and Bin Fan. Search what you want: Barrier panclty nas for mixed procision quantization. arXiv preprint ar XXi::2007.10020 2020.
[70] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 87.1 87.12. BMVA Press, September 2016. ISBN URL https: //dx dwi. 5244/C.30.
1-901725-59-6.
doi: 10.5244/C.30 C.30..7
[71] Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless cnns with low-procision weights. International Conference on Learning Representations, 2017.
[72] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruming for model compression, aix preprint 1rXi::7110.0188 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 6.2
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec. 6.2
(d) Have you read the hicis review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix and B
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No] The code proprietary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 6 and Appendices E-G
15

---

(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No] The computational cost of running NAS repeatedly to generate error-bars exceeds our computational capabilities.
(d) Did you include the total amount of compute and the type of resources used (o.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We report the search cost, normalized to typical GPU NN training setup of 460 images / second, in Table 3. We also report the search cost in absolute terms in Table 5.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing releasinn new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We used the CIFAR100, ImageNet, div2k, set5, and set14 datasets with the appropriate creators cited.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material as
URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/ curatinn? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally
identifiable information offensive content? [N/A]
5. Tf you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots,
if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "vcpuK5qYbi", "reviewer": "Reviewer_AJPZ", "summary": "The paper proposes a way of exploring compressible NNs across different architectures, sparsity levels, and quantization levels during training. The proposed framework is called Unified DNAS for Comppresssible (UDC) NNs and it can yield compressed NNs that show better accuracy-model size trade-offs than prior work. ", "questions": "Please see the previous section. ", "limitations": "The authors discussed the limitations of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper tackles an important problem and the proposed solution seems reasonable. The authors claim that UDC outperforms prior work in terms of accuracy-model size and the experiments seem to support this. A brief description of the prior methods that the UDC framework is compared to such as HAQ, Gong et al. [27], McDonnel, [51], FBNet, MBNetV2, Choi et al., [18] would be really helpful for readers. \n\nIn my opinion, the paper is a bit hard to follow. Specifically, Introduction could be a lot clearer with a more structured presentation and perhaps with a few subsections.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "klLT4YWJSFv", "reviewer": "Reviewer_2bCy", "summary": "This paper is on model compression (quantize and prune) with hardware constraints. The proposed search method is an extension of differentiable NAS to learn weight sparsity ratios and bit-widths per layer. Conceptually, this work combines previous ideas, such as DNAS (of layer width, depth, or operator) and learning of layer-wise bitwidth and sparsity, and yield pareto-optimal results on model size vs. accuracy. \n", "questions": "- What is the role of the Vela compiler in the proposed framework? Are the results on model size, e.g., in Fig. 4 & Fig. 5, from the use of the Vela compiler afterwards? \n- What are the performance results, in terms of latency/inference time, of the searched, compressed, and deployed models? Setting the optimization goal to be smallest model size is reasonable if the model size matters the most. But what is the impact on inference performance? The work mentioned using unstructured pruning, would that hurt the performance? Please comment on that.\n- Can you compare the searching costs with other work? For example, the GPU days and training FLOPs could be helpful in understanding this work when compared with others.", "limitations": "The authors have addressed the potential societal impact. \nThe limitations are not addressed enough. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The methods used in the paper are not new. But this work is a sort of combination of well-known techniques. Thus, making it work, i.e., DNAS for model architecture with pruning and quantization, is a new contribution. The related work part discusses the differences from prior work, and the related work is adequately cited to my knowledge.\n- The submission is technically sound with mostly empirical results. But it does not evaluate the weaknesses. For example, the searching costs of the proposed algorithm. \n- The writing is okay. The design decisions and the proposed algorithm are described with details. Since the major contribution, to my understanding, is the practical issues of making DNAS with compression techniques working, I think more details are necessary to help readers and practitioners to use the findings in this work. For example,  experiment setups and code examples are helpful, especially if no source code available. \n- The results are useful but not significant. In some scenarios with strict limitations on model size, the proposed method could be used. The reported benefits (model size vs. accuracy) are not unique.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TsmjiWKhlgl", "reviewer": "Reviewer_c4gz", "summary": "The authors present am updated Differential Neural Architecture Search algorithm, UDC, which incorporates model compression features such as weight sparsity and quantization as part of the search space for the ideal (high accuracy, small model size) neural network (NN) model. In addition, keeping in mind the constraints typically faced by TinyML models, the authors also enforce a strict limit on the model size to ensure it can be implemented in resource constrained IoT applications. Further, the presented algorithm enables effective exploration of accuracy vs model size tradeoff. Lastly, the paper demonstrates the pareto dominance of NNs designed by UDC compared to prior art for a range of ML benchmarks and varying hardware constraints on Ethos U55 NPU.", "questions": "Refer to questions addressed in Strengths/Weakness section.", "limitations": "Authors do not present any data showing the timing performance of UDC compared to other approaches, some perspective on that would be helpful. Also any explanation on how the memory size constraints were selected for presented experiments would be helpful. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents a thorough comparison of UDC against past techniques that attempt similar co-optimization of ML models while accounting for hardware constraints. UDC tackles a wider range of constraints compared to all listed in prior art. \n\nSection 3 and 4 describe the details of the design model and the proposed DNAS algorithm. These sections were somewhat hard to follow, however, it is understandable considering the mathematical nature of problem. It is not obvious what these variables denote, ξ & ν, kindly add a description. \n\nSection 6 shows a comparative study of generating NNs using UDC and other prior art for state-of-the-art (SOTA) ML benchmarks under different memory size constraints. Figure 4 illustrates the superiority of UDC in terms of accuracy and model size, could the authors also shed light on the runtime it takes to converge to the final NN design UDC picks compared to other approaches (e.g., MCUNet). Considering UDC accounts for a larger set of constraints compared to other approaches it would be interesting if it can demonstrate comparable runtimes too. \n\nFigure 5b, compares UDC results to random search, and the authors do share some insight on UDC runtime compared to training a baseline network (~2x), if they could share this runtime information it would be handy. Also note that authors should be careful that while comparing to random search the resource utilization for UDC should be similar e.g., running equal number of parallel GPU instances for a fair comparison. It would be interesting to know how long it takes to converge the random search to the optimal design picked up by UDC (withing reasonable time constraints of course).\n \nAdditionally, could authors share how they selected the memory size constraint values for the experiments shared in the paper, is there some understanding if the superiority of UDC holds for larger/smaller mem constraints and if yes how a user could understand if they fall in that range. \n\nTo conclude, this work promises a very useful optimization tool for designing small but high accuracy NNs that can fit on typical IoT devices servicing TinyML applications.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LfOBDuLi1rS", "reviewer": "Reviewer_uYMT", "summary": "This paper presents a differentiable NAS approach targetting compressible NN for NPUs with low memory footprint.\nThe approach conducts a joint search over NN architecture, weight bitwidths and layer-wise weight sparsity levels to find models with the smallest model size.", "questions": "- What is the decompression overhead when the compressed model is executed on the target device?", "limitations": "The limitations are adequately discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- A theoretical lower-bound of the storage size that is cheap to be computed.\n- Clear three-stage training process to deal with quantization and sparsity.\n- Improved pareto-front of model size vs accuracy over previous work.\n- The search algorithm has better sample efficiency vs random method.\n\nWeaknesses:\n- Make it clear that the target device is mobile-based NPU with limited model storage.\n- Models targetting small NPU and MCU have strigent latency and memory utilization constraints, which are not discussed in this paper.\n- Limited comparison with other NAS algorithms.\n- There are other NAS papers for MCU, please also cite and compare to them. For example: https://arxiv.org/abs/2010.14246", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept