QUTE: Quantifying Uncertainty in TinyML models with Early-exit-assisted ensembles for model-monitoring
Authors: Nikhil Pratap Ghanathe
Steven J E Wilton
Venue: ICML 2025 poster (2025)
Keywords: ML Model monitoring
TinyML
Resource-efficient learning
Uncertainty quantification
Early-exit networks
Trust in ML
Lightweight neural networks
Ensemble learning
Failure detection
Primary area: general_machine_learning->hardware_and_software
TL;DR: QUTE is a resource-efficient early-exit ensemble method for on-device uncertainty quantification (UQ) and monitoring in tinyML, reducing model size by 59% and latency by 31%, while detecting accuracy drops. It fits on a 256KB microcontroller.

Abstract: Uncertainty quantification (UQ) provides a resource-efficient solution for on-device monitoring of tinyML models deployed remotely without access to true labels. However, existing UQ methods impose significant memory and compute demands, making them impractical for ultra-low-power, KB-sized tinyML devices. Prior work has attempted to reduce overhead by using early-exit ensembles to quantify uncertainty in a single forward pass, but these approaches still carry prohibitive costs. To address this, we propose QUTE, a novel resource-efficient early-exit-assisted ensemble architecture optimized for tinyML models. QUTE introduces additional output blocks at the final exit of the base network, distilling early-exit knowledge into these blocks to form a diverse yet lightweight ensemble. We show that QUTE delivers superior uncertainty quality on tiny models, achieving comparable performance on larger models with 59% smaller model sizes than the closest prior work. When deployed on a microcontroller, QUTE demonstrates a 31% reduction in latency on average. In addition, we show that QUTE excels at detecting accuracy-drop events, outperforming all prior works.

QUTE: Quantifying Uncertainty in TinyML models with Early-exit-assin ensembles for model-monitoring
Nikhil Pratap Ghanathe
Steven J E Wilton
Abstract
Input (f))
fott
Po(sl (x)
Uncertainty quantification (UQ) provides a resource-efficient solution for on-device monitor- ing tinyML models deployed remotely with- out access true labels. However, existing UQ methods impose significant memory and compute demands, making them impractical for ultra-low- power, KB-sized tinyML devices. Prior work has attempted to reduce overhead by using early-exit ensembles to quantify uncertainty a single for- ward pass, but these approaches still carry pro- hibitive costs. To address this, we propose QUTE, novel resource-cfficient carly-exit-assiiee semble architecture optimized for tinyML models. QUTE introduces additional output blocks at the final exit of the base network, distilling early-exit knowledge into these blocks to form a diverse yet lightweight ensemble. We show that QUTE delivers superior uncertainty quality on tiny mod- cls, achieving comparable performance on larger models with 59% smaller model sizes than the closest prior work. When deployed controller, QUTE demonstrates 31 % reduction in latency on average. In addition, we show that QUTE excels at detecting accuracy-drop events, outperforming all prior works.
f2
fp=
fD
finds
Pox (1)x))
9h,
Sants
Po 2r)
Sunts
Resource hungry ECs assist the output blocks [out Hi only during training, promoting diversity. Removed for inference
ENSEMBLE (K.) (pp, P. (ollz)) Por (wl
) Figure 1: QUTE architecture. {.foua] k= represents the 'K' additional output blocks at the final exit, which assisted by 'K' early-exit blocks {ge, fK=1 only during training to promote diversity (see Figure 2). For inference, all early- exits and fout are removed
en-
and covariate- shiffedd data caused by environmental and sen- sor variations that manifest often unpredictably in the field. The ability of tL models to accurately measure the un- certainty of their predictions is crucial for two reasons. First, these devices generate data that is frequently used in critical decision- decision-making. In the context of an autonomous vehicle (AV), uncertain predictions could result in the downstream system making cautious driving decisions (Tang et al., 2022). Second, tinyML devices often operate remote environ- ments (Vargas al., 2021). If the inputs the model changes (perhaps due spatter, fog, frost, noise, motion blur, etc.), being aware of the model's unreliability may prompt an engineer to take remedial action. Many prior methods for drift/corruption detection are either statistical tests/methods that operate directly on input data (Bifet & Gavalda, 2007) with huge memory requirements, or require true labels for error-rate calculation (Gama al., 2004; Baier et al., 2022). In contrast, uncertainty quantification (UQ) methods aim to estimate model's prediction confi- dence without requiring true labels, providing efficient solution for model monitoring, particularly in resource and power-constrained environments.
micro-
1. Introduction
Recent advancements in embedded systems and machine learning (ML) have produced a new class of edge devices containing powerful ML models. These milliwatt-scale KB- sized devices, often termed TinyML devices, have low com- pute and memory requirements. They are often deployed in remote environments with availability of true labels. This makes them susceptible to both out-of-distributte (OOD)
'Department of Electrical and Computer Engineering, Univer- sity of British Columbia, Vancouver, Canada. Correspondence to: Nikhil Pratap Ghanathe <nikhilghanathe @ wcw..ubc.ca.......oooo.c........
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In ML models, there are two main types of uncer- tainty (Kendall & Gal, 2017): epistemic and aleatoric. Epis-

---

QUTE: Quantifying Uneertainty in TIn models with Exrty-eiit-assiseed ensembles for model-monitoring
temic uncertainty is reducible and stems from limited data or knowledge, while aleatoric uncertainty is irreducible, arising from inherent randomness in data or model limita- tions. In dield-deployed tinyML models, both types of uncer- tainty can emerge due to semantic (OOD) and non-semantic changes in inputs. The more prevalent non-semantic cate- in-distribution (CID) data,
respectively. Section 7 concludes the paper.
2. Related Work
Uncertainty quantification Bayesian neural networks (BNN) are well-suited to quantify uncertainty of a model (Blundell et al., 2015; Hernandez-Lobato & Adams, 2015; Teye al., 2018), but are parameter-ineffiiinn and incur high resource/compute overhead. Monte Carlo Dropout (MCD) (Gal & Ghahramani, 2016) creates im- plicit ensembles by enabling dropout during multiple in- ference passes. Recently, ensemble networks like Deep- Ensembles (Lakshminarayanan et al., 2017) have been shown to produce good uncertainty estimates (Zaidi & Zela, 2020; Wenzel et al., 2020; Rahaman et al., 2021). How- ever, this too requires multiple inferences of individual net- works, and is impractical in terms of memory for tinyML. Other prior works have proposed multi-input and multi- output networks that combine multiple independent net- works into one (Havasi et al., 2021; Ferianc & Rodrigues, 2023), but they do not scale well and remain impractical for tinyML. (Ahmed et al., 2024) ensembles only normal- ization layers, but requires specialized hardware. Alter- natively, prior works (Qendro et al., 2021; Ferianc & Ro- drigues, 2023) have leveraged early-exit networks to cre- ate implicit ensembles. The closest work to QUTE is EfE-ensemlle (Qendro et al., 2021), which uses outputs of carly-exits as ensemble members. However, the early- exits appended with extra learning layers incur a prohibitive cost (Section 3). In contrast, ensemble distillation meth- ods (Havasi et al., 2021; Malinin et al., 2019; Tran et al., 2020) filter the knowledge of all ensemble members into a conventional neural network (NN). Hydra (Tran et al., 2020), which also uses single multi-headed network is closest to our work. Appendix B.2 discusses other non- bayesian single-pass deterministic works (Van Amersfoort et al., 2020; Mukhoti et al., 2023; Sensoy et al., 2018; Deng et al., 2023) for UQ that are relevant for the resource- constrained tinyML space.
gory, which we term corrupled-inddistrbbuiiii occurs when sensor malfunctions or environmental factors result in corrupted versions of expected data (e.g., fogged camera lens). CID data introduces both aleatoric (e.g., due to random noise or weather) and epistemic (e.g., from blur or digital corruptions) uncertainties, often simultancously, making it crucial to capture these corruption ceruptioo-induced induced un- certainties to ensure model reliability. Unfortunately, mod- ern neural networks are poor in estimating uncertainty of their predictions (Ovadia al., 2019). Many prior works have proposed uncertainty-awaa networks: ensemble net- works (Laakshminarayanan et al., 2017) have been found to be effective in capturing both alcatoric and epistemic uncertainties. Alteraatielly, early-exit networks have been converted into ensembles (Antoran et al., 2020; Qendro et al., 2021; Ferianc & Rodrigues, 2023). However, both these approaches incur high memory/compute overheads and are untenable for tinyML.
In this paper, we propose QUTE (Figure 1), novel and resource-efficient early-exit-assisted ensemble archi- tecture that enables high-quality uncertainty quantification in tinyML models in the context of both in-distribution (ID) and corrupted-ing in-distribution (CID) data. As shown in Fig- ure 1, we append additional lightweight classification heads to an existing base neural network to create ensemble mem- bers, and crucially, these classification heads are assisted by the carly-exits using a novel distillation scheme (Ghanathe & Wilton, 2023) to promote diversity. Post-training, we eliminate the resource-hungry early-exits while retaining only the economical additional classification heads. Our approach has significantly less memory and compute over- head compared to prior works (59% smaller models and 3.2x fewer FLOPS compared to the most relevant prior work (Qendro et al., 202 Furthermore, QUTE performs better than prior methods in estimating uncertainty caused due to CID error sources, and on-par with prior methods for uncertainty due to OOD. We further show that higher un- certainty correlated with a drop in accuracy. We evaluate QUTE's ability to detect such accuracy drop events caused by CID against prior methods, and show that QUTE outper- forms all prior methods. To the best of our knowledge, this is the first early-exit ensemble architecture for uncertainty quantification optimized for tinyML models.
Model monitoring There has been a plethora of work to detect OOD samples, which represents a semantic shift from in-distribution (Yang et al., 2021; 2022; Zhang et al., 2023; Liu et al., 2020; Hendrycks & Gimpel, 2018; Dinari Freifeld, 2022). . However, fewer works deal with detecting corrupted-ID, which represents non-semantic shift w.r.t. ID, and are much harder to detect (Liang et al., 2020). Some prior works utilize corrupted- siruppteddDD/coovaiatt--isiiiiie ter generalize on OOD (Bai et al., 2023; Katz-Samuels et al., 2022), which is difficult to obtain. The most relevant work that leverages only training data is generalized-ODN ODIN (G- ODIN) (Hsu et al., 2020). It adds 1) preprocessing layer that perturbs the input image and 2) decomposes confidence score for better OOD detection. However, the preprocessing
ID to bet-
This paper is organized as follows. Section 2 presents related work. The context and problem formulation are in Section 3. Our approach is described in Section 4. The experimental methodology and evaluation results are in Sections 5 and 6
2

---

QUTE: Quantifying Uncertainty in TInyML models with Earty-xxit-assisted ensembles for model-monitoring
Data Type
Model
Ace (t)
ECE (
BS (
NLL (
severely-corrupted SCID (Hsu et al., 2020). Interestingly, we find that the model becomes less overconfident to SCID as its size shrinks. This phenomenon is illustrated in Ta- ble 1. We report the calibration metrics of Resnet (He et al., 2016) on CIFAR 10, with four model sizes ranging from I to 4 residual stacks on both SID and SCID. For calibration metries (ECE, BS, NLL), lower is better. More details on calibration metries and CIFARIO-CIL can be found in Sec- tion 5 and Appendix D and A. As seen, the 4stack model with the highest model capacity is clearly the best on SID. However, on SCID, it performs the worst, remaining over- confident in the presence of corruptions, even outperformed by Istack. In contrast, 2stack and 3stack model are much better on SCI This demonstrates that smaller models are less overconfident for corruption in inputs, leading to better calibration on SCID-
Istack
0.72
0,0343
0,0381
0.8174
In-
2stack
0.78
0.0315
0.0305
0.6441
distribution
3stack
0.87
0.0373
0.0190
0.4063
4stack
0.89
0.0562
0,0161
0,4019
Istack
0.20
0.279
0,0993
3,216
Corrupted
2stack
0.20
0.2518
0.0969
3,158
in-distribution
3stack
0.27
0.2534
0.0953
3.201
4stack
0.25
0.4255
0.1087
3,677
Table 1: Calibration metrics of Resnet with [1,2,3,4]-stack models on CIFAR-10. Top: in-distribution data. Bottom: corrupted in-distribution data. Best results marked in bold. For ECE, BS, NLL, lower is better.
might prove costly/impracticaal tinyML devices. Recent studies (Xia & Bouganis, 2022; 2023; Jaeger et al., 2023) propose failure detection with reject option, rejecting high- uncertainty samples from both ID (potentially incorrect pre- dictions) and OOD, unlike traditional OOD detection which only separates ID from OOD.
The capabilities of smaller models can be harnessed for better uncertainty estimation on SCID through early- exit networks, as illustrated by prior research like EE- ensemble (Qendro et al., 2021), which combines the pre- dictions from multiple early-exits and final exit to form an ensemble. For example, in our evaluations, we find that on TinyImagenet-ID (Le & Yang. 2015) with Mo- bilenetV2 (Howard et al., 2017), the negative log- likelihoodd (NLL) of EE-ensemble is 26% lower than the popular Deep- Ensemble (Lakshminarayanan et al., 2017), and 34% lower on Tiny Samaagnne-Corrupler (Hendrycks & Dietterich, 2019) (lower is better). This showcases the crucial role of early- exits in achieving higher quality uncertainty overall. Further, these results align with prior findings (Hsu et al., 2020) that show that larger models overfit, leading to overconfidence, as they extract high-level abstractions that make distinguish- ing corrupted from clean inputs harder. In contrast, smaller models benefit from implicit regularization, preventing over- fitting. They generalize better on corruptions by relying on more stable features instead of memorizing fine-grained ones. Our empirical results (Table 1) support this. From a Bayesian perspective, smaller models can exhibit wider posterior due to implicit regularization, making them better at capturing uncertainty.
Early-exit networks Early-exit networks add intermediate exits along the length of the base network thereby, provid- ing avenues for reduction in average inference time (Teer- apittayanon et al., 2016; Kaya et al., 2019; Huang et al., 2017; Bonato & Bouganis, 2021; Ghanathe & Wilton, 2023; Jazbec et al., 2024). Ghanathe & Wilton (2023) introduces an early-exit architecture optimized for tinyML models, and develops a knowledge distillation mechanism to mitigate network overthinking.
3. Background and Problem formulation
Consider an in-distribution dataset SID = (Ix, yn N of size N where, Ty and yn are the nh input sample and its cor- responding true label respectively. A discriminative model Me(a) learns parameters θ on SIp and outputs a class posterior probability vector pe (e(yyz)) (z). which yields pre- dicted label j. For classification problem, i E (1, J. where L is the number of classes. Uncertainty quantification methods endeavor to improve uncertainty estimation quality of (z) on STp. Specifically, we want to calibrate the model such that its predictive confidence (CMa) is in sync with its accuracy (Ama). ). A well-calibrated model will see commensurate drop in CMa AMe drops. The predictive mar pe (|zz) Confidence-calibratee is a well studied problem, and it helps achieve synergy between the predicted probabilities and ground truth correctness likelihood (Guo et al., 2017). However, when Me is deployed in a real-world scenario, it may encounter either 1) 2) corrupted/covariate-hhhn ronmental/sensor variations (E.g., frost, noise, fog, rain etc.). Unfortunately, we find that despite good uncertainty estima- tion quality on SIp, many uncertainty- aware aware networks see a drop in quality and remain overconfident, particularly for
However, we find that EE-ensemble is resource-intensing because it adds additional learning layers at the early-exits to accommodate their varying learning capacities. For ex- ample, adding two early-exits without additional learning layers (say EE-O and EE-1) after 1st and 2nd residual stack of 3-residual stack Resnet for CIFAR10 10 results in indi- vidual accuracies of 0.602 and 0.707 at EE-Q and EE-1 respectively. This is significantly worse than the individual accuracy of the final exit, 0.84, which in turn leads to a poor ensemble behavior. To address this, Ee-ensemble uses early- exits with additional denseffully-commeede incurs a high memory overhead affecting practicality in tinyML (Overheeadgg-mmmmt: = 397K params compared to
confidence is given by, CMs (z) IE(1.2....L)
data (SoOD) or version (ScI)), due to envi-
layers, which
3

---

QUTE: Quantifying Uneertainty in TinyML models with Early-exit-assisted ensembles for model-monitoring
OverheadouTe 8.2K params shown in Section 4 & 6). a:-1
ft
aD-1
fo
Pox (M)
We present an alternative strategy to leverage the knowl- edge of early-exits in rusemble-creation that is extremely resource- -hourre-efficient thereby enabling superior uncertainty esti- mates on both SLD and SCID. which allows us to reliably detect accuracy-drops in the model due to SCID.
0x
fouts
hol
-Py (1ll2))
Figure 2: Farly-view-asistance architecture. One assist- ing early-exit ge, and corresponding early-view-assisted exit foute is shown. Early-exits weights 6g are transferred/- copied to hes before each train batch, i.c., Ox = 0₁
4. QUTE
A neural network (NN) like the one shown in Figure 1, with depth D is composed of several blocks of linear/non- linear operations (e.g., convolution, pooling) that are stacked. The NN consists of D - 1 intermediate blocks (f(())==): = f1) (2) fa(.)... fD- (.)o (.)) and an output block (.) fo()) is the input block. In the base network, the network processes the input fo(.) through each block f.) until layer D. ultimately producing predic- tion pe pe(u)z)) through output block (-).
termed as early-view since it is assisted by the early-exit. After attaching carly-exits (EE) and carly-view-assissee assisted (EV) exits to the untrained base network, all weights are learned simultancously during training. Unlike some prior works (Lakshminarayanan et al., 2017) that use extra data (e.g., adversarial samples), we only use the available train- ing data. We use random data augmentation (E.g.. random rotation/flip/crop), a well-known method to improve model robustness to corruptions to ensure faimess in accuracy-drop detection experiments (Section 6.2). The EEs are trained with a weighted loos function, similar to previous stud- ies (Kaya et al., 2019). Further, we assign a higher weighting factor, WEV, for losses at E'Vexiits to minimize the effect of overconfident fout (original output block). In addition, we increase each WEVi by 8 such that WEV, WEV,_, + 8 to vary the influence each EV-exit has on the weights of the shared base network, thereby promoting diversity. We empirically determine that WVV, = 3 and = 0.5 yields the best results (see Appendix A for details). During training.
Given a base NN (the grey blocks in Figure 1), we first create K additional classification heads/output blocks {foutz 1K ===1 at the final exit (after fD) as shown in Figure 1. These output blocks constitute the ensemble AC. Next, we add K early-exit blocks, {go, K along the length of the base network, which are used only during training for knowledge distillation. A more intuitive view of QUTE is presented in Figure 5. In Figure 1, K is set to D- - 1. In practice, K is a hyperparameter that depends on computation/resouree bud- get and the required ensemble size |/], where the ensemble size is bounded above by the depth of the network. Thus, [fouty = produces K predictions, where each fout, is assisted in producing predictions by its corresponding early- exit 90, using early-view assistance method.
1) Before each train batch, we copy weights of hthh EE-exit (he,) into hthh EV-exit (hor) ) i.e., ΦR OR- Thus, each EV-exit iterates through train batch with weights of the EE-exit. Weight transfer is limited solely from ha, to hau with no other layers involved.
K. = (p))()))) pos (y) 1x), Pos (ulx),..... ...... Pox (ul
(1)
Figure 2 illustrates the architecture of varl assistance, where a single final exit block Sout, and its assisting early- exit go, is shown. As shown, we create an additional learn- ing layer has (.) at the h4th output block. An identical learn- ing layer he, (-) is also added at the 1tth early-exit. heg with parameters Φe is responsible for assimilating knowledge from the corresponding heg with parameters Og. ol.) is the output activation (E.g., dofseesoftmax). har and hes are a single depthwise-convolutie owing to its low computation and memory demand. We en- sure equal dimensionality between lt,, and he, by adding a pointwise-convolude (not shown in figure for brevity). Fundamentally, our aim is to imbue the diverse predictive behaviors of early-exits from different depths into the output blocks (Sout) } R=1-
2) After a train batch, the loss computed at an EV-exit is w.r.t. the weights copied from EE, after which weights of he, are updated during backpropagation. However, before the next train batch begins, we overwrite the weights of has by copying from EE Therefore, the weights learnt by layers before/after hag always align with EE weights. Thus, this does not affect convergence (see Figure 6 and Section B.3).
layer in our evaluations
K Σ
K
C
CEE. + ELEV. + LEF (2)
layer before heg at all carly-exits
LEE. = Tk- LCE
LEV. = WEV, LCE
The training objective reduces to minimizing Eqn 2, where L is the total loss of QUTE, which is the sum of the losses of EE, EV and final exits. Log is the cross-entroge loss, and CEF = LCE, which is loss of original final exit. LEE, and LEV, are losses at the th EE-exit and EV-exit respectively.
Training Unlike EE-ensemble, all output blocks of QUTE have similar learning capacities since the input to all (he, ] is fn of the base network. The output of heg is
4

---

QUTE: Quantifying Uncertainty in TInyML models with Earty-exit-assisted ensembles for model-monitoring
Th is the weighting applied to EE loss (Kaya et al., 2019).
4-layer depthwise-separable model (DSCNN) (Zhang et al.,
WEV, is the weighting applied the kth EV-exit.
2017) for keyword spotting task, 3) CIFAR1O (Krizhevsky, 2009) on Resnet-8 (from MLPerf tiny benchmark suite (Ban- bury et al., 2021)) and 4) TinyImagenet (Le & Yang, 2015) on MobilenerV2 (Howard et al., 2017). While Tinylma- geNet/MobileNet22 isn't a typical tinyML dataset/model, we include it to show QUTE's broader applicability. De- spite their relative simplicity, these datasets exemplify the problem sizes often handled in KB-sized tinyML. devices.
The weights of base network (only grey blocks in Figure 1) are frozen for the last 10% of epochs while continuing weight transfer, enabling isolated training of EV-exits to foster increased diversity. In this way, the EE-knowledge transferred to the final exit(s) to obtain better uncertainty estimation on SID/So SSID. In contrast, without EV-assistance, the various final exits risk learning the same predictive dis- tribution, leading to poor ensemble behavior. We compare the effectiveness of EV-assistance in Appendix B.3.
CID datasets: For CID datasets, we use corrupted ver- sions of ID i.e. MNIST-C (15 corruptions) (Mu & Gilmer, 2019), CIFAR 10-C (19 corruptions) and TinyImagenet-C (15 corruptions) (Hendrycks & Dietterich, 2019), with var- ious corruptions such as frost, noise, fog, blur, pixelate etc. that induce both epistemic and aleatoric uncertainties (see Appendix A.2). For corrupting SpeechCmd, we use the audiomentations library (Jordal, 2024). We apply 11 types of corruptions such as noise, air absorption, time masking/stretching to obtain noted SpeechCmd-C) These corruptions affect the input data in various ways by altering key visual features (see Ap- pendix A.2), potentially degrading model performance. For example, noise corruptions (e.g., impulse or gaussian) intro- duce random pixel-level variations, while weather-related corruptions (e.g., fog, snow, or rain) obscure critical visual elements necessary for accurate feature recognition.
Inference During inference, the resource- EE-blocks are removed from the model, leaving only the base network and the lighter output blocks {fontz K We also elimi- nate the original output block four due its overconfident behavior, which compromises calibration quality. The final prediction (z))) is obtained by calculating the mean of all [A]] prediction vectors of K.
(de-
K Pou (ulz))
1 |C|
pe
(3)
5. Evaluation Methodology
For our microcontroller (MCU) evaluations, we utilize two types of boards.
OOD datasets: SpeechCommands has utterances of 35 words. Like Zhang et al. (2017), we train DSCNN to recognize ten words and use the rest as OOD (de- noted SprechCmd-OOD Furthermore, we use FashionM- NIST (Xiao et al., 2017) and SVHN (Netzer et al., 2011) as OOD for MNIST and CIFARIO respectively. More details models and datasets used can be found in Appendix A.
1) Big-MCU: The STM32 Nucleo-144 board (STM32F767ZD) (STMicroelectronics, 2019) with 2MB flash and 512KB SRAM, clocked at 216MHz with a typical power consumption of 285mW. This board accom- modates high-resource baselines that require substantial memory and processing power.
2) Small-MCU: The STM32 Nucleo-32 board (STM32L432KC) (STMicroelectronies, 2018) with 256KB flash and 64KB RAM,, clocked at 80MHz with typical power consumption of 25mW. This ultra-low-pows device better reffects our goal of deployment on smaller, energy-efficient systems.
Placement of Early-exits The number of carly-exits (K) is dictated by the resource budget. There are only a handful locations a tL model where we can insert the early- exits. Our strategy is to insert them at equally- tions along the base network. For MNIST and SpeechCmd, which both use 4-layer models, we insert two EEs after the 1st and 2nd layers. For Resnet-8 with 3 residual stacks, we insert EEs after 1st and 2nd residual stacks. We insert 5 EEs for MobilenetV2 at equally-spaced locations starting from the input.
loca-
We evaluate QUTE against several relevant baselines (Ap- pendix A. in three settings: 1) Accuracy-drop/CID de- tection, which monitors system performance over time to detect gradual shifts and temporal degradation patterns (Sec- tion 6.2), 2) Failure detection, which targets instance-level identification of both misclassifications and OOD samples (Section 6.3), and 3) uncertainty estimation quality (Sec- tion 6.4). Since the focus of this work is detect failures reliably in-the-field, accuracy-droy and failure detection experiments evaluate this capability.
dccuracy-drop detection Unlike with OOD data, correct predictions may still occur on CID inputs. Therefore, we focus on detecting accuracy-drop events caused by CID inputs. We present a realistic mechanism with no additional overhead that monitors confidence to detect a possible drop in accuracy. We formulate this as a binary classification problem. First, for each baseline we evaluate, we iterate over only ID and obtain model predictions while computing the moving average of accuracy of the past m predictions using sliding- window... The accuracy of the sliding-window
Datasets and Models We evaluate QUTE on one audio classification and three image classification tasks of dif- fering complexities: 1) MNIST (LeCun et al., 1998) on a 4-layer CNN, 2) SpeechCommands (Warden, 2018) on a
window is
5

---

QUTE: Quantifying Uncertainty in TinyML models with Early-xxit-assisted ensembles for model-monitoring
Big-MCU
Big-MCU
Small-MCU
Small-MCU
400
200
600
100
150
300
BASE
400
ell-ens
50
100
DEEP
200
50
COM
00M
100
COM
QUTE
0
0
SpCmd
cfr10
SpCmd
efr10
SpCmd
ct:10
SpCmd
ct:10
(a) Size (KB)
(b) Latency (ms)
(c) Size (KB)
(d) Latency (ms)
Figure 3: Microcontroller results for SpeechCmd (SpCmd), CIFARIO (cfr10) on Big-MCU and Small-MCU (lower is
better). EE--nneemlee and DEEP on CIFARIO do not fit i.e., out-of-memory (0OM)
denoted as Asw and the accuracy distribution thus obtained is denoted as AID with mean HID and saadard-deviation aID respectively. In our empirical evaluations, we find that m = 100 is reasonable. We then construct corrupted datasets by appending ID with each corrupted-ID version. Next, we iterate over these ID+CID datasets (one for each sliding-window window while computing
bration error (ECE) (Guo et al., 2017) due to it popularity, Appendix D discusses the limitations of ECE.
6. Results
The results section is organized as follows. First, Section 6.1 examines the MCU fit of QUTE compared to resource- heavy prior works, highlighting its 27 % lateney reduction on Big-MCU and 29% smaller model size compared to the lowest-latency baseline. On Small-MCU, high-resource prior methods cannot even fit when using CIFAR10, further underscoring QUTE's suitability for constrained environ- ments. Next, Section 6.2 demonstrates QUTE's superiority over all baselines in accuracy-drop detection. On failure detection (Section 6.3), QUTE exceeds prior methods in distinguishing between IDV/ IDx and is competitive on IDV| OOD with other baselines. Finally, Section 6.4 shows that QUTE exhibits good calibration quality especially on tiny-sized models with 59% fewer model parameters on average compared to EE-ensemble. Additional results and ablation studies are presented in Appendix B.
type of corruption) using a sliding- the moving average of confidence (Csw). A CID-detected event corresponds to Csw < p. and a true positive event occurs when Csw < p and Asw S PID - 3 IID. where p is a user-defined threshold. We vary P from 0 to 1 in step sizes of 0.1 and repeat the whole experiment. Appendix A.6 provides additional details. Finally, we report the average area under precision-recall curve (AUPRC).
Failure detection Failure detection aims at rejecting both ID misclassification and OOD instances, as both indicate potential failure. To this end. we define two binary classifica- tion tasks: 1) IDV| IDx, which separates correct predictions from incorrect ones within ID/CID samples, and 2) IDV| OOD, which distinguishes correct predictions from OOD. For the former, we use ID+CID datasets where errors can arise from hard-to-classify ID samples CID inputs, while for the latter, we use ID and OOD data. (Xia & Bouganis, 2023) leverages uncertainty estimates obtained through cas- caded ensembles for failure detection, optimizing for a user- defined threshold. However, since this threshold may vary in the field depending on application requirements, we report Area Under the Receiver Operat- ing Characteristic curve (AUROC). Finally, Xia & Bouganis (2023) shows that Deep Ensembles (DEEP) achieve superior performance on threshold-fres metrics, making DEEP the strongest baseline in our comparisons.
6.1. MCU Fit: QUTE vs Resource-Heavy methods
We evaluate MCU implementations of QUTE on two MCUs of different sizes. Since the primary objective of TinyML systems is to minimize the energy-per-ppe pereedee--nniiiioonn (Ghanathe & Wilton, 2023), our target platform is the Small-MCU, as the base network fits comfortably on this smaller device.
the prreshold-indeeendeee
We compare QUTE against the two most relevant baselines: e.s-eeneeblle and DEEP. We exclude MCD due to its re- liance on a specialized dropout module (Ahmed et al., 2023), which is impractical on MCUs. We also exclude HYDRA because it is often suboptimal. shown in the following sections. Our evaluation uses CIFARIO and SpeechCom- mands, as TinylmageNe/MMobilNNeeWy exceeds tinyML de- vice limits. On Big-MCU, QUTE achieves 31 % and 47 % latency reductions over EE-ensemble and DEEP, respec- tively, and maintains accuracy parity with both, even with 58% and 26% smaller models. On Small-MCU, both EE-
Uncertainty quantification For UQ, we report the class- weighted Fl score and two proper scoring metrics (Gneiting & Raftery, 2007): Brier score (BS) (Brier, 1950), which measures the accuracy of predicted probabilities and nega- tive log-likelihood (NLL), which measures how close the predictions are to the ground truth (see Appendix D). Also, despite its documented unreliability in measuring uncer- tainty quality (Nixon et al., 2019), we report expected cali-
6

---

QUTE: Quantifying Uneertainty in TinyML models with Earty-cxit-assisted ensembles for model-monitorinn
AUPRC (1)
MNIST-C
SpeechCmd-C
0.3
0.8
BASE
BASE
0.54
0.52
MCD
MCD
0.45
0.56
0.5
0.6
DEEP
DEEP
0.55
0.57
LB eas
Entembeeble
0.63
0.58
0.4
G-ODEN
G-ODIN
0.48
0.37
0.2
(LYRA
QUTE
HYDRA
0.53
0.51
QQUEE
0.63
0.61
(a) AUPRC MNIST-C MNIST & SpeechCmd-C
Severity
Severity
(b) AUPRC CIFARIO-C
(c) AUPRC Tinylmagenet-C
Figure 4: Accuracy-drop detection results. AUPRC is reported for all evaluated baselines (higher is better)
ensemble and DEEP do not on the device for CIFARIO; EFEeensemlle model size exceeds Small-MCU's on-device memory by 270KB, whereas DEEP exceeds by 1.9KB. No- tably, EE-enseble has the highest peak RAM usage than DEEP and QUTE due to its need to store large intermediate feature maps during early-exit computation. In contrast, DEEP processes each ensemble member sequentially there is no parallel compute capability on MCUs. This char- acteristic allows fngle-forward -aasddd--assssee pass methods like QUTE and EE-ensemble to achieve reduced latency, especially as the ensemble size increases. Appendix C delves into more details. These results highlight the memory limitations of ultra-low-power MCUs for tasks like model monitoring, em- phasizing QUTE's practicality in such constrained settings where the alternative may be no UQ monitoring.
AUROC
ID<| IDx
[D/| OOD
Mnist
SpCmd
cfr10
Mnist
SpCmd
cfr10
-BAE
0.75
0.90
0.84
0.07
0.90
0.88
-MCD
0.74
0.89
0.87
0.48
0.89
0.89
DEEP
0.85
0.91
0.86
0.78
0.91
0.92
-EE-ens
0.85
0.90
0.85
0.85
0.90
0.90
-G-ODIN
0.72
0.74
0.83
0.4
0.74
0.95
HYDRRA
0.81
0,90
0.83
0.71
0,90
0.90
QUTE
0.87
0.91
0.86
0.84
0.91
0.91
Table 2: Failure detection results
age median confidence on CID is 0.57, which helps create a much clearer distinction between ID and CID. EE-ensemble performs comparably to QUTE for tiny/medium-sinn els. However, for large models, the additional learning lay- ers added to balance the learning capacities of all exits cause the early-exits to function like conventional deep networks. Therefore, EE-ensemble becomes overconfident for severe corruptions, which results in a 8.9% drop in AUPRC on Iinylmagenet-C. Similarly, G-ODIN remains overconfident on CID, and is surpassed by most baselines on all corrupted datasets. This suggests that general OOD-detectors are not automatically effective for CID detection, consistent with previous findings that OOD detectors find it hard detect non-semantic shifts (Hsu et al., 2020). Further, HYDRA is consistently subpar at accuracy-drop detection, often being outperformed even by BASE. This highlights HYDRA's need for larger classification heads with more parameters to fully integrate misemble-knowleddee consistent with the original paper's methodology. Contrarily, QUTE's early- exit distillation method with much fewer parameters (only single depthwise CONV layer) proves to be more effective.
mod-
6.2. Accuracy-drop detection
As described in Section 5, we aim at detecting accuracy- drops due CID by monitoring the confidence. We also explore alternative uncertainty measures like predictive tropy. However, we choose confidence because it produces similar results without extra processing, a practical advan- tage for MI.. Figure 4 reports AUPRC averaged over all corruptions for all datasets, and for five severity levels in the case of CIFARIO-C and Tinylmagenet-C. By com- bining knowledge from EE at the final exits, QUTE consis- tently outperforms all baselines in detecting accuracy-dre droops caused due to CID across all datasets, which is crucial for maintaining model reliability in deployment.
MCD performs poorly in all cases either due to reduced model capacity (due to dropout) or due to extreme cali- bration. For example, on Tinylmagenet-ID. MCD with an accuracy of 0.34 has median confidence value of 0.345 across all predictions. However, on Tinylmagenet-CID the overparameterized nature of modern neural networks does not allow the median confidence of MCD to drop any fur- ther than 0.27 i.e., a drop of 0.075, which is not discernible enough to detect drop accuracy. This observed with EE- ensemble too for larger model sizes. In contrast, QUTE's median confidence on Iinylmagenet-ID 0.69 and the aver-
6.3. Failure detection
Table 2 reports the AUROC for failure detection exper- iments for ID/ IDx and IDv| OOD. As seen, for ID=I IDx, QUTE outperforms all baselines on MNIST and SpeechCmd. MCD performs slightly better on CI- FARIO, likely due to the higher number of epistemic uncertainty-indee induciig corruptions present CIFARIO-C. such as blur and digital distortions. This observation aligns
7

---

QUTE: Quantifying Uneertainty in TInyML models with Earts-exit-assisted ensembles for model-monitoring
Model
F1 (1)
BS
NLL
ECE
impractical for tinyML because they both require multiple
MNIST
BASE
0.910=0..02
0.0134-0.000
0.292:40.006
0.0144+0.001
inference passes or specialized hardware.
-MCD
0.886=0.004
0.018+00000
0.382=00004
0.071+00.006
DEEP
0.901
0.010:10.000
0.207
0.03411.004
QUTE with relaxed resource constraints: QUTE also outperforms EE-ensemble, the most relevant prior work, across all datasets with models that are, on average, 59% smaller, except on larger datasets and models like Tinylm- agenet on MobileNetV2. Since QUTE's s primary focus is resource efficiency, the limited parameters in its out- to single depthwise convolutional (DCONV) layer-can can constrain calibration performance rel- ative to high-resoure methods like DEEP and EE-ensemble.
-EEeensemble
0.939=0.002
0001000:000
0.266=00005
0. 0.108++0.002
-HYDRA
0.932=0.006
0.010::0.000
0.230=0.012
0.01410.005
-QUTE
0.941=0.004
0.009:10.000
0.010
0.026:10.03
Spe
-BASE
0.923=0.007
0.010:10.000
0.026::0.001 0.216
-MCD
0.917:10006
0.01 00010000000
0.201-0.0013
0.048.10.02
-DEEP
0.934=0.008
0.008:00.000
0.205=
0.034+00.006
EE-enseeblee
0.926::0.002
0.009:10000
0.206
0.029:10.001
-HYDRA
0.902
0.008
0,203 0.016
0.018:10.004
put
-QUTE
0.933=00006
0.008:00.000
0.002
0.018:00001
CIFARIO
-BASE
0.834=00.055
0.023+:0.000
0.623
0.049440.003
-MCD
0.867=0.002
0.019=000000
0,017+00000 0.096=0.003
To normalize the comparison, we enhanced the QUTE ar-
-DEEP
0.877=0.003
0.017::0.000
0.015
0.015:10.003
ensemblle
0.85410.001
0.021110.000
0.446:1.011
0.033110.001
chitecture with additional learning layers at each output block, creating QUTE + For Resnet-8 on CIFARIO, IO, we added two Repthwise-separbbl CONV layers, while for MobileNetV2 on TinylmageNet, we included additional dense (fully connected) layer to form QUTE + As seen in Table 3, F1 score of QUTE + exceeds that of DEEP on CI- FARIO, I0, however, it slightly degrades on Tinylmagenet. Fur- ther analysis revealed that since the base network is trained simultaneously with all early-exits (EE) and EV-exits, the weights of the shared base network negatively impacted since the training routine tries optimize for all exits. In contrast, EE-ensemble method employs only half as many exits (only EEs) compared to QUTE, and the EE's EE's regu- larization effect helps network accuracy (Teerapittayanon et al., 2016). Appendix B.4 studies the effect of ensem- ble size on QUTE's s calibration performance. Nevertheless, QUTE + significantly improves calibration, outperform- ing all baselines BS and achieving NLL on par with the oo-performing method. These results indicate that, in unconstrained setting, QUTE can achieve calibration performance comparable to high-resource methods
-HYDRA
0.818=0.004
0.026::0.000
0.632=0.011
0.069:100011
-QUTE
0.858
0.020:10.000
0.408:00.019
0.025:10.003
-QUTR+ +
0.017
0.369 00008
0.026-1-0001
Tiny Imagenet
BASE
0.000
0.004110.00
0.084
0.416:10.003
-MCD
0.332=00.04
0.003++0.000
2.844=0.028
0.061++0.005
-DEEP
0.414=0.006
0.003::00000
3,440=0.049
0.115+0.003
-EE-ensemlle
0.430=00005
0.003=0.000
0.534
0.032:100.06
-HYDRA
0,376=0.004
0.004=000000
3.964=0.036
0.328-000044
-QUTE
0.395=0.014
0.00410.000
3.70
0.282::00009
-QUTE+ +
0.38110.010
0.003:10.000
2.757-10.04
0.122
Table 3: Calibration Metrics for all baselines on ID data.
The best results are marked in bold. All results are mean + std-dev for three independent splits of ID test data.
with prior work highlighting MCD's strength in captur- ing instance-level epistemic uncertainties (Kendall & Gal, 2017). For IDv| OOD, QUTE exceeds all baselines on SpeechCmd, and is close second MNIST. Surprisingly, G-ODIN, specialized OOD detector, is poor MNIST and SpeechCmd, even outperformed by BASE. Further anal- ysis revealed that G-ODIN is under-confident on these ID- datasets and over-confident on OOD. As an ablation study, we increased the number of training epochs of G-ODIN by 30 epochs, which resulted in a 55% o improvement on ID-| OOD on MNIST. Contrarily, on larger model, Resnet-8 on CIFARIO. 10, G-ODIN's performance notably improves on ID~I OOD, consistent with previous findings. This may suggest the need to rethink OOD detection with extremely tiny models. These results showcase QUTE's versatility/ef- fieacy in detecting both CID and OOD data.
Additionally, we evaluate QUTE's early-exit distillation against HYDRA's ensemble-distillation. While HYDRA's lightweight heads capture ensemble knowledge in tiny mod- els, they falter with larger models, reducing accuracy and calibration, even underperforming BASE on CIFARIO. This highlights HYDRA's dependency on larger heads for effec- tive ensemble distillation.
7. Conclusion and Discussion
6.4. Uncertainty quantification
Uncertainty quantification is essential for model monitor- ing in tinyML. yet many prior works fail to address the resource constraints inherent in this domain. We propose novel resource-efficient ensemble architecture for uncer- tainty estimation that is optimized for tinyML. At the core of our methodology is our finding that model overconfidence decreases with its size. We leverage the better uncertainty es- timation quality of early-exits (EE) by injecting EE weights into the multiple lightweight classification heads created at the output. QUTE provides reliable uncertainty esti-
QUTE consistently matches or even outperforms all base- lines on tiny-sized models such as MNIST and SpeechCmd, and remains competitive on medium and large-scale bench- marks like CIFAR-I0 and dinylmageNetddepptte operat- ing with significantly fewer computational resources. All methods outperform BASE, except MCD on MNIST and SpeechCmd, where inference-time dropout reduces model capacity, causing accuracy drops and poor calibration. DEEP showcases good calibration on all datasets owing to its larger model capacity. However, MCD and DEEP are
8

---

QUTE: Quantifying Uneertainty in TinyML models with Earty-exit-assisted ensembles for model-monitoring
mates in a single forward pass with 59% smaller models on average and 31% % average reduction in lateney on low-power MCUs. Furthermore, QUTE effectively distills EE knowledge into the final exits, enabling it to capture both aleatoric and epistemic uncertainties more effectively. This capability leads to superior performance in detecting accuracy-drop events (corruptions) across diverse range of datasets and model complexities. QUTE serves as a practi- cal and cost-effective accutrcy-monitrring mechanism for field-deployed models.
References
Ahmed, S. T., Danouchi, K., Münch, C., Prenat, G., Anghel, L.,, and Tahoori, M. B. Spindrop: Dropout-base bayesian binary neural networks with spintronic imple- mentation. IEEE Journal on Emerging and Selected Top- ics in Circuits and Systems, 13(1):150-164 2023, doi: 10.1 W..1000ECAASS22332222111
Ahmed, S. T., Hefenbrock, M., and Tahoori, M. B. Tiny deep ensemble: Uncertainty estimation in edge ai accel- erators via ensembling normalization layers with shared weights. In Proceedings of the 43rd IEEE/ACM Interna- tional Conference on Computer- Design, pp. 1-9, 2024. Antoran, J., Allingham, J., and Hermández-Lobato, J. M. Depth uncertainty in neural networks. In I Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 10620-10634, Curran Associates, Inc., 2020. URL httpps::/prooccc.nggggg//// cc htr/:pper_fiilleeprrr///00000000 wew.eeerdddnnnnnnnne..e.e..... neurips. 781 1 pdf.
The ensemble size (K) is restricted by the depth of the base network, which may not provide uncertainty estimates of sufficient quality for a given safety- critical application. In addition, we find that increasing K beyond certain point negatively influences the shared base network, leading to degradation in calibration (abla- tion study in Appendix B.4). Furthermore, while QUTE achieves the best calibration on tiny-sized models under severe resource constraints, it requires additional learning layers on large models and datasets. Alternatively, methods that improve early-exit calibration (Jazbec et al., 2024) could potentially improve effectiveness of QUTE's EV-assistance. Interestingly, we observe that excessively increasing EV- assistance, such by raising the depth multiplier in QUTE 's output blocks, negatively impacts accuracy despite im- proving calibration. However, its worth noting that the training times with QUTE are higher compared to other methods (except HYDRA) due to weight transfer between train-batches. Finally, despite QUTE's versaiilitleeffectiee ness in OOD detection, specialized OOD detectors should be considered, especially for large models.
Bai, H., Canal, G., Du, X., Kwon, J., Nowak, R. D., and Li, Y. Feed two birds with one scone: Exploiting wild data for both out-of-distribution generalization and detection. In International Conference on Machine Learning, pp. 1471. PMLR, 2023.
Baier, L.,, Schlör, T., Schöffer, J., and Kühl, N. Detect- ing concept drift with neural network model uncertainty, 2022.
Acknowledgments
Banbury, C. R., Reddi, V. J., Torelli, P., Holleman, J., Jef- fries, N., Király, C., Montino, P., Kanter, D., Ahmed, S., Pau, D., Thakker, U., Torrini, A., Warden, P., Cordaro, J., Guglielmo, G. D., Duarte, J. M., Gibellini, S., Parekh, V. Tran, H., Tran, N., Niu, W., and Xu, X. Mlperf tiny benchmark. CoRR, abs/2106.075977, 2021. URL https: http://axxxxv..raabaa/ 07597.
NSERC and COHESA support gratefully acknowledged.
Impact Statement
This work presents QUTE, framework for efficient un- certainty quantification and model monitoring in field- deployed TinyML models, pushing the boundaries of constrce- machine learning. QUTE supports reliable, real-time decision-making in resource-constraine high-stakes scenarios. While the work primarily aims to improve TinyML reliability, care must be taken to ensure its responsible use in safety-critical applications. In era of abundant compute, where conventional neural networks often operate inefficiently (Patterson et al., 2021), QUTE exemplifies the potential of energy-efficient strategies to reduce unnecessary energy costs.
Bifet, A. and Gavalda, R. Learning from time-changing data with adaptive windowing. In Proceedings of the 2007 SIAM international conference on data mining, pp. 443 448. SIAM, 2007.
Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613-1622. PMLR, 2015.
Bonato, V. and Bouganis, C.-S. Class-specific early exit design methodology for convolutional neural networks. Applied Soft Computing, 107:107316, 2021. ISSN 1568- 4946. doi:
9

---

QUTE: Quantifying Uneertainty in TinyML models with Earty-cxit-assisted ensembles for model-monitoring
URL
com/ sci ast.ace///rttcee///////566446463
https://ww.scieecceddee
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International conference machine learning, pp. 1321-1330. PMLR, 2017.
Brier, G. W. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Havasi, M., Jenatton, R., Fort, S., Liu, J. Z., Snock, J., Lakshminarayanan, B., Dai, A. M., and Tran, D. Training independent subnetworks for robust prediction, 2021.
Charpentier, B., Zügner, D., and Günnemann, S. Posterior network: Uncertainty estimation without ood samples via density-based pseudo-coume counts. Advances in neural information processing systems, 33:1356-1333 1367, 2020.
He. K., Zhang, X., Ren, S., and Sun. J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10. 1119//VP..0006..99
Deng, D., Chen, G., Yu, Y., Liu, F., and Heng, P.-A. Uncer- tainty estimation by fisher information-base evidential deep learning. In International Conference on Machine Learning, pp. 7596-7616. PMLR, 2023.
Heidemann, L.., Schwaiger, A., and Roscher, K. Measuring ensemble diversity and its effects on model robustness. In AlSafery@ IJCAI, 2021.
Deng, J. Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A lcage-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturba- tions. arXiv preprint arXiv: 12261, 2019.
leee, 2009.
Dinari, O. and Freifeld, O. Variational- and metric-based deep latent space for outttf-distribution detection. In Cussens, J. and Zhang, K. (eds.), Proceedings of the Tighth-Eighth Conference on Uncertainty in Artificial Intelligence, volume 180 of Proceedings of Machine Learning Research, pp. 569-578. PMLR, 01-05 Aug 2022. URL https: /ttpss://prrceeee pttpp/eecceeeeeseee mlr. press// v180/dinarisn html.
Hendryeks, D. and Gimpel, baseline for detecting misclassified and out-of-distribution examples in neural networks, 2018.
Hernández-Lobato, J. M. and Adams, R. Probabilistic back- propagation for scalable learning of bayesian neural net- works. In International conference machine learning, pp. 1861-1869, PMLR, 2015.
Ferianc, M. and Rodrigues, M. Mimmo: Multi-inphr mas- sive multi-output neural network. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 4563-4568, June 2023.
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint SEdiv:1704.04800 2017.
Gal, Y. and Ghahramani, Z. Dropout as a bayesian approx- imation: Representing model uncertainty in deep learn- ing. In international conference on machine learning, pp. 1050-1059, PMLR, 2016.
Hsu, Y.-C,, Shen, Y., Jin, H., and Kira, Z. Generalized odin: Detecting nut-of-distribution image without learn- ing from distri-iistti data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10951-10960, 2020.
Gama, J., Medas, P., Castillo, G., and Rodrigues, P. Learning with drift detection, Advances Artificial Intelligence- SBIA 2004: 17th Brazilian Symposium on Artificial In- telligence, Sao Luis, Maranhao, Brazil, September 29. Ocotber 1, 2004. Proceedings 17. pp. 286-295. Springer, 2004.
Huang, G. Chen, D., Li. T., Wu, F., van der Maaten, L., and Weinberger, K. Q. Multi-Scale Dense Networks for Resource Efficient Image Classification. arXiv e-prints, March 2017.
art. arXiv: 1 111117733.9944
Jacger, P. F., Lüth, C. T., Klein, L., and Bungert, T. J. A call to reflect on evaluation practices for failure detection in image classification, 2023. URL org / abs 2211 15259.
Ghanathe, N. P. and Wilton, S. T-recx: Tiny- Tiny-resource effi- cient convolutional neural networks with early-exit. In Proceedings of the 20th ACM International Conference on Computing Frontiers, pp. 123-133, 2023.
Jazbec, M., Allingham, J., Zhang, D., and Nalisnick, E. To- wards anytime classification in early-exit architectures by enforcing conditional monotonicity. Advances in Neural Information Processing Systems, 36, 2024.
Gneiting. T. and Raftery, A. E. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477)359-377, 2007.
10

---

QUTE: Quantifying Uncertainty in TinyML models with Earty-cxit-assisted ensembles for model-monitoring
Jordal, I.
Audiomentations, 2024. URL https:// github httom/vve///aam httm://veer5///dd.emeeeeeee comntatiio. Ver- sion 0.32.0.
Malinin, A. and Gales, M. Predictive uncertainty estima- tion via prior networks. Advances in neural information processing systems, 31, 2018.
Katz-Samuels, J., Nakhleh, J. B., Nowak, R., and Li, Y.
Malinin, A., Mlodozeniec, B., and Gales, M. Ensemble distribution distillation. arXiv preprint arXiv: 2019.
Training ood detectors in their natural habitats, In Inter- national Conference on Machine Learning, pp. 10848- 10865. PMLR, 2022.
Meronen, L., Trapp, M., Pilzer, A., Yang. L.,, and Solin, A. Fixing overconfidence in dynamic neural networks. In Proceedings of the IEEE/CVE winter conference on applications of computer vision, pp. 2680-2690, 2024.
Kaya, Y., Hong, S., and Dumitras, T. Shallow-deep net- works: Understanding and mitigating network overthink- ing. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Pro- ceedings of the 36th International Conference Ma- chine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3301-3310. PMLR, 09-15 Jun 2019. URL https:::: http://pooceeeeeee eggs ww..mlr.ppr. press/ v97 v97/kayal 9a. html.
Mu, N. and Gilmer. J. Mnist-c: A robustness benchmark for computer vision, 2019.
Mukhoti, J., Kirsch, A., van Amersfoort, J., Torr, P. H., and Gal, Y. Deep deterministic uncertainty: A new simple baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24384- 24394, 2023.
Kendall, A. and Gal, Y. What uncertainties do we need in bayesian deep learning for computer vision? In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.). Advances in Neural Information Process- ing Systems, volume 30. Curran Associates, Inc., 2017. URL attps://prcceednggg.ee.. neurips. cc
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
550666986664052555222552222 pdf.
Nixon, J., Dusenberry, M. W., Zhang, L.,, Jerfel, G., and Tran, D. Measuring calibration in deep learning. In CVPR workshops, volume 2, 2019.
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J.. Lakshminarayanan B., and Snoek, J. Can you trust your model's uncertainty? evalu- ating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Sim- ple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the I International Conference on Neural Information Processing Systems, NIPS' 17, pp. 6405 64166, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.- M., Rothchild, D., So, D., Texier, M., and Dean, J. Car- bon emissions and large neural network training, arXiv preprint arXiv:2 10350, 2021.
Le, Y. and Yang, X. Tiny imagenet visual recognition chal- lenge. CS 231N, 7(7):3, 2015.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 86(1) 1):2278-2233 23224, 1998.
Pei, J., Wang, C., and Szarvas, G. Transformer uncertainty estimation with hierarchical stochastic attention. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pp. 11147-11155, 2022.
Liang, S., Li, Y., and Srikant, R. Enhancing the reliability of out-of-distribution image detection in neural networks, 2020.
Qendro, L., Campbell, A., Lio, P., and Mascolo, C. Early exit ensembles for uncertainty quantification. In Machine Learning for Health, pp. 181-195. PMLR, 2021.
Liu, W., Wang, X., Owens, J., and Li, Y. Energy-based out- of-distribution detection. Advances in neural information processing systems, 33:21464-21144
2020.
Rahaman, R. et al. Uncertainty quantification and deep ensembles. Advances in Newral Information Processing Systems, 34:20063-20075, 2021.
L Liu, Z., Zhao, C., landola, F., Lai, C., Tian, Y., Fedorov, I., Xiong, Y., Chang, E., Shi, Y., Krishnamoorthi, R., et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. In Forty-first Interna- tional Conference on Machine Learning, 2024.
Sensoy, M., Kaplan, L.,, and Kandemir, M. Evidential deep learning to quantify classification uncertainty. Advances neural information processing systems, 31, 2018.

---

QUTE: Quantifying Uncertainty in TinyML models with Earty-exit-assisted ensembles for model-monitoring
STMicroelectronniss STM321432KC Datasheet: Ultra-low- power Arm Cortex-M4 32-bit MCU+FPU, 100DMIPS, up to 256KB Flash, 64KB SRAM, USB FS, analog, audio, 2018. URL attps://www. st com/ resource/ htn/dttasheet/stt33333 1e05221432kkc 431.1.....c........ pdf.
Xia, G. and Bouganis, C.-S. Window-based early-exit cas- cades for uncertainty estimation: When deep ensembles are more efficient than single models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17368-17380, 2023.
STMicroelectronics. STM32F76721 Datasheet: ARM Cortex-M7 Microcontroller with 512 KB Flash, 216 MHz CPU, ART Accelerator, FPU, and Chrom-ART Accelerator, 2019. URL https: htttpp::/www../ /www. st. com/
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 708.008800 7774077777 0 2017.
Yang, J., Zhou, K., Li, Y., and Liu, Z. Generalized out-of-distributioo detection: A survey, arXiv preprint arXiv:2110.11334 2021.
Tang, X., Yang, K., Wang, H., Wu, J., Qin, Y., Yu, W. and Cao. D. Prediction rrediction-uncertainty www.uuuinty awware decision-making for autonomous vehicles. IEEE Transactions on Intel- ligent Vehicles, 7(4):849-86) 2022. doi: 10.1109/TIV.
Yang. J., Wang. P., Zou, D., Zhou, Z., Ding, K., Peng, W. Wang, H., Chen, G., Li, B.. Sun, Y., et al. Openood: Benchmarking generalized out-of-distribttion detection. Advances in Neural Information Processing Systems, 35: 32598-326111 2022.
2022.3188662.
Teerapittayanon, S., McDanel, B., and Kung, H. Branchynet: Fast inference via early exiting from deep neural net- works. In 2016 23rd International Conference Pat- tern Recognition (ICPR), pp. 2464-2469, 2016. doi: 10.1
Zaidi, S. and Zela, A. Neural ensemble search for perfor- mant and calibrated predictions. 2020.
Zhang, X.-Y,, Xie, G.-S., Li, X., Mei, T., and Liu, C.-..
Teye, M., Azizpour, H., and Smith, K. Bayesian uncertainty estimation for batch normalized deep networks. In Inter- national conference on machine learning, pp. 4907-4916. PMLR, 2018.
survey on learning to reject. Proceedings of the IEEE, 111 (2):185-215 2023. doi: 10. N0.1009/PRO..2223.. 3238024.
Zhang, Y., Suda, N., Lai, L., and Chandra, V. Hello edge: Keyword spotting on microcontrollers. CoRR,
Tran, L., Veeling, S., Roth, K., Swiatkowski, J., Dillon, J. V., Snoek, J., Mandt, S., Salimans, T., Nowozin, S., and Jenatton, R. Hydra: Preserving ensemble diversity for model distillation. arXiv preprint arXiv:2001.0 2044044000.04644 2020.
abs/1711.07128, 2017. URL http://arxiv.org abs/1711.07128.
Zheng, Y., Chen, Y., Qian, B., Shi, X., Shu, Y., and Chen, J. A review on edge large language models: Design, execution, and applications. ACM Computing Surveys, 57 (8):1-35, 2025.
Van Amersfoort, J., Smith, L., Teh, Y. W., and Gal, Y. Un- certainty estimation using a single deep deterministic neural network. International conference on machine learning, pp. 9690-9700. PMLR, 2020.
2025.
Vargas, J., Alsweiss, S., Toker, O., Razdan, R., and Santos, J. An overview of autonomous vehicles sensors and their vulnerability weather conditions. Sensors, 211rs,,,11111 21(16)55397, 2021.
Warden, P. Speech commands: A dataset for limited- vocabulary speech recognition. CoRR, Abs/1804403329, 2018. URL http://arxiq/ wwt.riiiiiooggs////os// 03209.
Wenzel, F., Snoek, J., Tran, D., and Jenatton, Hyperpa- rameter ensembles for robustness and uncertainty quan- tification. Advances in Neural Information Processing Systems, 13:6514-6553 2020.
Xia, G. and Bouganis, C.-S. Augmenting softmax informa- tion for selective classification with out-of-distribution data. In Proceedings of the Asian Conference on Com- puter Vision, pp. 1995-1999992 2012, 2022.
12

---

QUTE: Quantifying Uncertainty in TinyML models with Earty-cxit-assisted ensembles for model-monitoring
Appendix
A. Training and dataset details
In this section, we describe the details of the models evaluated and specifics of training and the experimental setup.
A.1. BASELINES
BASE: Unmodified implementations of models evaluated.
Monte Carlo Dropout (MCD) (Gal & Ghahramani, 2016): computes mean of prediction vectors from K inference passes by activating dropout. For fairness, dropout and EE locations same. We use dropout rate of 0.1, 0.05, 0.2, 0.2 for MNIST, SpeechCmd, CIFARIO and Tinylmagenet respectively.
Deep Ensembles (DEEP) (Lakshminarayaaann et al., 2017); explicit ensemble of models with same base model architecture but with different random weight initializations. The number of ensemble members = K.
Early-exit Ensembles (EE-ensemble) (Qendro et al., 2021): attaches multiple early-exits with additional fully-connected (FC) layers. The size of the FC layers is chosen such that the model delivers the best uncertainty estimation. All early-exit prediction vectors including that of the final exit are averaged to get the final ensemble prediction. We place the early-exits at the same locations QUTE. Ablation studies with EE-ensemble are presented in Appendix B.5.
HYDRA (Tran et al., 2020): uses the same architecture as QUTE. Instead of early-exit distillation like QUTE, it employs ensemble distillation, capturing unique predictive behavior of individual ensemble members. We include this baseline to assess QUTE's early-exit distillation method against ensemble distillation approaches.
Generalized-ODIN (G-ODIN) (Hsu et al., 2020): a generalized OOD detection method that decomposes the confidence score and introduces& a preprocessing layer that perturbs the inputs for better OOD detection. The perturbation magnitude determined small held-out validation set.
A.2. Datasets
In our evaluations, we use four in-distribution datasets for training all baselines methods we evaluate: 1) MNIST (LeCun et al., 1998), 2) SpeechCommands (Warden, 2018), 3) CIFAR1O (Krizhevsky, 2009) and 4) Tinylmagent (Le & Yang, 2015).
MNIST MNIST is dataset of handwritten digits containing 60,000 grayscale images of size 28 28 and test set of 10,000 images. MNIST contains 10 classes. Although a simple dataset, MNIST reflects the problem sizes typical of many tinyML applications.
SpeeanCommands Speech-Cmd a collection of short audio clips, each spanning 1 second. The dataset consists of utterances for 35 words and is commonly used for benchmarking keyword spotting systems. We train our model (DSCNN) of 35: Down, Go, Left, No. Off. On, Right, Stop. Up. Yes. Thus, the number of classes is 10.
to recognize ten words The audio files in WAV format are preprocessed to compute Mel- Mel-frequency cepstral coefficients (mel-spectograms)) The mel-spectograms are of size 49x 10 with single channel.
CIFARIO CIFARIO dataset consists of 60,000 32x32 rgb images out of which 10,000 images are in the test set. It contains 10 classes and thus 6000 images per class.
Tiny Imagenttt Tinylmagenet is a smaller version of the Imagenet (Deng et al., 2009) dataset containing 200 classes instead of 1000 classes of the original Imagenet. Each class in Tinylmagenet has 500 images in the train and the validation set contains 50 images per class. The size of the images are resized and fixed at 64x64x3.
1. CORRUPTED DATASETS
For image classification tasks, we use the following corrupted versions of ID: 1) MNIST-C (Mu & Gilmer, 2019). 2) CIFAR1O-C and 3) Tinylmagenet-C (Hendrycks & Dietterich, 2019). All corruptions are drawn from 4 major sources: noise, blur, weather and digital. The digital and blur corruptions induce high epistemic uncertainty, whereas, noise corrup- tions induce higher aleatoric uncertainty. contrast, weather corruptions tend to encompass both epistemic and aleatoric components. Most corruptions are systematic transformations of input images, which predominantly introduce epistemic uncertainty. Pure noise corruptions are the primary source of alcatoric uncertainty; however, many corruptions exhibit both
13

---

QUTE:
Quantifying Uneertainty in TinyML models with Earty-cxit-assisted ensembles for model-monitoring
components, often leaning toward epistemic.
The MNIST-C dataset contains 15 corrupted versions of MNIST shot_noise, impulse_noie noise, glass_ fog, spatter, dotted_ line, zigzag, canmy_edse _eggee motion_ shear, scale, rotate, brightness, translate, stripe, identity. All the corruptions are of a fixed severity level.
MNIST-C
CIEARIO-C CIFARIO-C includes 19 different types of corruptions with 5 severity levels which gives us 19x5=95 corrupted versions of CIFAR-10 ID. We use all 95 corrupted versions in our experiments. The list of corruptions are: gaussian noise, brighmess, contrast, defocus blur, elastic, fog, frost, frosted _eaeid_ggs_______eeeeee __ur, gaussian blur, impulse_non noise, jpeg_compression, motion_blee _bur, pixelate, saturate, shot_ shot_noise, snow, spatter, speckle_ noise, soom_blum
TinyImagenet Tinylmagened includes 15 different types of corruptions with 5 severity levels which gives 15x5-75 corrupted versions of ID. We use all 75 corrupted versions in our experiments. The list of corruptions are: gaussian noise, brightness, contrast, defocus babor elastic_ trariiir_raffom.mmo fog, frost, glass _lur, impulse _oise,,__oiieeeeeeeeeee pree__comreesioon, motion_blut pixelate, shot poiot_ snow, zoom blur.
The above corruptions distort key visual features of the input, as outlined below.
Noise corruptions (c.g., gaussian/shot//maaao. gaussialshoo/imppllsssspell noise) introduce random pixel variations, which disrupts the model's pattern recognition capability.
Blur corruptions (e.g., defonnshnotoo/aasssii//ooooooooo glass blur) distort the edges and textures of the objects in the images, affecting spatial feature extraction.
Digital corruptions (e.g.. JPEG compression artifacts, pixelation, contrast shifts, saturation) modify pixel distributions, which impact feature consistency.
Weather corruptions (c.g., fog, frost, snow, rain) obscure key visual components important for feature recognition, reducing visibility and contrast.
SpeechCmd-C For the audio classification task of Keyword spotting on SpeechCmd, we corrupt the audio before converting them into mel-spectro using the audiomentations library (Jordal, 2024) with the following 11 corruptions: gaussian noise, air absorption, band pass filter, band stop filter, high pass filter, tanh distortion, time mask, time stretch.
passfilte filter, high shelf filter, low pass filter, low shelf filter, peaking
A.3. Training details
We evaluate four models in our experiments:1) 1) 4-layer CNN on MNIST, 2) -layer aenghwisesseparabble CNN (DSCNN) on SpeechCmd, 3) Resnet-8 with residual stacks from the MLPerf Tiny benchmark suite (Banbury et al., 2021) on CIFARIO and 4) MobilenetV2 (Howard et al., 2017) on Tinylmagenet.
The 4-layer CNN is trained for 20 epochs with batch size of 256, the DSCNN model is trained for 10 epochs with a batch size of 100, the Resnet-8 model is trained for 200 epochs with batch size of 32 and the MobilenetV2 model is trained for 200 epochs with a batch size of 128. All models are trained with Adam optimizer with momentum of 0.9 and an initial learning rate of 0.001, expect DSCNN on SpeechCmd which uses an initial learning rate of 0.0005. The learning rate decayed by a factor of 0.99 every epoch for all image classification tasks. We follow a step function for SpeechCmd that reduces learning rate by half every epochs. For models with QUTE architecture, we transfer weights from early-exits to all (ffor()) ())) using callback that sets the weights of each fouls with weights copied from the corresponding assisting early-exit at the beginning of each training batch.
A.4. Weighting the loss at EV-exits
As described in Section 4, we weight the loss at EV-exits with higher factor Wav, such that Wgy, WEV--, 8. This is done to further promote diversity across the EV-exits and to vary the influence each EV-exit has on the shared network's weights. Utilizing distinct weighting factors across EV-exits implies that each EV-exit contributes to the final loss calculation with varying degrees of influence. Consequently, this diversity extends to the weight updates of the shared base network, thereby injecting diversity into the classification heads. We empirically set to be 0.5. To determine WgV. we repeat training with WEV, set (2,3, 4,5]. We find that for WEV, > 3, the NLL starts dropping steadily because the higher
14

---

QUTE: Quantifying Uncertainty in TLL models with Exrly-exit-assisted ensembles for model-monitoring
DURING TRAINING
DURING INFERENCE
Input image
Input image
BM33 network
CONV
DCONV 1E-1
EE-1
Resource .hungry EEs
BASE network
CONV
CONV
CONV
DCONV E0-2
EE-2
CONV
Share/oppy fiter weights from to EV
CONV
ECONV
DCONV 2
DCONV
DCONVgv2 2
1 EV- exit-2
Final exit
EV-
EV- exit-1
EV- exit-2
exit-1
hos
DCONV m1-1i hoz = DCONV EE-2
Lightwoight classification hoads
h1
DCONV gvai l = DCONV gv-2
Figure 5: Depiction of QUTE during training and inference. The early-exits (EE) are only used during training for weight- transfer. During inference, the EEs are removed, and only EV-exits retained. We do not include other layers such as dense and pointwise convolution in the figure for brevity.
loss at EV-exits overshadow the losses of the early-exits. This diminishes the influence of early-exits at EV-exits, which is undesirable. Hence, we set WEV, = 3 in all our experiments.
A.5. cerrutted-indiistribe datasets
We construct the carruteddin-distribee (CID) datasets for evaluations as follows. For MNIST-C with a fixed severity level, we append each corrupted version of MNIST to the MNIST-ID dataset to create 15 ID+CID datasets. For CIFARIO-C and Tinylmagenet-C, which have 19 and 15 different corruptions respectively with 5 severity levels each, we construct the corrupted datasets as follows. For each type of corruption, we randomly select p samples from each of the severity levels. Next, we concatenate all these samples to create a new corrupted dataset of size x p. p is selected such that x p size of ID dataset. This process is repeated for all corruptions. The datasets thus obtained contain samples from all severity levels. For example, for CIFAR1O-C which consists of 19 corruptions, this process yields 19 ID+CII datasets.
A.6. dccuracy-drop detection experiments
For experiments to detect accuracy drop/CID detection described in Section 6.2, we append the ID dataset with each corrupted dataset obtained from the methodology described above. For example, for experiments with CIFARIO-C, we obtain 19 ID+CID datasets that contains both ID samples and corrupted samples (from all severity levels). Next, described in Section 5, for each prior work we evaluate, we first iterate over only ID) and obtain predictions from the model while computing the moving average of accuracy of the past m predictions using a sliding-window. In this way, we obtain the accuracy distribution of the blddnggwindow And on ID, and then compute its mean JTD and Handard-deviation IID- The accuracy of the bidding-window at any given point of time is denoted as Asw. Next, we iterate over all ID+CID datasets while computing the moving average of confidence of the past m predictions using a sliding-windee window (Csw). We record all instances of Csw dropping below certain threshold p. These events are denoted as CID--redicted events. At the same time, we also compute Asw for evaluating how many CIDD-predcctd events actually Citeedetected event (true positive). Finally, the description of true positive (TP), false positives (FP), false negatives (FN) and true negatives (TN) are as follows.
-IDDID True positive: Csw < p and Asw S PID -3 OID
False positive: Csw < p and Asw > PID 3 FID
True negative: Csw > p and Asw PID 3 01D
False negative: Csw > p and Asw S PTD 3-DID
15

---

QUTE: Quantifying Uncertainty in TinyML models with Earty-cxit-assisted ensembles for model-monitoring
Model
F1 (1)
BS (1)
NLL (4)
MNIST
BASE-TS
0.937.10.002
0.009-10.000
0.203:10.005
QUTE-TS
0.942.10.001
0.008
0.191-10.005
QUTE
0.941110.004
0.009:1:0000
0.199:10.010
Sp
BASE-TS
0.923
0.009:10.000
0.229:10.017
QUTE-TS
0.926110.006 0.009:10.000
0.233-10.010
QUTE
0.933
0.008
0.202=100016
CIFARIO
BASE-TS
0.834+00000
0.023++0.000
0.493++0.000
QUTE-TS
0.8022
0.021+000000
0,444++0.013
QUTE
0.85811.001
0.020-1.0000
0.428=1.019
Table 4: Calibration with temperature scaling (TS) on ID
AUPRC (1)
MNIST-C
SpeechCmd-C
CIFAR10-C (Severity to 5) 0.30, 0.42, 0.38, 0.50, 0.61 0.26, 0.45, 0.51, 0,65, 0.76
BASE-TS
0.47
0.52
QUTE-TS
0.52
0.53
QUTE
0.63
0.61
0.30, 0.47, 0.52, 0.65, 0.78
Table 5: Accuracy-die detection with semperature-scaling
In this way, we collect the TP, FP. TN, FN from each ID+CID datasets and compute the average precision and recall for a given threshold p. Finally, we report the AUPRC. We choose AUPRC because the precision-recall curve is resistant to imbalance in datasets.
B. Ablation Studies and Additional Results
B.1. Comparison with Temperature scaling
Temperature scaling (TS) (Guo et al., 2017) is simple post-training calibration technique that applies a linear scaling factor to logits (pre-softmax) to mitigate model overconfidence. Table 4 and Table 5 present the calibration and accuracy- drop detection results, respectively, for TS versions of BASE (BASE-TS) and QUTE (QUTE-TS), For QUTE-TS, the same architecture as QUTE is used, but without weight transfer. We employed the best- best-performing pool-then-cale than-calibbatee configuration from (Rahaman et al., 2021), which applies temperature scaling to ensembles to obtain QUTE-TS, As the results show, QUTE consistently outperforms BASE-TS on both calibration and accuracy drop detection. However, despite improving calibration some scenarios, QUTE-TS performs worse than QUTE on accuracy-drop detection, particularly with tiny datasets such as MNIST-C and SpeechCmd-C. A deeper analysis revealed that QUTE-TS tends be overconfident for certain corruptions. For instance, under fog corruption on MNIST-C, QUTE-TS exhibits an overconfident behavior with a median confidence of 0.99, while QUTE maintains a much lower confidence of 0.51, leading to superior accuracy-drop detection capability.
B.2. Comparison with single-pase pass deterministic methods
Related work: Several single-pass (non-Bayesinn methods for UQ (Van Amersfoort et al., 2020; Mukhoti et al., 2023; Sensoy et al., 2018; Deng et al., 2023) offer lower memory footprints compared to ensemble methods. However, they remain impractical for tinyML due to specialized output layers or architectural constraints. DUQ (Van Amersfoort et al., 2020) adds a specialized layer post-softmax, drastically increasing resource use (e.g., 10x more parameters for Resnet on CIFAR 10). DDU (Mukhoti et al., 2023) simplifies this but relies on residual connections for feature space regularization, limiting its applicability. Prioretts (Malinin & Gales, 2018) require OOD data, which is often unrealistic. Postnets (Charpentier et al., 2020), though OOD-free, focuses primarily on OOD detection, often sacrificing accuracy below the base network's level. Other methods like (Meronen et al., 2024) that address overconfidence in early-exit networks require excessive computational resources (e.g., laplace approximations).
Comparison with Postnets Postnets (PostN) (Charpentier et al., 2020) is the most relevant single-pas pass deterministic method compared to QUTE, which uses normalizing flows to predict posterior distribution of any input sample with no additional
16

---

QUTE: Quantifying Uncertainty in TinyML models with Earty-cxit-assisted ensembles for model-monitoring
Model
F1 (f)
(L)
NLL (4)
MNIST-PostN
0,92
0.012
0.286
MNIST-QUTE
0.94
0.009
0.199
CIFAR-PostN
0.84
0.022
0.462
LIFAR-QUTE
0.858
0.020
0.428
Table 6: Calibration comparison with Postnets on ID
Ff
(L)
NL)
Model
EV
Bo-EV
EV
Bo-EV
EV
Bo-EV
CIFRRI
0.858
0.85
0.020 0.225
0.020600.000
0.435:00007 3.81311.105
0,05
Tinylnngentt
0.38010.0111
0.35100004
L..7-5
4.743.0.111
Table Effectiveness of early-view assistance. Results averaged over 3 independent training runs.
memory overhead. We evaluate PostN on MNIST and CIFARIO by substituting the encoder network architectures with the ones used in our evaluations. We use identical hyperparameter settings of Charpentier et al. (2020) and train for same number of epochs as in our evaluations but with early stopping. Table 6 reports the calibration metrics for PostN and QUTE, Since PostN don't 't necessarily focus on accuracy, we found that MNIST-PostN needed a 50% increase number of epochs even surpass the accuracy of MNIST-BASE. We report this result. For CIFARIO, we found that early stopping halted the training, and report those results. As seen, QUTE consistently outperforms PostN all metries on both datasets, making QUTE more efficient choice for resource-constrrained environments.
B.3. Effectiveness of EV-assistance method and its effect convergence
To investigate the effect of the early-view (EV) assistance method ensemble quality, train non-EV versions of QUTE i... the same architecture (with early-exits) except without the weight transfer mechanism. Table reports the calibration metrics for both EV and non-EV versions for CIFARIO and Tinylmagenet. As shown, EV-assistance has massive influence on improving calibration, and improves the network accuracy on TinyImagenet by 8%. Also, we find that the ability of non-EV versions to detect accuracy-drop events reduces by 15% on average compared to EV-versions. These results show that the notably lower NLL obtained with EV-assistance (5% lower on CIFARIO and 19% lower on TinyImagenet) not only improves uncertainty estimates, but it also proves crucial in both CID and OOD detection. Furthermore, we conducted an ablation study to isolate the impact of early-exit (EE) knowledge transfer ensemble diversity. We quantify QUTE's ensemble diversity using the normalized disagreement (ND) metric from Heidemann et al. (2021), evaluated with and without EV-assistance (higher is better). On CIFAR10-ID. the ND drops by 71% without EV-assistance; MNIST-ID, it drops by 19%. These results indicate that EE knowledge transfer significantly enhances ensemble diversity, which in turn contributes to better calibration.
In addition, we investigate the effect of weight transfer on model convergence. Figure plots the loss after each train-batch one of the EV-blocks of Resnet-8 on CIFARIO. As seen, model convergence is not affected because the weight transfer mechanism is designed such that the loss at EV-exit is always computed with respect to the copied weights (see Section 4).
5
w/ weight transfer
4-
w/o weight transfer
Train batch Loss
3
2
OK
50K 100K 150K 200K 250K 300K Train batch Number
Figure 6: Batch-loss at one of the EV-exits of Resnet-8 on CIFARIO.
17

---

QUTE: Quantifying Uneertainty in TinyML models with Earty-cxit-assisted ensembles for model-monitoring
In essence, EV-assistance uses the EE-weights extract and reemphasize the EE features at the final exit(s).
B.4. Uncertainty quantification vs Ensemble size
We conduct an ablation study to investigate the effect of number of early-exits on uncertainty estimation quality. The ensemble size |K] is an hyperparameter that depends on the compuaation/resoure budget, and it is bounded above by the depth of the base network. We vary the ensemble size and investigate its effect calibration quality in MobilenetV2. Figure 7 shows the effect on accuracy and NLL in MobilnetV2 for [ ranging from 2-10. The red line shows the accuracy
0.56
5.5
0.37
0:36
0:30
3.5
Easemble Size (IKT)
Easemble Size (IC))
(a) Accuracy (f)
(b) NLL (L)
Figure 7: Effect of Ensemble size on Accuracy and NLL in MobilenetV2 on Tinylmagenet. Red dotted horizontal line indicates accuracy and NLL of BASE in respective plots.
and NLL of base network in respective plots. As shown, the accuracy does not always improve with ensemble size. This demonstrates that there is limit to the improvement in accuracy due early-exit ensembling. On the other hand, NLL steadily improves as ensemble size increases with the least NLL obtained for |K] 8. For |AC] = 10, the NLL rises slightly. Our investigations revealed that since the base network, carly-exits and the carly-view (EV) exits are part of the same network architecture, and are trained simultaneously, the disruption in base network's 's weights due to the losses of early-exits and EV-exits reaches a tipping- point after a certain ensemble size thereby, causing a drop off in performance.
B.5. Ablations with EE-ensemble
B.5.1. EXCLUDING FINAL EXIT FROM FINAL PREDICTION
As described in Section 4, during inference, we exclude the original output block of the base network when computing the final prediction vector for QUTE because we find it that is overconfident and harms calibration quality. However, we include the original output block's ' prediction vector along with that of of all early-exits for computing the final prediction vector for EE-ennebbl,, consistent with the original paper Qendro et al. (2021). We conduct an ablation study on CIFAR10 using Resnet-8, where like QUTE, exclude the original output block's 's prediction vector when computing the final prediction vector. Table 8 reports the calibration metries on CIFAR IO-ID for two configurations: 1) including original output block in computing final prediction vector and 2) excluding original output block in computing final prediction vector.
Including final-exit
Excluding final-exit
Resnet-8 CIFARTO
F1 (f) BS (L) NLL (L)
F1 (1)
BS (L)
NLL (L)
EE-ensemble
0.854
0.021
0.446
0.818
0.0026
0.561
Table 8: Calibration Metrics of EE-ensemble for Resnet-8 on CIFAR10 1FFAR10-ID computed including original output block and excluding original output block from computation of final prediction vector.
As seen, the removal of original output block negatively impacts EE-ensemble leading to poor accuracy and calibration. Therefore, we include the original output block in computing the final prediction vector in our main results for EE-ensemble.
B.5.2. USING QUTE'S S OUTPUT BLOCK ARCHITECTURE FOR EARLY-EXITS IN EE-ENSEMBLE
Qendro et al. (2021) adds additional fully- layers at the early-exit for EE-ensemble to match the learning capacities of all exits, unlike QUTE, which uses a single depthwise CONV layer for its output blocks. However, this leads to a
18

---

QUTE: Quantifying Uneertainty in TinyML models with Earty-exit-assisted ensembles for model-monitoring
large overhead (Section 6.1). In this ablation study, show that the additional layers at the early-exit necessary for ElRennsemle achieve good accuracy and calibration. We replace the resource-hungry early-exits of EE-ensemble with the architecture of QUTE's output blocks.
In-distribution
Resnet-8
(1)
BS
NLL
FT
BS (J)
NLL
CIFAR10
TiE-ensemble w/ additional layers
0.85
0.021
0,446
0.64
0.0049 0.0058
1.256
En-emmmmll w/o additional layers
0.78
0.031
0.661
0.57
1.459
Table 9: Calibration Metrics for Resnet-8 on CIFARIO-ID computed for EE-ensemble with additional FC layers and without additional FC layers
Table 9 reports the calibration metrics for two configurations: 1) EE-ensemble with additional FC layers at the early-exit and 2) EE-ensemble with QUTE's lightweight output block architecture at the early-exits. As seen, the configuration with extra learning layers clearly outperforms the one without on both ID and CID terms of accuracy and calibration. The carly-exits placed at different depths work with much less information compared to the final exit. As a result, the accuracy and calibration invariably degrade if the learning capacities of all early-exits do not match that of the final exit.
B.6. QUTE with deeper models
To demonstrate the scalability of our proposed method, we apply QUTE to a traditional Resnet50 (He et al., 2016), a much larger model with higher learning capacity. We train Resnet50 on Tinylmagene for epochs with batch size of 128. The rest of the training methodology is the same as described in Section 4. We evaluate its accuracy-drop detection capability on Tiny iinylmagenet-coruppee... Figure plots the average AUPRC over all corruptions for five severity levels.
0.8
0.6
BASE
BE: BE-enseable
0.4
0.2
Severity
Figure 8: Accuracy-drop detection results (AUPRC) for Resnet50 Tinylmagenet-C.
As seen, QUTE comfortably outperforms EE-ensemble and BASE on accuracy-drop detection, especially showing remarkable 19% improvement the highest severity level compared to EF-ensemble. This illustrates QUTE's superior accuracy-drop detection capabilities across datasets/models of varying complexities, providing a highly-effecttve and cost-efficient accuracy monitoring mechanism.
C. MCU results and Further Discussion
Tables 10 and 1l reports the on-device results for Big-MCU and Small-MCU respectively. report the accuracy, code-size and the latency of each method. We also report the peak random- mudom-access-meen -ccesssmmcsssery (RAM) usage of all methods when deployed BCC (Table 10).
On Big-MCU, all methods fit on the device. The low feoory-footprint of QUTE allows it reduce the code-size consistently across datasets, leading to lower latency and, consequently, reduced energy prediction. In contrast, EE-ensemble has higher memory requirements due to the expensive dense layers at its early exits. Furthermore, with no parallel execution capabilities on an MCU, the latency of DEEP with two ensemble members takes roughly twice as long to execute compared BASE. This issue could worsen significantly with a larger ensemble size.
19

---

QUTE: Quantifying Uncertainty in TinyML models with Exrty-exit-assisted ensembles for model-
SpeechCmd
CIFARIO
Model
Accuracy
Size (KB)
Latency (ms)
RAM (KB)
Accuracy
Size (KB)
Latency (ms)
RAM (KB)
BASE
0.92
294.5
22.7
79.3
0.83
344.4
58.04
78.5
-DEP
0.93
321.6
45.4
86.2
0.88
423,4
116.2
84,2
-ER-enemmlle
0.92
361.5
51.7
92.9
0.85
735.5
65.4
88.2
QUTE
0.93
297.8
24.07
81.7
0.85
355.3
59.6
81.1
Table 10: Microcontroller results on Big-MCU (STM32F7672I)
SpeechCmd Size (KB)
CIFARIO
MNIST Size (KB)
Model
Accuracy
Latency (ms)
Accuracy
Size (KB)
Latency (ms)
Accuracy
Latency (ms)
BASE
0.92
84.8
160.7
0.83
187.5
291.2
0,906
100.3
5.01
DEEP
0.93
111.3
321.2
DNF/ /OOM
0.93
109.3
9.3
-EE-ensemble
0.92
157.7
376.3
DNE/OOM
0.93
113.5
23.7
-QUTE
0.93
93
173.1
0.85
201
298.05
0.92
108.2
5.8
Table 11: Microcontroller results on Small-MCU (STM32L432KC), DNE /OM indicates did-not- fidd-ro-filluu-----mmmrr
We observe a similar trend on the Small-MCU, with QUTE providing the best uncertainty quantification (UQ) per unit of resource used. However, both EF-ensemble and DEEP exceed the available 256KB of cead-onlymemooy (ROM//Flahh SMall-MCU, making them unsuitable for ultra-low-power devices. In such scenarios, QUTE emerges as the only practical solution for effective UQ. Furthermore, we observe an interesting trend on the MNIST dataset. When using 4-convolution layer model (4-layer CONV), the code sizes of all methods are quite similar, even exceeding that of the SpeechCmd BASE model, which employs a 4-layer depthwise-separabhl convolutional model (4-layer DSCNN) with 20,000 more parameters. deeper analysis of the MAP files, which detail memory allocation, reveals that MNIST uses standard convolutions instead of separable convolutions. Therefore, the memory required for code significantly higher for the 4-layer CONV model compared to the 4-layer DSCNN. In fact, the memory required for operators and code in the 4-layer CONV model is four times greater than the memory needed to store its weights.
In summary, these results highlight the efficiency of QUTE. While other methods may struggle with memory limitations, QUTE manages to deliver effective UQ without compromising on resource utilization. Its design allows for optimal performance even in constrained environments, making it a compelling choice for applications where both accuracy and efficiency are of paramount importance.
C.1. QUTE from a system's perspective
The QUTE architecture modifies the computational structure of the model during both training and inference. During training, the addition of early-exits and extra classification heads increases the computational and memory demands. However, since we remove the early-exits during inferencee/dployymen, retaining only the ligtweight classification heads (made up of single depthwise CONV layer), it significantly reduces the computational and memory burden the low-power MCUs in deployment. The removal of early-exits also alleviates potential memory bottlenecks, particularly on devices with limited RAM.
Moreover, because all lightweight classification heads in QUTE share the output of the final CONV block as their input, the increase in peak RAM usage is minimal compared to other baselines, as shown Table 10. For example, QUTE exhibits just 7% increase in peak RAM usage, which is lower than the 12% average increase observed in the HE-ensemble approach because EE-ensemble requires storing of intermediate feature maps in RAM during early-exit computation. Additionally, QUTE requires only single forward pass replicate the behavior of ensemble, making it far more compute- and memory-efficient than traditional ensemble methods.
Importantly, the QUTE architecture eliminates variability in prediction timing, unlike some prior methods such as cascaded deep ensembles (Xia & Bouganis, 2023). This predictability is crucial for tinyML systems, which often serve as the first component in larger ML pipelines interacting with the real world. Consistent output timing simplifies coordination with downstream tasks and ensures compatibility with systems requiring fixed output intervals.
C.2. Applicability to Transformers and Language models
Transformer-bast language and vision models have revolutionized natural language processing and computer vision, and there have been efforts to deploy them on-device in edge scenarios (Liu et al., 2024). However, the compute and memory
20

---

QUTE: Quantifying Uneertainty in TinyML models with Earty-cxit-assisted ensembles for model-monitoring
limitations of resource-constraineed edge devices still present significant challenges for deployment in the real-world (Zheng al., 2025). Moreover, uncertainty quantification in transformer architectures is also essential when deployed at the edge but lacks mainstream attention (Pei et al., 2022). The conventional methods for uncertainty quantification (UQ) such as MCD (Gal& & Ghahramani, 2016) and ensembles (Lakshminarayanall et al., 2017) are still applicable to transformer architectures. In such scenarios, it is imperative to reduce the overhead of UQ mechanism for transformer models when deployed on tinyML or edge devices, especially when the base transformer network already incurs a high resource overhead. The QUTE architecture and the EV-assistance method can be adapted and tailored for transformer architectures. For example, before transferring knowledge from the shallow layers the deeper layers, an auxiliary model could be added to act as intermediary for knowledge transfer. This could potentially mitigate the structural and semantic differences between the early-exit/hhallw layers and the final layers. We leave this exploration as part of the future work.
D. E Evaluation metrics
We evaluate uncertainty using the following metrics.
Brier Score (lower is better): It is proper scoring rule that measures the accuracy of predictive probabilities. Incorrect predictions with high predictive confidence are penalized less by BS. Thus, BS is less sensitive corruptions and incorrect predictions. Therefore, NLL is better measure of uncertainty to compare with other methods.
BS
Σ
(po( 1))
(4)
N
(E(1.2...)
where, 1l is an identity function and L is the number of classes.
Negative log-likelihood (lower is better): It is a proper scoring rule that measures how probable it is that the predictions obtained are from the in-distribution set. It depends on both the uncertainty (predictive confidence) and the accuracy of the predictions.
NLL =
1(y 1) -loogpp [[x) + (1 1 1)) log(1 p(u = [[x))
(5)
LE
Expected Calibration Error (ECE) (lower better): ECE measures the absolute difference between predicted confidence and actual accuracy across different confidence intervals. The confidence is in the range [0,1]. ECE divides the confidence range into M intervals/bins of size 1 and computes the bin accuracies and confidence of each bin before averaging them to provide the final ECE score.
Bml |aco(Bm)) f (Bm)
ECE
(6)
where, N total number of predictions, Bm the mh bin spanning the interval ( (m=-1 m and indicates less disparity between confidence and accuracy across all intervals. However, the ECE score has several limitations and susceptibilities.
M ). low ECE score is desirable,
Limitations of ECE
1) ECE divides predicted probabilities into discrete bins. However, there might be a severe imbalance of samples across the bins because neural networks tend to always predict with high confidence (Nixon et al., 2019). This causes only a few bins (concentrated towards the high confidence region) to contribute the most to ECE. In addition, ECE is also sensitive to the bin boundaries i.e., the number of bins.
2) ECE does not differentiate between specific types of miscalibration such as overonfidme/mmeerroiiinn. often providing simplistic and (overly) optimistic view of the calibration quality. Therefore, ECE is not proper scoring metrie. A model might have a lower ECE without genuinely having good calibration.
For these reasons, we do not emphasize ECE in our main results as they are not indicative of the model's true calibration quality. However, it is worth noting that there are improved versions the ECE metric such as adaptive calibration error (ACE) and static calibration error (SCE) (Nixon et al., 2019). We focus more on proper scoring metrics like NLL and BS in our main results.
21


Reviews:
[{"review_id": "Xfo1TUoada", "reviewer": "Reviewer_XBf5", "summary": "In this paper, the authors propose a novel method for uncertainty estimation in low-resource configurations. Specifically, the method builds on the well-known early-exit approach—where the model produces multiple predictions as the prediction depth increases and then combines them in an ensemble-like manner—but modifies it to be more efficient for TinyML applications while maintaining high uncertainty quality. To achieve this, the authors train the early-exit blocks with a set of ensemble heads applied at the end of the model. However, during inference, they completely discard the early exits, retaining only the ensemble heads while reusing the early-exit blocks' weights for these ensembles. The authors compare their method against several popular uncertainty estimation approaches on widely used benchmarks and demonstrate that it provides high-quality uncertainty estimates while being significantly faster than other alternatives.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and insightful comments. We are glad to hear that you appreciate the novelty of our approach and recognize the thorough evaluation of the proposed method.\n\n> experiments on larger datasets would be also appreciated\n\nTo demonstrate the scalability of our proposed method, we apply QUTE to a traditional Resnet50, a much larger model with higher learning capacity. We train it on TinyImagenet for 50 epochs with a batch size of 128. The rest of the training methodology is the same as described in the paper.\nWe evaluate its accuracy-drop detection capability on TinyImagenet-corrupted. We report the average AUPRC over all corruptions for five severity levels below.\n\n| Method | Sev-1 | Sev-2 | Sev-3 | Sev-4 | Sev-5 | \n| :---: | :---: |  :---: | :---: |  :---: |  :---:  |\n| BASE | 0.26 | 0.54 |  0.70 | 0.67 |  0.61 | \n| EE-ensemble |  0.32 | 0.60 |  0.73 | 0.67 | 0.66 |\n| QUTE | **0.39** | **0.63** |  **0.79** | **0.75** | **0.79** |\n\nAs seen, QUTE comfortably outperforms EE-ensemble and BASE on accuracy-drop detection, especially showing a remarkable 19% improvement at the highest severity level compared to EE-ensemble. This illustrates QUTE's superior accuracy-drop detection capabilities across datasets/models of varying complexities, providing a highly-effective and cost-efficient accuracy monitoring mechanism. We will include this result in the final version of the paper.", "claims_and_evidence": "The authors make several claims about the proposed method, including increased efficiency (in terms of both latency and model size), improved calibration, and higher uncertainty quality of the predictions. They adequately justify these claims by discussing their motivation and supporting them with experimental results in the experimental section, along with extensive ablation studies.", "methods_and_evaluation_criteria": "The choice of baselines (ensemble methods) and benchmarks (MNIST, CIFAR, TinyImageNet, etc.) is appropriate and accurately reflects the current state of research in this area. The evaluations are rigorous and effectively demonstrate the approach's effectiveness from multiple perspectives.", "theoretical_claims": "No theoretical claims, theorems, or proofs are presented in the paper.", "experimental_designs_or_analyses": "The experimental design adequately covers various aspects of the proposed approach, including in-distribution prediction accuracy, generalization to corrupted data, calibration, and out-of-distribution detection quality. The rigorous ablation studies are highly appreciated and provide strong support for the method's claims. Furthermore, the comparison with several popular uncertainty estimation methods further reinforces the case for its effectiveness.", "relation_to_broader_scientific_literature": "The paper positions itself as a contribution specifically for low-latency edge applications, where uncertainty estimation is crucial despite limited computational resources. While the idea of early exits has been explored in the uncertainty estimation literature—for example, by Antoran et al. (2020)—the proposed method refines this concept to make it suitable for low-resource settings, which is one of its key contributions. In addition to demonstrating its efficiency, the authors also show its effectiveness compared to standard baselines in the field of uncertainty estimation.", "essential_references_not_discussed": "No essential references are missing.", "other_strengths_and_weaknesses": "In short, \"Strengths\" of the paper are: \n\n* The proposed method adapts early-exit strategies to improve efficiency in TinyML applications while maintaining high uncertainty quality. By reusing early-exit block weights for ensemble heads, it significantly reduces computational overhead during inference. It's an original idea which has interesting application and significance. \n\n* The paper thoroughly evaluates the method on widely used benchmarks, including MNIST, CIFAR, and TinyImageNet. Extensive ablation studies further support the claims, demonstrating strong performance across multiple uncertainty estimation tasks. Though, experiments on larger datasets would be also appreciated.\n\n* The method is specifically designed for edge applications where uncertainty estimation might essential (for example, AV context) but computational resources are limited. Its efficiency and effectiveness against standard baselines might make it a valuable contribution to real-world deployment scenarios.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zK4Er5KRrs", "reviewer": "Reviewer_eK42", "summary": "**Problem**\n- This paper focuses on uncertainty quantification (UQ) for TinyML models that are specifically designed to operate on microcontrollers with extremely limited memory and computational resources. \n\n**Method**\n- The authors propose QUTE, which combines ideas from early-exit (EE) ensembles and multi-head ensembles.\n- During training:\n  - For the selected intermediates layer of the neural network, the model trains early-exit branches (early-exit members) to predict the labels. At the same time, corresponding to each early exit, a separate lightweight classification head is also trained at the final layer (final-exit).\n  - The parameters from each early-exit head are partially copied to their respective final-exit classification heads, ensuring diversity among the final-exit heads.\n\n- During inference:\n  - All early-exit branches are completely removed, eliminating their memory and latency overhead.\n  - The model uses a single forward pass to reach the final layer, then computes predictions using the multiple lightweight classification heads at the final exit.\n  - The predictions from these diverse final-exit heads form an ensemble used to quantify uncertainty efficiently.\n\n**Experiments**\n- Experiments show that QUTE achieves comparable UQ performance with smaller model sizes and lower latency.\n\n\n## update after rebuttal\nAfter reading the author's response, I have no further concerns and will maintain the current score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We’re delighted that you found our contribution both novel and clear.\n\n> on temperature scaling results\n\nWe apply temperature scaling to BASE and QUTE (following [1] ). The results are included in Appendix B.1. (Tables 4 and 5).  \n\n> on absence of statistical significance tests; Experimental results for failure prediction and accuracy-drop detection are reported based on single runs [...]\n\nWe report metrics such as AUPRC and AUROC for accuracy-drop and failure detection, calculated by averaging precision, recall, true-positive rate, and false-positive rate values across multiple corruptions before deriving the final score. This approach provides a robust measure of model performance under varying conditions. However, while it offers valuable insights, it does not facilitate statistical significance testing across runs. We recognize the importance of statistical validation and plan to conduct independent evaluations in the final version.\n\n> Performance gains are smaller than the reported variance (Table 3)\n\nAs suggested, we will revise the claim to 'QUTE’s performance for uncertainty quantification is comparable to existing methods.'\n\n> For the EE-ensemble method, if we sequentially compute each early-exit's result without storing intermediate features (thus trading time for memory [...]\n\nEven when computing each early exit (EE) sequentially in the EE-ensemble, intermediate feature maps at the EE location still need to be stored. This is because the early exit point involves two branches of computation: the early-exit branch and the rest of the neural network. Since MCUs process sequentially, the intermediate feature maps at this location must be temporarily stored in the random-access-memory (RAM) to enable processing of the rest of the network after the EE outputs are computed. On the other hand, if intermediate feature maps are not stored then the whole network up till the EE has to be recomputed every time, leading to significant latency increase. \nNonetheless, the memory required to store the intermediate feature maps is nominal. Appendix C has more details.\n\n> In the right column of line 303, the authors state: “On Big-MCU, QUTE achieves 31% and 47% latency reductions over EE-ensemble and DEEP [...] \n\nThis question can be broken down into three sub-questions that we answer here.\n\n\n*Why EE-ensemble has more memory overhead compared to DEEP?*\n\nAs described in Section 3 and Appendix B.5.1, EE-ensemble method requires additional fully-connected (FC) layers at each EE to ensure that the learning capacities of all EEs closely match that of the final exit. Failure to do so will result in suboptimal results (Table 8). However, the FC layers are very parameter-heavy resulting in a significant increase in model parameters and thus model size. Furthermore, in the small models we evaluate (e.g., DSCNN, Resnet-8), which have only a few thousand parameters, the memory overhead of the FC layers of EE-ensemble is significantly greater compared to the base network size.\n\n*Why disparity in latency reduction?*\n\nWe use two ensemble members for DSCNN and Resnet-8.  For DEEP, the MCU has no parallel computation capability, hence, each ensemble member has to be computed sequentially. Therefore, for two ensemble members, the latency the doubles. On the other hand, the latency for EE-ensemble is less compared to DEEP despite having higher memory overhead because EE-ensemble only has the computation overhead of a FC layer at each early-exit. Therefore single-forward-pass methods like EE-ensemble (and QUTE) are always more compute efficient compared to DEEP, and this latency gap will become more prominent for larger ensemble sizes. Appendix C provides a more detailed analysis.\n\n*If computed sequentially, shouldn't the memory consumption of DEEP be lower?*\n\nThe memory numbers (size) reported in Figure3 is the *code-size* that reflects the amount of flash memory occupied by the program's code (instructions) and data (model parameters). This differs from RAM, which stores intermediate variables during execution. In DEEP, ensemble members are computed sequentially, avoiding extra storage for intermediate feature maps. Consequently, DEEP's peak RAM usage is consistently 4% lower than EE-ensemble (reported in Tables 10 & 11, Appendix C). However, *code-size* remains unchanged regardless of execution mode, as each ensemble member requires unique model parameters to be stored in the flash memory.\n\n[1] Rahaman et al., 2021: Uncertainty Quantification and Deep Ensembles NeurIPS 2021", "claims_and_evidence": "There are some experimental claims that I am not totally convinced:\n\n- **Limited experimental repetitions**: Table 1 and Figure 3/4 are based on a single experimental run. Repeated experiments would enhance confidence in these claims.\n\n- **Performance gains are smaller than the reported variance (Table 3)**: It is difficult to tell that the proposed methods outperform other methods since the variance is larger the the performance gain. \n\nNevertheless, considering the substantial benefits in memory and latency efficiency, I still regard the proposed method as preferable, particularly in resource-constrained TinyML scenarios.", "methods_and_evaluation_criteria": "**Method**\n\nThe proposed method is **well-suited** for addressing uncertainty quantification (UQ) in TinyML settings. To remove the memory consumption of existing ensemble methods (need multiple copies of the model, including the early-exit models), the proposed methods only use the early-exit members during the training stage. Unlike other distillation methods that distill the early-exit ensembles into one classification head (this type of method is usually suboptimal), the author also uses multiple classification heads at the final layer, each one inherits some of the parameters from the early-exit heads trained during the training stage. In summary, it balances the performance and also the memory consumption.\n\n**Evaluation**\n\nThe evaluation strategy of the paper **follows the standard evaluation settings**:\n\n- Resource Constraints: The experiments explicitly test model deployments on microcontrollers (MCUs) of varying memory and computational limitations, reflecting realistic TinyML scenarios.\n- Datasets: \n  - standard datasets including MNIST, CIFAR10, SpeechCommands, TinyImagenet, \n  - corrupted-in-distribution (CID) datasets for failure prediction\n  - out-of-distribution (OOD) datasets.\n- Metrics: \n  - Brier Score, Negative Log-Likelihood, Expected Calibration Error for calibration performance\n  - AUPRC and AUROC for failure prediction and OOD detection.\n- Baselines: standard ensemble-based methods including Monte Carlo Dropout, Deep Ensembles, EE-ensemble, and HYDRA", "theoretical_claims": "There is no theoretical claim in the paper.", "experimental_designs_or_analyses": "Pros:\nThe overall experimental design in this paper is sound and valid:\n1. The datasets used are representative. However, **more challenging benchmarks for OOD evaluation** could further strengthen the validity of the analysis.\n2. The selection of models is good.\n3. The baseline methods cover most of the common methods. It would be better if the author could **include more deterministic types of baselines such as temperature scaling**.\n4. The evaluation metrics employed are comprehensive and standard.\n\nCons:\n1. Experimental results for failure prediction and accuracy-drop detection are reported based on single runs, without repeated trials or statistical summaries (e.g., means and standard deviations), potentially limiting their reliability.\n2. Calibration results (e.g., Table 3) show that the proposed method's advantages are not clearly demonstrated, especially when considering the reported standard deviations. Thus, it might be more accurate to state that the performance of the proposed method is comparable to existing methods rather than definitively superior.", "relation_to_broader_scientific_literature": "In my understanding, the key contribution of this paper is that: focusing on the TinyML deployment scenarios, it cleverly ccombines the ideas from early-exit ensembles (EE-ensembles) and multi-head ensembles, to maintain the good performance and reduce the memory and latency consumption.\n\nPrior work on EE-ensembles typically faces trade-offs in terms of either memory overhead (due to saving intermediate layer outputs) or compromised latency (due to sequential computation). On the other hand, ensemble distillation is suboptimal since it is very challenging to distill the whole distribution into one classification head.  \nThe authors introduce a training scheme where each early-exit member is associated with a distinct lightweight classification head at the final layer. Partial parameter copying (distillation) from each early-exit head ensures that the final heads remain diverse. Then at inference, intermediate early-exit features are not stored, thus simultaneously achieving low memory usage and minimal latency.", "essential_references_not_discussed": "While there might exist relevant papers beyond my knowledge, the authors appear to have discussed most of the essential references.", "other_strengths_and_weaknesses": "Strengths:\n- The proposed method creatively integrates early-exit (EE) ensembles and multi-head ensembles, achieving diversity among ensemble heads without incurring the typical memory and latency overhead associated with early exits. This design is particularly suitable for resource-constrained TinyML environments.\n- The paper is clearly written, straightforward, and easy to understand.\n\nWeaknesses:\n- Experimental evaluations for failure prediction and accuracy-drop detection are conducted based on single runs without reporting repeated trials or statistical summaries (e.g., mean and standard deviation), reducing confidence in the reliability of results.\n- Calibration experiments do not clearly demonstrate the method's superiority over baselines, as the reported advantages are minor relative to the standard deviations provided.", "other_comments_or_suggestions": "- Some visual details, particularly in Figure 1 and Figure 2, could be presented more clearly. For instance, explicitly indicating in Figure 2 that θ() includes a dense layer would enhance reader understanding.\n\n- For Table 2, for the DEEP, it should also be highlighted in bold.", "questions_for_authors": "1) For the EE-ensemble method, if we sequentially compute each early-exit's result without storing intermediate features (thus trading time for memory), how much latency would this actually introduce compared to QUTE? Considering that each early-exit has only a single layer of parameters, the computational overhead might be minimal.\n\n2) In the right column of line 303, the authors state: “On Big-MCU, QUTE achieves 31% and 47% latency reductions over EE-ensemble and DEEP, respectively, and maintains accuracy parity with both, even with 58% and 26% smaller models.” Could the authors clarify why the latency and memory reductions during inference compared to DEEP are only 47% and 26%, respectively? Specifically, how many ensemble members are used for DEEP, and are these ensemble members computed sequentially? If computed sequentially, shouldn't the memory consumption of DEEP be lower?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o0SI1YgmEj", "reviewer": "Reviewer_H2zK", "summary": "The paper introduces QUTE, a novel uncertainty quantification (UQ) framework optimized for TinyML models, addressing the challenge of efficient model monitoring in resource-constrained environments. QUTE leverages early-exit-assisted ensembles, where lightweight classification heads at the final network exit receive knowledge distilled from early-exits, ensuring diverse and resource-efficient ensemble predictions. Unlike prior methods that suffer from high computational and memory overheads, QUTE achieves 59% smaller model sizes and a 31% reduction in inference latency while maintaining superior uncertainty estimation quality. The proposed approach excels at detecting accuracy drops due to corrupted in-distribution (CID) data and outperforms state-of-the-art UQ methods in failure detection and calibration, making it a practical solution for on-device model reliability in real-world TinyML deployments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback. We appreciate the recognition of our work’s originality and the insightful suggestions.\n\n>  an ablation study isolating the impact of early-exit (EE) knowledge transfer and stronger justification for diversity benefits of EV-assistance\n\nAppendix B.3 explores the effect of EV-assistance on calibration and convergence. Also, we conducted an ablation study using the normalized disagreement (ND) metric [2] to measure QUTE’s ensemble diversity **with** and **without** EV-assistance (*higher is better*). On CIFAR10, ND is 71% lower without EV-assistance; on MNIST-ID, it's 19% lower. These results demonstrate that EE knowledge transfer improves ensemble diversity and, consequently, calibration quality. We will add these findings in the final version.\n\nAlso, see our response to reviewer y5iN for further discussion.\n\n> on QUTE’s reliance on additional layers for improved calibration in larger models (QUTE+)\n\nYes, QUTE does require additional layers to match the calibration quality of prior methods in larger models/datasets. However, QUTE (without additional layers) still outperforms prior methods on large datasets/models in accuracy-drop detection and failure detection, which are central to our work.\n\n\n> On missing lightweight bayesian approximation techniques\n\nBayesian batch norm requires stochastic sampling and multiple inferences (like Monte Carlo Dropout), which significantly increases latency on MCUs. Appendix B.2 discusses several single-pass deterministic methods and compares QUTE to PosteriorNets [1], which uses normalizing flows (see Table 6).\n\n> On missing references\n\nThank you for your detailed suggestions on potential citations to include. Many suggested references (Banbury et al., 2021; Jazbec et al., 2024; Antoran et al., 2020; Xia & Bouganis, 2023; Liang et al., 2020; Ovadia et al., 2019) are already covered in related work. We will further strengthen this section by incorporating additional recommended references.\n\n> on suggested theoretical analysis. \n\nSee our response to reviewer y5iN.\n\n> on statistical significance testing\n\nSee our response to reviewer eK42.\n\n\n> on justification for Hyperparameter Choices $w_{EV_k}$ and $\\delta$\n\nSection 5 explains that the number of EEs (K) is constrained by depth of neural networks we evaluate, which only have a few layers. For example, for Resnet-8 with three residual stacks, we place two EEs after the 1st and 2nd residual stack. For deeper networks, we place exits at evenly spaced locations. Appendix B.4 studies the trade-off between number of EEs (K) and uncertainty quality. Furthermore, we choose $w_{EV_k}$ and $\\delta$ based on a systematic empirical analysis as described in Appendix A.2. \n\n\n> on theoretical limitations of QUTE for non-classification tasks\n\nWhile our evaluations primarily focus on classification tasks, our method is not inherently limited to classification and can naturally extend to other problem domains. Ensemble methods, including those similar to ours, have been successfully applied to non-classification tasks in prior research. For the final version, we will explore evaluating QUTE on a regression-based task. Otherwise, as suggested, we will explicitly clarify that our current results are on classification tasks.\n\n\n> on measuring energy efficiency and estimating QUTE’s power savings compared to Deep ensembles and EE-ensemble\n\nWe agree that power consumption analysis is beneficial for TinyML deployment. However, direct power measurements are non-trivial due to factors like sleep mode power wastage, transition costs between sleep and active states, and peripheral activity. Instead, we estimate power consumption using typical values from MCU datasheets: ~285mW for the Big-MCU (STM32F767ZI) and ~25mW for the Small-MCU (STM32L432KC) in active mode with all peripherals disabled. We focus on estimating only active mode power because our work reduces the processing time/latency compared to other baselines.\nSince our method reduces inference latency compared to Deep ensembles and EE-ensemble, it proportionally lowers energy consumption, as energy-per-prediction is  power$\\times$time. While this is an estimate and real-world factors may influence it, QUTE effectively extends battery life without additional hardware modifications by accelerating inference.\n\n> on other calibration metrics\n\nWe discuss this Appendix D.\n\n> Other ablation studies\n\nAppendix B contains extensive ablation studies such as studying the effect of weight transfer on model convergence, investigating  the trade-off between uncertainty quality and ensemble size and effectiveness of the EV-assistance method. \n\n\n\n[1] Charpentier et al., Posterior network: Uncertainty estimation without ood samples via density-based pseudo-counts. NeurIPS 2020\n\n[2] Heidemann et al., Measuring Ensemble Diversity and Its Effects on Model Robustness IJCAI 2021", "claims_and_evidence": "The paper makes several key claims about QUTE, and most are well-supported by experimental evidence, but some areas could benefit from additional clarification or justification. Below is a critical assessment of the claims and the corresponding evidence:\n\n- Claim: QUTE is more resource-efficient than prior uncertainty quantification methods, with 59% smaller model sizes and 31% lower inference latency.\n- Evidence: The paper provides empirical results comparing QUTE against EE-Ensemble, Deep Ensembles (DEEP), and MCD on MCUs (Big-MCU and Small-MCU). Figure 3 and Section 6.1 show clear reductions in model size and latency, particularly demonstrating that some baselines (e.g., DEEP) do not fit on memory-constrained devices.\nAssessment: Well-supported. The quantitative results strongly back this claim.\n\n- Claim: QUTE outperforms all baselines in detecting accuracy drops due to corrupted in-distribution (CID) data.\n- Evidence: Section 6.2 presents AUPRC scores across multiple datasets (MNIST-C, CIFAR10-C, TinyImageNet-C), showing that QUTE consistently achieves the highest values, outperforming EE-Ensemble, DEEP, and G-ODIN. The discussion highlights how QUTE's early-exit knowledge transfer improves uncertainty calibration.\nAssessment: Well-supported. However, an ablation study isolating the impact of early-exit knowledge transfer versus conventional early-exit ensembles would strengthen this claim.\n\n- Claim: QUTE provides better uncertainty calibration than competing methods.\n- Evidence: Table 3 presents expected calibration error (ECE), negative log-likelihood (NLL), and Brier Score (BS), showing that QUTE achieves lower ECE and NLL in TinyML settings. Additionally, QUTE+ (with enhanced output heads) further improves calibration, approaching Deep Ensembles in performance.\nAssessment: Mostly supported. The results are compelling, but QUTE’s reliance on additional layers for improved calibration in larger models (QUTE+) suggests a trade-off between efficiency and calibration performance, which should be explicitly acknowledged.\n\n- Claim: QUTE enables better failure detection, distinguishing both in-distribution misclassifications (ID✓ | ID×) and out-of-distribution samples (ID✓ | OOD).\n- Evidence: Table 2 reports AUROC values, demonstrating that QUTE outperforms all baselines in ID✓ | ID× detection and is competitive in ID✓ | OOD detection. However, in larger models, G-ODIN performs better in OOD detection.\nAssessment: Well-supported with a minor caveat. The claim should clarify that QUTE excels in CID detection and misclassification detection, but specialized methods like G-ODIN may still be preferable for OOD scenarios.\n\nProblematic or Weakly Supported Claims:\nQUTE achieves superior calibration without increasing model complexity.\n\n- Issue: The results indicate that for larger models, QUTE+ (which adds learning layers) is needed to match calibration quality. This suggests that superior calibration does require additional complexity in certain cases, contradicting the claim.\n- Suggested Fix: The authors should clarify that QUTE is highly efficient in TinyML settings, but larger models may require additional modifications to achieve optimal calibration.\nQUTE generalizes well across all TinyML applications.\n\n- Issue: The evaluation is focused on image and audio classification tasks (e.g., MNIST, CIFAR10, SpeechCommands). There is no evidence for its effectiveness in other TinyML applications like time-series forecasting, anomaly detection, or reinforcement learning.\n- Suggested Fix: The authors should either broaden the scope of their evaluation or refine the claim to state that QUTE is optimized for classification tasks in TinyML.", "methods_and_evaluation_criteria": "The methods and evaluation criteria used in the paper are generally well-chosen for the problem of uncertainty quantification (UQ) in TinyML models, but there are some areas that could be improved or clarified. Below is a critical assessment:\n\nStrengths of the Methods and Evaluation Criteria:\nChoice of Benchmark Datasets:\n\nThe authors evaluate QUTE on four different datasets:\nMNIST (4-layer CNN)\nSpeechCommands (4-layer DSCNN for keyword spotting)\nCIFAR10 (ResNet-8, MLPerf benchmark model)\nTinyImageNet (MobileNetV2)\nJustification: These datasets represent varying levels of complexity and modality (image vs. audio), which is appropriate for evaluating TinyML models.\nAssessment: Appropriate choice. However, all tasks are classification-based, and there is no evaluation on non-classification tasks (e.g., time-series forecasting or regression), which limits generalization.\nEvaluation Metrics for Uncertainty Quantification:\n\nThe paper reports Expected Calibration Error (ECE), Brier Score (BS), and Negative Log-Likelihood (NLL) to assess uncertainty quality.\nThe use of AUROC and AUPRC for failure detection and accuracy-drop detection is well-motivated, as they provide threshold-independent performance evaluations.\nAssessment: Well-founded choices. However, ECE is known to have limitations, and an alternative like adaptive calibration error (ACE) or logit-scaled calibration metrics could provide a more robust evaluation.\nComparison with Relevant Baselines:\n\nThe paper compares QUTE with several state-of-the-art UQ methods:\nMonte Carlo Dropout (MCD)\nDeep Ensembles (DEEP)\nEE-Ensemble (prior early-exit-based ensemble method)\nG-ODIN (OOD detection method)\nHYDRA (ensemble distillation)\nAssessment: Comprehensive comparisons. The selected baselines are appropriate, covering both ensemble-based UQ approaches and early-exit methods. However, Bayesian Neural Networks (BNNs), which are a key competitor in UQ, are not included in the evaluation, even though they are mentioned in the related work.\nMicrocontroller (MCU) Deployment Evaluations:\n\nThe paper evaluates QUTE on two MCU platforms (Big-MCU: STM32F767ZI, Small-MCU: STM32L432KC) to assess real-world feasibility.\nThe results show that QUTE reduces latency by 31% and has a 59% smaller model size, highlighting its TinyML suitability.\nAssessment: Crucial evaluation for real-world deployment. However, power consumption analysis (e.g., energy per inference) would further strengthen the real-world applicability.\n\n\n\nWeaknesses and Areas for Improvement:\nLack of Justification for Hyperparameter Choices:\n\nThe number of early-exits (K) and the weighting factors (wEVk and δ) for knowledge transfer are empirically chosen, but no systematic analysis is provided.\nFix: An ablation study on K and weighting factors should be included to demonstrate their impact on performance.\nAbsence of Statistical Significance Testing:\n\nThe paper presents mean and standard deviation for some metrics but does not report statistical significance tests (e.g., t-tests, Wilcoxon signed-rank tests) to confirm that performance differences are meaningful.\nFix: Including confidence intervals or statistical significance testing would improve result robustness.\nLimited Generalization Beyond Classification Tasks:\n\nThe evaluation is entirely focused on classification tasks, which makes it difficult to assess how well QUTE generalizes to other TinyML applications (e.g., time-series forecasting, anomaly detection, or reinforcement learning).\nFix: The paper should either broaden its evaluation scope or clearly limit its claims to classification-based tasks.", "theoretical_claims": "The paper does not present formal mathematical proofs but makes several theoretical claims regarding the behavior of uncertainty quantification in TinyML models, particularly in the context of early-exit-assisted ensembles. Below is a critical evaluation of these theoretical claims:\n\nKey Theoretical Claims and Their Validity\nClaim: Early-exit knowledge distillation enhances uncertainty estimation while reducing model size and compute overhead.\n\nExplanation: The paper introduces a method where early-exit layers are used during training to distill uncertainty-related knowledge into lightweight output heads at the final exit. The claim is that this approach enables diverse ensemble behavior without significant computational overhead.\nEvaluation:\nThe method is well-motivated by prior works on early-exit architectures (Teerapittayanon et al., 2016; Ghanathe & Wilton, 2023).\nThe empirical results show that QUTE reduces latency (31%) and model size (59%), supporting the efficiency claim.\nHowever, the claim that this leads to \"diverse ensemble behavior\" is only empirically shown, but lacks a formal proof on how early-exit distillation systematically maintains diversity in predictions.\nPotential Issue:\nThe paper could benefit from a theoretical analysis showing how the weight transfer mechanism preserves predictive diversity across ensemble members.\nClaim: QUTE's predictive confidence is correlated with accuracy, leading to improved calibration.\n\nExplanation: The paper states that QUTE produces better-calibrated uncertainty estimates because its final exit integrates knowledge from early-exits, reducing overconfidence issues common in deep networks.\nEvaluation:\nThe claim is supported by empirical calibration metrics (ECE, NLL, BS in Table 3), showing that QUTE achieves lower calibration errors than baselines.\nHowever, there is no theoretical justification explaining why early-exit knowledge distillation improves uncertainty calibration.\nPotential Issue:\nA mathematical framework or proof showing how QUTE affects the confidence distribution could strengthen this claim. For example, a formal derivation of confidence variance reduction due to early-exit knowledge integration would be useful.\nClaim: Smaller models are naturally less overconfident on corrupted in-distribution (CID) data.\n\nExplanation: The paper suggests that smaller models, by nature, produce less overconfident predictions under covariate shift (CID data) compared to large models, which tend to overfit to training distribution features.\nEvaluation:\nThe claim is partially supported by empirical observations (Table 1), which show that smaller ResNet models (2-stack and 3-stack) have better calibration on CID data than deeper variants.\nHowever, the claim is not rigorously proven. The idea aligns with prior findings on overparameterization and generalization (Hsu et al., 2020), but a formal proof (e.g., based on Bayesian uncertainty theory or PAC-Bayes bounds) would be beneficial.\nPotential Issue:\nThe claim should either be framed as an empirical observation rather than a theoretical result, or a formal derivation of model confidence behavior with respect to depth should be provided.\n\n\nAreas Where Theoretical Rigor Could Be Improved:\nMathematical Proof for Ensemble Diversity Preservation:\n\nThe authors claim that early-exit knowledge transfer ensures diversity among ensemble members, but this is only demonstrated empirically. A theoretical analysis of how early-exit weight transfer affects predictive diversity would strengthen the paper.\nFormal Justification for Calibration Improvement:\n\nWhile empirical results show better calibration, the paper lacks a theoretical justification for why QUTE’s uncertainty estimates are better calibrated. A derivation using information-theoretic arguments (e.g., entropy reduction) could help.\nExplicit Bounds on Uncertainty Estimation Quality:\n\nThe paper could derive theoretical bounds on QUTE’s uncertainty estimation efficiency compared to standard ensembles or Bayesian approaches, quantifying trade-offs between accuracy, uncertainty, and computational cost.", "experimental_designs_or_analyses": "The experimental design of the paper is well-structured and aligns with the research objectives, but there are some areas that require further clarification or improvements to ensure robustness and validity. Below is a critical analysis of the soundness and validity of the experimental design and analyses.\n\nStrengths of the Experimental Design\nComprehensive Benchmarking Against Relevant Baselines\n\nThe paper compares QUTE against multiple state-of-the-art uncertainty quantification (UQ) methods, including:\nMonte Carlo Dropout (MCD)\nDeep Ensembles (DEEP)\nEE-Ensemble (Early-exit-based ensemble method)\nG-ODIN (OOD detection method)\nHYDRA (Ensemble distillation method)\nAssessment: Sound choice of baselines as these methods represent different categories of UQ techniques. However, a Bayesian Neural Network (BNN) baseline is missing, which would have provided further context.\nWell-Defined Evaluation Criteria for Uncertainty Estimation\n\nThe paper evaluates uncertainty quantification using:\nExpected Calibration Error (ECE)\nNegative Log-Likelihood (NLL)\nBrier Score (BS)\nAssessment: These are standard metrics in uncertainty estimation. However, ECE is known to have limitations (e.g., sensitivity to binning choices), and an alternative such as Adaptive Calibration Error (ACE) could have been included for a more robust analysis.\nReal-World Feasibility Analysis on Microcontrollers (MCUs)\n\nThe authors deploy QUTE on two embedded MCUs:\nBig-MCU: STM32F767ZI (high resource)\nSmall-MCU: STM32L432KC (low resource, power-efficient)\nResults demonstrate QUTE’s efficiency in terms of memory, latency, and fit on constrained devices.\nAssessment: Strong validation of real-world applicability, but energy consumption per inference is not reported, which is crucial for embedded deployments.\n\n\n\nIssues and Areas for Improvement\n1. Hyperparameter Sensitivity Analysis is Lacking\nIssue:\nThe paper empirically chooses hyperparameters (e.g., number of early exits (K), weighting factors (wEVk, δ)) but does not systematically analyze their impact.\nFix:\nAn ablation study on how K and weighting factors influence uncertainty quality, accuracy, and computational efficiency would improve rigor.\n2. Statistical Significance Testing is Missing\nIssue:\nThe paper reports mean and standard deviations but does not include statistical significance testing (e.g., t-tests, Wilcoxon signed-rank tests) to validate performance differences across methods.\nFix:\nReporting confidence intervals or p-values for comparisons (e.g., QUTE vs. EE-Ensemble) would confirm whether differences are statistically meaningful.\n3. Limited Generalization to Non-Classification Tasks\nIssue:\nAll experiments focus on classification tasks (MNIST, SpeechCommands, CIFAR10, TinyImageNet).\nNo evaluation is provided for time-series forecasting, anomaly detection, or reinforcement learning, which are relevant in TinyML applications.\nFix:\nThe scope of generalization should be clearly stated. Alternatively, a small-scale experiment on a time-series dataset could strengthen claims of broader applicability.\n4. No Analysis on Failure Cases\nIssue:\nWhile QUTE outperforms baselines on accuracy-drop and failure detection, there is no discussion on failure cases (e.g., when QUTE fails to detect uncertainty correctly).\nFix:\nA failure case analysis (e.g., qualitative examples of misclassified instances with poor uncertainty estimates) would provide deeper insights.", "relation_to_broader_scientific_literature": "The paper presents QUTE, an early-exit-assisted ensemble method for uncertainty quantification (UQ) in TinyML models. Its key contributions relate to several existing areas of machine learning research, including uncertainty estimation, early-exit networks, model monitoring, and TinyML deployment. Below is a structured evaluation of how QUTE connects to prior research:\n\n1. Uncertainty Quantification (UQ) in Neural Networks\nRelation to Prior Work\nThe problem of uncertainty quantification in deep learning is well studied, with two major categories:\nBayesian Approaches\n\nBayesian Neural Networks (BNNs) (Blundell et al., 2015) model uncertainty via a probabilistic distribution over weights.\nMonte Carlo Dropout (MCD) (Gal & Ghahramani, 2016) approximates Bayesian inference by applying dropout during inference.\nQUTE’s Connection: Instead of using probabilistic approaches (which are computationally expensive for TinyML), QUTE uses ensemble-based methods to estimate uncertainty efficiently.\nNovelty: Unlike MCD or BNNs, QUTE does not require multiple inference passes and is optimized for low-resource TinyML settings.\nDeep Ensembles for Uncertainty Estimation\n\nLakshminarayanan et al. (2017) introduced Deep Ensembles, which train multiple independent models for uncertainty estimation.\nQUTE’s Connection: QUTE leverages ensembles but does not train separate models; instead, it distills uncertainty knowledge from early exits into lightweight ensemble members.\nImprovement: QUTE achieves similar uncertainty estimation quality as Deep Ensembles while using 59% fewer parameters and 31% less inference latency, making it more practical for TinyML.\nKey Differentiation\nQUTE improves on ensemble-based UQ methods by using a single forward pass instead of multiple inference runs.\nNo prior UQ method explicitly optimizes for ultra-low-resource TinyML in this way.\n2. Early-Exit Networks and Model Efficiency\nRelation to Prior Work\nEarly-exit networks were originally introduced to reduce inference latency in deep models (Teerapittayanon et al., 2016; Kaya et al., 2019).\nPrior methods like EE-Ensemble (Qendro et al., 2021) leveraged early-exit networks for ensemble-based uncertainty estimation, but these required additional learning layers, increasing memory overhead.\nQUTE’s Connection: Instead of directly using early-exit outputs as ensemble members (like EE-Ensemble), QUTE distills their knowledge into lightweight final classification heads.\nImprovement: This allows QUTE to achieve better calibration and accuracy-drop detection without additional per-exit learning layers.\nKey Differentiation\nEE-Ensemble uses early-exits directly, requiring additional computational layers.\nQUTE removes early-exits after training and retains only the lightweight ensemble heads, reducing resource consumption.\n3. TinyML Deployment and Model Monitoring\nRelation to Prior Work\nTinyML aims to run ML models on ultra-low-power microcontrollers (MCUs) with limited memory (≤256KB RAM, <1W power).\nPrior work on TinyML deployment (Banbury et al., 2021; Ghanathe & Wilton, 2023) focused on latency, power efficiency, and model size.\nQUTE’s Connection:\nUnlike general TinyML models, QUTE is explicitly designed for real-time model monitoring and uncertainty estimation in TinyML settings.\nPrior model monitoring methods (Bifet & Gavalda, 2007; Hsu et al., 2020) rely on either large statistical tests or access to true labels, which are impractical in real-world TinyML deployments.\nQUTE provides uncertainty-aware monitoring without access to ground truth labels.\nKey Differentiation\nExisting TinyML models prioritize efficiency but do not focus on model monitoring and uncertainty quantification.\nQUTE fills this gap by introducing uncertainty estimation tailored for real-time TinyML inference.\n4. Robustness to Corrupted In-Distribution (CID) and Out-of-Distribution (OOD) Data\nRelation to Prior Work\nRobustness against corrupted in-distribution (CID) data is less studied compared to out-of-distribution (OOD) detection.\nPrior OOD detection methods (Hendrycks & Gimpel, 2018; Liang et al., 2020) focused on confidence-based rejection techniques.\nG-ODIN (Hsu et al., 2020) introduced a preprocessing-based approach for OOD detection.\nQUTE’s Connection:\nQUTE outperforms OOD detectors like G-ODIN in detecting accuracy drops due to CID data.\nUnlike traditional OOD detectors, QUTE is designed for low-power TinyML devices.\nKey Differentiation\nMost prior works focus on OOD detection, while QUTE addresses the more practical problem of CID robustness in TinyML.", "essential_references_not_discussed": "The paper does a good job of citing relevant literature in uncertainty quantification (UQ), early-exit networks, model monitoring, and TinyML deployment. However, there are several key references missing that would help place QUTE’s contributions into a more complete scientific context. Below are some critical missing references that should be included:\n\n1. Missing Work on Lightweight Bayesian Uncertainty Quantification\nWhy It’s Important?\nThe paper only briefly mentions Bayesian Neural Networks (BNNs) (Blundell et al., 2015) and Monte Carlo Dropout (Gal & Ghahramani, 2016) as baselines.\nHowever, more recent lightweight BNN approaches exist that optimize Bayesian inference for edge devices, which are directly relevant to QUTE’s efficiency claim.\nMissing References:\nTeye et al. (2018): Bayesian Batch Normalization for Uncertainty Estimation\n\nIntroduced Bayesian BatchNorm, which achieves Bayesian-like uncertainty estimation without needing full BNNs.\nWhy it's relevant? This method enables UQ in a single forward pass, similar to QUTE, but using batch normalization instead of ensembles.\nWhy cite? Helps contextualize alternative lightweight Bayesian methods for uncertainty quantification.\nOsawa et al. (2019): Practical Deep Learning with Bayesian Approximation\n\nProposed low-cost Bayesian learning techniques optimized for deep models in low-resource settings.\nWhy cite? QUTE’s motivation (low-power UQ for TinyML) is similar, and comparison with lightweight BNNs is missing.\nLouizos & Welling (2017): Multiplicative Normalizing Flows for Bayesian Deep Learning\n\nUsed normalizing flows to achieve Bayesian uncertainty with lower overhead than traditional BNNs.\nWhy cite? The work shows how uncertainty can be captured without full ensembles, similar to QUTE.\nWhat’s the Problem?\nThe paper only contrasts QUTE against classical Bayesian UQ (BNNs, MCD) but does not compare against these more efficient Bayesian methods.\nFix: Include recent lightweight Bayesian methods to better contrast why QUTE is a superior choice for TinyML.\n2. Missing Work on Early-Exit Networks for Uncertainty Estimation\nWhy It’s Important?\nThe core idea of QUTE is distilling early-exit knowledge into final ensemble heads.\nHowever, recent works have already explored early-exit UQ but are not cited.\nMissing References:\nJazbec et al. (2024): Conditional Monotonicity in Early-Exit Networks\n\nInvestigated how early-exit networks should be structured to ensure uncertainty-aware decisions.\nWhy cite? This paper provides theoretical insights into why early-exit-based ensembles (like QUTE) work.\nAntoran et al. (2020): Early-Exit Networks for Depth Uncertainty in Neural Networks\n\nProposed a method to quantify uncertainty by leveraging intermediate (early-exit) layers.\nWhy cite? This is one of the closest prior works to QUTE, yet it is not cited.\nFerianc & Rodrigues (2023): Multi-Exit Neural Networks for Uncertainty-Aware Inference\n\nProposed a multi-exit architecture that integrates uncertainty estimation at each exit.\nWhy cite? This is a direct baseline for QUTE, but is not discussed.\nWhat’s the Problem?\nThe paper claims QUTE is the first early-exit ensemble architecture for uncertainty quantification, but this is misleading—previous works have explored early-exit UQ approaches, though with different architectures.\nFix: The authors should acknowledge prior work on early-exit UQ and clearly explain how QUTE is different.\n3. Missing Work on Out-of-Distribution (OOD) and Corrupted In-Distribution (CID) Detection\nWhy It’s Important?\nThe paper evaluates QUTE on both OOD detection and CID robustness.\nHowever, several key works on robust UQ for corrupted data are missing.\nMissing References:\nOvadia et al. (2019): Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty in Deep Learning\n\nFound that modern deep networks fail to estimate uncertainty correctly under distribution shifts.\nWhy cite? This work motivates QUTE’s focus on detecting accuracy drops in CID data.\nXia & Bouganis (2023): Failure Detection in Neural Networks using Uncertainty Estimates\n\nProposed uncertainty-based failure detection for real-world models.\nWhy cite? This paper discusses uncertainty-driven failure detection, which is one of QUTE’s key claims.\nLiang et al. (2020): Enhancing the Reliability of Out-of-Distribution Detection in Neural Networks\n\nInvestigated why deep networks struggle with OOD data and proposed improved techniques.\nWhy cite? Helps frame why QUTE is evaluated against OOD baselines.\nWhat’s the Problem?\nThe paper does not connect QUTE’s performance on CID/OOD to prior literature.\nFix: Cite these works to show how QUTE extends prior research on uncertainty estimation under distribution shifts.\n4. Missing Work on Energy-Efficient Model Deployment for TinyML\nWhy It’s Important?\nQUTE is designed for TinyML, but the paper does not cite foundational works on energy-efficient ML.\nRecent studies have proposed alternative ways to reduce energy consumption, such as pruning and quantization.\nMissing References:\nBanbury et al. (2021): MLPerf Tiny Benchmark for Ultra-Low-Power ML\n\nStandardized benchmarks for TinyML efficiency.\nWhy cite? QUTE is evaluated on TinyML devices, so this benchmark should be referenced.\nHoward et al. (2019): Searching for MobileNetV3\n\nMobileNetV3 is designed for efficient inference on TinyML devices.\nWhy cite? QUTE uses MobileNetV2, but citing MobileNetV3 would strengthen the discussion on TinyML efficiency.\nHan et al. (2016): Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization, and Huffman Coding\n\nIntroduced pruning and quantization to make ML models more efficient.\nWhy cite? These techniques could be combined with QUTE for even better TinyML deployment.\nWhat’s the Problem?\nThe paper focuses only on MCU deployment but does not cite prior works on efficient TinyML models.\nFix: Cite MLPerf Tiny and model compression techniques to show how QUTE fits into broader TinyML efficiency research.", "other_strengths_and_weaknesses": "This paper makes a strong contribution to the field of uncertainty quantification (UQ) in TinyML through a novel early-exit-assisted ensemble approach. Below is a breakdown of its originality, significance, and clarity, as well as potential limitations that should be addressed.\n\n\n1. Originality:\nCreative Combination of Ideas:\n\nThe paper innovatively integrates early-exit networks with ensemble-based uncertainty estimation.\nWhile early-exit architectures and ensemble-based UQ have been studied before, QUTE’s novel contribution is its distillation-based ensemble construction, which eliminates early-exits after training to reduce overhead.\nThis approach differs from prior methods (e.g., EE-Ensemble) that retain early-exits during inference, leading to higher computational costs.\nFirst UQ Approach Optimized for TinyML:\n\nWhile UQ in deep learning is a well-explored field, most state-of-the-art methods (BNNs, Deep Ensembles, MCD) are computationally expensive and impractical for TinyML devices.\nQUTE is the first method explicitly designed for resource-constrained TinyML model monitoring.\n2. Significance:\nAddresses a Real-World Challenge in TinyML:\n\nTinyML models are deployed on edge devices with no access to ground truth labels, making uncertainty-aware model monitoring essential.\nQUTE enables on-device uncertainty estimation with minimal computational cost, making it practical for autonomous systems, medical IoT, and embedded AI applications.\nImproves Over Key Baselines:\n\nCompared to Deep Ensembles (Lakshminarayanan et al., 2017), EE-Ensemble (Qendro et al., 2021), and MCD (Gal & Ghahramani, 2016), QUTE achieves:\n59% model size reduction\n31% lower inference latency\nBetter uncertainty calibration (lower ECE, BS, NLL)\nThese improvements are critical for real-world deployment on ultra-low-power MCUs.\nStrong Experimental Validation on Microcontrollers (MCUs):\n\nUnlike many ML papers that rely on simulated efficiency claims, QUTE is deployed on real-world MCUs (STM32 Big-MCU, Small-MCU).\nThis enhances practical significance, showing that the method is deployable, not just theoretically interesting.\n3. Clarity:\nClearly structured methodology and motivation:\n\nThe problem statement, approach, and contributions are well-articulated.\nFigures clearly illustrate QUTE’s architecture and how it differs from baselines.\nThe paper provides a solid review of related work (although some missing references should be added).\nStrong result visualizations:\n\nThe tables and graphs effectively present key findings, especially in uncertainty calibration and accuracy-drop detection.\nHowever, some figures (e.g., confidence calibration histograms) would improve clarity on model uncertainty performance.\nWeaknesses\n1. Limited Theoretical Analysis\nNo Formal Proofs on Uncertainty Calibration Improvement:\n\nWhile empirical results show that QUTE improves uncertainty calibration, there is no theoretical justification for why the method produces better uncertainty estimates.\nSuggested Fix:\nA mathematical analysis (e.g., using confidence variance bounds) could strengthen the claim that early-exit-assisted ensembles improve calibration.\nNo Formal Diversity Analysis of the Ensemble:\n\nThe paper claims that early-exit knowledge distillation maintains predictive diversity, but this is only shown empirically.\nSuggested Fix:\nA Shannon entropy analysis or mutual information study between QUTE’s ensemble members would provide a stronger justification for its diversity benefits.\n2. Missing Statistical Significance Testing\nWhile performance improvements are clear, statistical significance is not tested.\nResults are reported with mean and standard deviation, but no hypothesis testing (e.g., t-tests, Wilcoxon signed-rank tests) is provided.\nSuggested Fix:\nReporting confidence intervals or p-values would confirm whether QUTE’s improvements over baselines are statistically significant.\n3. Lack of Generalization Beyond Classification Tasks\nQUTE is only tested on image and audio classification tasks.\nMany TinyML applications involve time-series forecasting, anomaly detection, and reinforcement learning, but these are not explored in the evaluation.\nSuggested Fix:\nA small experiment on a TinyML time-series dataset (e.g., Google Smartwatch Health Data) would strengthen the claim that QUTE generalizes to all TinyML settings.\nIf not feasible, the paper should clearly limit its scope to classification-based TinyML monitoring.\n4. Missing Comparison with Lightweight Bayesian Methods\nThe paper contrasts QUTE only against traditional BNNs and MCD, but does not compare against more efficient Bayesian methods like:\nBayesian BatchNorm (Teye et al., 2018)\nVariational Bayesian Dropout (Osawa et al., 2019)\nMultiplicative Normalizing Flows (Louizos & Welling, 2017)\nWhy This Matters?\nThese methods also reduce inference overhead while maintaining Bayesian uncertainty estimation.\nWithout this comparison, the efficiency advantage of QUTE is slightly overstated.\nSuggested Fix:\nInclude a brief discussion on why QUTE is preferable to these lightweight Bayesian techniques.", "other_comments_or_suggestions": "No", "questions_for_authors": "Q1: The paper empirically demonstrates that QUTE achieves better uncertainty calibration (lower ECE, NLL, BS) than Deep Ensembles and MCD. However, there is no theoretical justification for why early-exit knowledge transfer improves calibration.\nCould you provide a mathematical explanation or theoretical analysis (e.g., variance reduction, entropy-based confidence bounds) that supports this claim?\n\nQ2: While the paper reports mean and standard deviation for key performance metrics, there is no statistical significance testing (e.g., t-tests, Wilcoxon signed-rank tests).\nDid you conduct statistical significance tests to confirm that QUTE’s improvements over baselines are meaningful? If not, could you provide confidence intervals or p-values for the comparisons?\n\nQ3: The number of early-exit ensemble members (K) and the weighting factors (wEVk, δ) appear to be chosen empirically, but no systematic analysis of their impact is provided.\nCould you clarify how these values were selected and whether an ablation study was conducted to determine their effect on accuracy, uncertainty estimation, and efficiency?\n\nQ5: All experiments focus on classification tasks (MNIST, CIFAR10, TinyImageNet, SpeechCommands), but many TinyML applications involve time-series forecasting, anomaly detection, and reinforcement learning.\nDid you test QUTE on any non-classification TinyML tasks, or do you see any theoretical limitations that would prevent it from generalizing to these domains?\n\nQ6: While the paper reports model size and inference latency, there is no mention of power consumption, which is a key metric for TinyML deployment.\nDid you measure energy efficiency (e.g., power per inference) on the MCUs, or can you provide an estimate of QUTE’s power savings compared to Deep Ensembles and EE-Ensemble?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "raemzfV6OT", "reviewer": "Reviewer_y5iN", "summary": "This paper proposes QUTE, a new uncertainty quantification (UQ) method for tinyML models on low-power devices. It uses a lightweight early-exit ensemble to reduce size and computation while maintaining accuracy. QUTE is 59% smaller than previous models which can reduce latency by 31%, and as a result, it improves accuracy-drop detection.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments and thoughtful feedback. \nGiven that your comments are highly encouraging and did not raise significant concerns, we would appreciate any insights into the current score to help improve our work. We look forward to having a fruitful discussion and addressing any concerns/suggestions.\n\n> on a few missing essential references \n\nThank you for the suggestions. We have already included MIMMO (Ferianc et al., 2023) and will add the others.\n\n> on why transferring early-exit (EE) knowledge improves uncertainty estimation\n\nPlease see the response to reviewer H2zK for ablation results on EV-assistance's impact on ensemble diversity.\n\n**Intuition**: Theoretical analysis on how EE knowledge transfer improves calibration would undoubtedly strengthen the paper. Our work builds on the intuition and empirical observations from prior studies, such as [1] and [2], which leverage EEs to form ensembles. These studies suggest that intermediate classifiers extract progressively refined feature abstractions, incorporating complementary feature representations that enhance uncertainty estimates.\nUnlike prior works, which combine these diverse feature representations from multiple EEs to form an ensemble, our approach generates complementary feature representations from different depths by directly utilizing the weights of EE layers, which is significantly more resource-efficient. QUTE's improved calibration supports this hypothesis.\n\n> on suggested theoretical analyses \n\nWhile theory is not central to this work, we thank reviewers y5iN and H2zK for highlighting some interesting avenues for theoretical investigations. Specifically, questions such as *why transferring early-exit knowledge improves calibration?* and *why QUTE’s weight transfer mechanism imbibes diversity?* could certainly strengthen the paper. To address this, we have conducted an ablation study to investigate the impact of EV-assistance on ensemble diversity (see reply to reviewer H2zK). While formal derivations may be challenging within the rebuttal period, we plan to explore these questions further and consider including them in the final version. We will also clarify that our findings are empirical rather than theoretical.\n\n> on sensitivity of results to confidence threshold\n\nWe report threshold-free AUPRC and AUROC metrics by varying the confidence threshold from 0 to 1 in steps of 0.1. These metrics summarize the trade-off between precision-recall and TPR-FPR across different thresholds, ensuring results are not optimized for a specific threshold.\n\n> [...] authors claim that smaller models perform better under corruption, but how and why?\n\nPrior work [3] shows that larger models overfit, leading to overconfidence, as they extract high-level abstractions that make distinguishing corrupted from clean inputs harder. In contrast, smaller models benefit from implicit regularization, preventing overfitting. They generalize better on corruptions by relying on more stable features instead of memorizing fine-grained ones. Our empirical results (Table 1) support this. From a Bayesian perspective, smaller models can exhibit a wider posterior due to implicit regularization, making them better at capturing uncertainty.\n\n>  Does the proposed QUTE model handle OOD? [...]\n\nSection 6.3 (Table 2) compares QUTE’s OOD detection against G-ODIN [3]. QUTE outperforms G-ODIN on tiny models and is competitive on larger ones. See Section 6.3 for details.\n\n>  How does QUTE reduce unnecessary energy costs?\n\nSee our response to reviewer H2zK.\n\n>  Does the proposed QUTE model work for large models?\n\nYes. We evaluate QUTE’s accuracy-drop detection on Resnet50. See our response to reviewer XBf5 for details.\n\n>  Is the removal of the final exit always important and beneficial?\n\nYes. The final exit tends to be overconfident under corruptions, degrading accuracy-drop detection. For example, on MNIST with fog corruption, the final exit's average confidence **increases** by 23% compared to its in-distribution confidence (i.e., overconfidence), whereas the QUTE ensemble's average confidence **decreases** by 52%, improving accuracy-drop detection.\n\n> Does QUTE work under distribution shift?\n\nYes. Sections 6.2 and 6.3 show QUTE detects accuracy drops under corruptions and OOD inputs, both forms of distribution shift. We also evaluate different corruption severity levels, simulating varying degrees of shift. However, failure detection differs from robust generalization (performing reliably under shifts) and adaptation (adjusting to unknown inputs). Like all baselines, QUTE is a failure detection method.\n\n\n\n[1] Qendro et al. Early exit ensembles for uncertainty quantification, PMLR 2021\n\n[2] Ferianc et al., Multi-input massive multi-output neural network. CVPR 2023\n\n[3] Hsu et al., Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data. CVPR 2020", "claims_and_evidence": "The claims in the paper are partially supported by the evidence provided. The authors present empirical results showing improvements in model size, latency, and accuracy-drop detection. While these results suggest the effectiveness of QUTE, additional evaluations on diverse architectures and real-world scenarios would further strengthen the claims.", "methods_and_evaluation_criteria": "Yes, the proposed method and evaluation criteria are appropriate for this case study. Using early-exit ensembles and uncertainty quantification fits well for tinyML models. The evaluation metrics are also relevant and help to further validate the proposed approach.", "theoretical_claims": "I found out that this paper does not provide formal proofs for its theoretical claims, focusing instead on empirical results. Theoretical analysis is not a major part of this work, and the claims are primarily supported by experimental evidence.", "experimental_designs_or_analyses": "I think the experimental designs and analyses are generally sound. The evaluation metrics seem appropriate to me for this problem. However, in my view, the experiments could benefit from a wider range of benchmarks, and testing on more diverse architectures or real-world data would make the results stronger. Some more detailed analysis of trade-offs would also help.", "relation_to_broader_scientific_literature": "In my view, the key contributions of the paper build on existing work in uncertainty quantification (UQ) and tinyML models. The idea of using early-exit ensembles for UQ has been explored, but QUTE improves efficiency by reducing model size and latency. Compared to previous studies, it addresses the challenges of applying UQ to tinyML, showing significant improvements in both efficiency and accuracy-drop detection.", "essential_references_not_discussed": "In my view, the paper cites relevant prior work, but there are a few essential references that could further strengthen the context for the key contributions. For example,\n1.\tMulti-Dimensional Conformal Prediction\n2.\tTinyTTA: Efficient Test-time Adaptation via Early-exit Ensembles on Edge Devices\n3.\tMIMMO: Multi-input massive multi-output neural network", "other_strengths_and_weaknesses": "Strengths:\n1.\tThe paper combines early-exit ensembles with uncertainty quantification (UQ) for tinyML models, providing a resource-efficient solution for low-power devices.\n2.\tThe results show notable improvements in latency and model size compared to prior methods, making it highly relevant for resource-constrained applications.\n3.\tThe experimental results are well-presented, clearly demonstrating the effectiveness of the proposed method.\nWeaknesses:\n1.\tThe paper could benefit from a more detailed discussion of recent related works, especially those from 2023, 2024 and 2025, to better contextualize the contributions.\n2.\tThe paper relies mostly on experiments and does not provide a formal theoretical analysis. So, I think some theoretical support would make it stronger.\n3.\tWhile the results are acceptable, more diverse real-world evaluations would strengthen the paper and further show the method’s generalizability.", "other_comments_or_suggestions": "1.\tThe study presents QUTE as a more efficient alternative to EE-ensemble, but there is no theoretical analysis showing why transferring early-exit knowledge improves uncertainty estimation.\n2.\tI would suggest providing a clear differentiation from existing uncertainty-aware early-exit architectures.\n3.\tThe proposed model copies early-exit weights to early-view blocks.\n4.\tThe confidence threshold (ρ) plays a critical role in the proposed model, so I want to know how sensitive the results to this threshold are.\n5.\tAfter reading the paper carefully, the authors claim that smaller models perform better under corruption, but how and why? \n6.\tA more detailed comparison with previous uncertainty quantification methods. \n7.\tDoes the proposed QUTE model handle OOD? If yes, how does QUTE work compared to standard techniques?\n8.\tHow does QUTE reduce unnecessary energy costs?", "questions_for_authors": "1.\tDoes the proposed QUTE model work for large models?\n2.\tIs the removal of the final exit always important and beneficial?\n3.\tDoes QUTE work under distribution shift?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "I did not flag this paper for an ethics review. There are no ethical concerns I identified in this paper."}}]

Author rebuttal:

Meta-review: This manuscript received favourable reviews from all 4 reviewers. The reviews and discussions highlighted both strengths and weaknesses but the general tendency is in favour.

On the side of strengths, it was commented that the paper proposed a method which adapts early-exit strategies to improve efficiency in TinyML applications while maintaining high uncertainty quality. By reusing early-exit block weights for ensemble heads, it significantly reduces computational overhead during inference. It's an original idea which has interesting application and significance. The paper thoroughly evaluates the method on widely used benchmarks, including MNIST, CIFAR, and TinyImageNet. Extensive ablation studies further support the claims, demonstrating strong performance across multiple uncertainty estimation tasks. The method is specifically designed for edge applications where uncertainty estimation might essential (for example, AV context) but computational resources are limited. Its efficiency and effectiveness against standard baselines might make it a valuable contribution to real-world deployment scenarios.

On the side of weaknesses, it was commented that experimental evaluations for failure prediction and accuracy-drop detection are conducted based on single runs without reporting repeated trials or statistical summaries (e.g., mean and standard deviation), reducing confidence in the reliability of results, calibration experiments do not clearly demonstrate the method's superiority over baselines, as the reported advantages are minor relative to the standard deviations provided. Experiments on larger datasets would be appreciated. The paper could benefit from a more detailed discussion of recent related works to better contextualize the contributions, the paper relies mostly on experiments and does not provide a formal theoretical analysis, implying that some theoretical support would make it stronger. While the results are acceptable, more diverse real-world evaluations would strengthen the paper and further show the method’s generalizability.

The rebuttals alleviated some concerns and the reviewers reiterated their general positive evaluation of this paper.
Decision: Accept (poster)