TinyTTA: Efficient Test-time Adaptation via Early-exit Ensembles on Edge Devices
Authors: Hong Jia
Young D. Kwon
Alessio Orsino
Ting Dang
Domenico Talia
Cecilia Mascolo
Venue: NeurIPS 2024 poster (2024)
Keywords: Test-time adaptation
efficiency
edge device
microcontroller
Primary area: infrastructure
TL;DR: Efficient Test-time Adaptation Framework for Microcontrollers along with an MCU TTA library.

Abstract: The increased adoption of Internet of Things (IoT) devices has led to the generation of large data streams with applications in healthcare, sustainability, and robotics. In some cases, deep neural networks have been deployed directly on these resource-constrained units to limit communication overhead, increase efficiency and privacy, and enable real-time applications. However, a common challenge in this setting is the continuous adaptation of models necessary to accommodate changing environments, i.e., data distribution shifts. Test-time adaptation (TTA) has emerged as one potential solution, but its validity has yet to be explored in resource-constrained hardware settings, such as those involving microcontroller units (MCUs). TTA on constrained devices generally suffers from i) memory overhead due to the full backpropagation of a large pre-trained network, ii) lack of support for normalization layers on MCUs, and iii) either memory exhaustion with large batch sizes required for updating or poor performance with small batch sizes. In this paper, we propose TinyTTA, to enable, for the first time, efficient TTA on constrained devices with limited memory. To address the limited memory constraints, we introduce a novel self-ensemble and batch-agnostic early-exit strategy for TTA, which enables continuous adaptation with small batch sizes for reduced memory usage, handles distribution shifts, and improves latency efficiency. Moreover, we develop the TinyTTA Engine, a first-of-its-kind MCU library that enables on-device TTA. We validate TinyTTA on a Raspberry Pi Zero 2W and an STM32H747 MCU. Experimental results demonstrate that TinyTTA improves TTA accuracy by up to 57.6\%, reduces memory usage by up to six times, and achieves faster and more energy-efficient TTA. Notably, TinyTTA is the only framework able to run TTA on MCU STM32H747 with a 512 KB memory constraint while maintaining high performance.

TinyTTA: Efficient Test-Time Adaptation via Early-Exit Ensembles on Edge Devices
Hong Jia
Young D. Kwon 1.4
Alessio Orsino 1.2
Ting Dang
Domenico Talia²
Cecilia Mascolo'
University of Cambridge
University of Calabria University of Melbourne
"Samsung AI Center, Cambridge
[hj359, ydk21, cm542] ac.uk
faorsino, talia]@dimea unical [hong, jia,ting dang) Qunimelb edu.
cal itt
Abstract
The increased adoption of Internet of Things (IoT) devices has led to the gen- eration large data streams with applications healthcare, sustainability, and robotics. In some cases, deep neural networks have been deployed directly on these resource-constrained units to limit communication overhead, increase efficiency and privacy, and enable real-time applications. However, a common challenge in this setting the continuous adaptation of models necessary accommodate changing environments, i.e., data distribution shifts. Test-time adaptation (TTA) has emerged as one potential solution, but its validity has yet to be explored resource-constrainee hardware settings, such as those involving microcontroller units (MCUs). TTA on constrained devices generally suffers from i) memory overhead due the full backpropagation of large pre-trained network, ii) lack support for normalization layers on MCUs, and iti) either memory exhaustion with large batch sizes required for updating poor performance with small batch sizes. In this paper, we propose Tiny TinYTA,, to enable, for the first time, efficient TTA constrained devices with limited memory. To address the limited mem- ory constraints, we introduce a novel self-ensemble and aatto-agnostic early-exit strategy for TTA, which enables continuous adaptation with small batch sizes for reduced memory usage, handles distribution shifts, and improves latency efficiency. Moreover, develop the TinyTTA Engine, a first-of-is-kind MCU library that enables on-device TTA. We validate TinyTTA on a Raspberry Pi Zero 2W and an STM32H747 MCU. Experimental results demonstrate that TinyTTA improves TTA accuracy by up to 57.6 57.6%, reduces memory usage by to six times, and achieves faster and more energy-efficient TTA. Notably, TinyTTA is the only framework able to run TTA on MCU STM32H747 with 512 KB memory constraint while maintaining high performance.
Introduction
Deploying deep neural networks on 1oT devices, such microcontroller units (MCUs), holds significant potential many applications requiring real-time data analysis and low-latency responses, such real-time human health monitoring [I] and robotics [2]. However, practical challenge in deploying these models on MCUs in real-world scenarios their adaptation capability. The data encountered in real settings often exhibit distribution shifts due unforeseen noise and environment. For instance, sensor data can be affected by natural fluctuations and distortions, including weather changes (e.g., fog and snow) and sensor-related issues (e.g.. defocus blur) K 4. Existing mitigation strategies, such as fine-tuning [. typically necessitate updating the entire model, which may be
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

impractical for edge devices due to constraints on memory, energy, and computational resources. Consequently, developing effective adaptation techniques for edge devices address such distribution shifts while maintaining efficiency is critical and pressing need.
Test-time adaptation (TTA) emerges as effective strategy to mitigate generalization issues [ By adapting a pre-trained model to unlabeled test samples, TTA aims to align the model with the current data distribution through entropy minimization. This technique reduces the model's prediction entropy without requiring extensive retraining [6] However, conventional TTA approaches face several limitations that prevent them from being applicable to edge devices. Firstly, TTA typically requires updates to either the entire network or several layers. Even when updating only a few layers, still incurs substantial memory overhead due to the need for backpropagation. This process requires storing activations for large pre- pre-trained model, therefore imposing substantial memory requirements. Secondly, few works have considered mixed distribution shifts and have uniformly updated networks for samples exhibiting varying distribution shifts. This approach not only results in varying accuracy for different levels of distribution shifts but also increases computational overhead, as samples with minor shifts may not, in theory, necessitate updating the entire network. Thirdly, the most effective TTA strategies often involve adapting the normalization layers [3 7 g). However, in practice, these layers are usually fused with convolutional layers when developed for MCUs to reduce computational and memory demands [I This poses practical challenges as currently there is no available library support for normalization layers in such hardware environments. Lastly, the effectiveness of normalization bbaade TTA typically relies on large batch sizes, such as 64 instances per update B), which significantly increases computational overhead. Instead, constrained hardware such as MCUs typically only allow a single batch of data due to limited memory resources. While smaller batch sizes could be more practical for adaptation in MCUs [5], existing approaches suffer from significant performance degradation under these conditions. As a result, there is still a lack of efficient TTA frameworks applicable to resource-constrained edge devices. This challenge even more pronounced for extremely constrained devices, such as MCUs, which play a pivotal role in many applications.
In this paper, we present a novel TTA framework named TInYTTA, which, for the first time, addresses the challenges posed by the efficiency requirements of constrained devices to enable TTA. We show how our design choices allow the framework to apply even to the extreme edge, i..., MCUs. To address the first challenge, we propose an approach based on self-ensemble and early exits. With self-ensembles, we innovatively group subsequent layers of the pre-trained network into submodules, approximating each submodule emulate the full network's capability. The early-exit strategy, combined with these submodules, allows instances to exit from different submodules rather than passing through the entire pre-trained network at test time. For example, samples that are easily predicted with high confidence, which is calculated from the model output, can exit from earlier submodules, alleviating the overhead associated with traversing subsequent layers and reducing the need for further propagation through the entire network. This design also accommodates mixed distribution shifts, addressing the second challenge. Samples with minor distribution shifts can be easily predicted, making a shallow submodulc sufficient for reliable prediction, thus facilitating an early exit. Conversely, samples with significant distribution shifts may necessitate traversing the entire network to achieve reliable adaptation. To address the third and fourth challenges, we propose a weight standardization -WSed [12] adaptation for TTA. This method facilitates batch-agnoste TTA by simulating the effects of normalization layers while alleviating the implementation complexity associated with those layers. Additionally, we construct an operator library, named TinyTTA Engine, to enable practical deployment MCUs.
We conducted extensive experiments to assess TinyTTA performance using four benchmark corruption datasets on two edge devices: the Microprocessor (MPU) Raspberry Pi Zero 2W with 512 MB DRAM and STM32H747 MCU with 512 KB of SRAM. Results show that TinyTTA improves TTA accuracy by up to 57.6%, reduces memory usage by up to six times, achieves faster latency, and is more energy- efficient than conventional baseline methods on MPUs. Notably, TinyTTA the only framework able to run on the MCU STM32H747 with a 512 KB memory constraint while maintaining high performance, opening the door, for the first time, to performing TTA on resource- devices.
2 Related Work
This section reviews the key methods for performing TTA, emphasizing their main limitations. Subsequently, we examine the literature on early exit mechanisms.
2

---

1000
Modusting
Out Mansayy
Firetune
Pistherr
Mennay
Monory (83)
40-
500
of Meneny
500-
Pheoderr Pr 2
10
Roshardo
MCUNM
Barch Bite
Bashh Eize
Banch Bite
(a) Tent memory (ResNet50)
(b) Tent memory (MCUNet)
(c) Tent Accuracy
Figure I: Motivation study. (a)-(b) Both modulating and fine-tuning TTA have similar memory usage and remain memory-intensive, leading to out-of-memory issues during deployment on MPUs and MCUs. (c) TTA accuracy is highly reliant on batch sizes.
TTA. Existing TTA work can be categorized into fine-tuning TTA and modulating TTA. In particular, fine-tuning TTA methods, such COTTA I and Test-Time Training (TTT) 14) updates the entire model architecture, leading to intensive memory usage. Modulating TTA methods, instead, adjust normalization layers while keeping other components frozen, seen in TENT (6).
TIN [5]. NOTE [66.. and SoTTA [7]. However, as shown in Figure Ta and [b] modulating TTA still exhibits memory usage similar fine-tuning TTA, since the model still needs to backpropagate through the full model structure.
Memory-efficient TTA. To address the memory inefficiency of previous TTA methods, various memory-efficient TTA techniques have been proposed, primarily targeting GPUs. These include EATA 77. which filters out high-entropy data to minimize optimization cache, and MECTA (18), which stops backpropagation caching. However, their analyses remain theoretical and impractical, and memory usage has been shown be similar to that of modulating TTA [13]. Even the most efficient TTA method, EcoTTA [3], which achieves memory efficiency on GPUs by distilling the pre-trained model via shallower models, still relies updating normalization layers, which are not supported in MCUs.
TTA under diverse batch sizes. As shown in Figure Ic the success of most TTA methods relies on large batch sizes, due more reliable statistics provided for normalization adaptation. Several methods target batch size 1 for more efficient adaptation, by using different normalization techniques like group norm (GN) 8 layer norm (LN) (3) and adaptive norm (AdaptBN) 13. However, all these methods still exhibit poor TTA performance and require updating many normalization layers, which is computationally expensive and impractical for MCUs.
TTA under changing distribution shifts. Existing TTA methods often overlook the dynamic nature of distribution shifts and apply the same adaptation principle across all samples. However, real-world data on MCUs, such as sensor readings, exhibit highly diverse distribution shifts over time [19]. To date, only few studies have considered the presence of different levels of distribution shifts. For example, NOTE [16] and DELTA [20] employ modified normalization layers, while SAR 3 proposes sharpness-aware TTA. However, these methods still rely on normalization layers and thus are not applicable when the batch size is one for MCU deployment.
Early exit. Early exit strategies have been explored reduce computational costs in various machine learning architectures, particularly for real-time and resource-constrained applications. Notably, BranchyNet (21) introduced early exit mechanisms for deep neural networks, enabling faster inference by adding exit points and allowing models to make predictions at intermediate layers instead of traversing the full model for every input. Similarly, Shallow-Deep Networks 22 aim to reduce inference costs by dynamically selecting the exit point based on input complexity. For these reasons, early exits in the TTA setting can offer significant computational benefits. However, most early exit methods have not been designed to address distribution shifts or batch size constraints typical of MCUs.
3 Efficient TTA via Early-exit Ensembles and TinyTTA Engine
TinyTTA is designed for continuous adaptation various types of distribution shifts in data during inference resource-constrained edge devices. This novel framework, as shown in Figure 2 aims
3

---

Salf-ansemble
(d) TinyTTA
On-device TTA
Engine
Onlina single sample
Pretrained model (e.gi, MCUNe()
BETE
MCU
CNN
Batch Norm
WSip Linear
Exi
Exit
Exit
(a](b((Nd))
Frazen
Ensembled
TinyTTA
MPU
Block-wise
Memory
Backward
(c) WS normalization
(b) Earty-oxits
carbinc bine dog cot bird dog cat brrd dog
Figure 2: TInyTTA framework overview: TinyTTA (a) begins by analyzing and partitioning a pre- trained network into submodules based on memory usage similarity, named self-ensemble network, (b) introduces early-exits for sample inference through specific submodules based on predicted confidence levels, (c) integrates WS normalization into each submodule achieve batch-agnostic normalization, and (d) ultimately compiles and deploys the model on MCUs and MPUs for on-device TTA through the TinyTTA Engine.
to improve system memory efficiency during TTA, adaptability of the pre-trained model to diverse distribution shifts, and deployment feasibility on edge devices. Tiny comprises four modules: i) a self-ensemble network that partitions the network into submodules, each approximating the capability of the full model; ii) early-exits, which allow samples be inferred through specific submodules based on predicted confidence levels. optimizing different submodules for memory and computational efficiency while managing varying levels of distribution shifts; iti) WS normalization, integrated into each submodule to achieve Batch-agnostic normalization; and iv) Tiny TIy Engine, an MCU library for on-device TTA.
3.1
Self-ensemble network
For edge devices such as MCUs, it has been observed that sharing operational layers and weights, for instance through TensorFlow Lite Micro (TFLM) (1], can significantly reduce on-chip memory usage. Instead of utilizing a pre-trained network in its entirety, partitioning it into submodules can offer substantial benefits terms of memory usage. Consequently, we propose self-ensembling pre-trained models by dividing layers into submodules, grouping similar subsequent layers, and approximating each submodule with the full model's capabilities. critical decision in this process is determining the number of partitions required and how to effectively divide the network into submodules. Evidence suggests that i) adjacent layers in neural networks tend exhibit high feature similarities and ii) these layers often have comparable memory consumption [5.. This provides basis for grouping layers with similar memory consumption together as submodules.
We initially conducted an analysis of memory usage during TTA across the layers of pre-trained model, ResNet50. This analysis, as illustrated Figure 3 compares memory usage for both activations and weights between fine-tuning (top figure) and modulation (bottom figure) TTA, aiming to identify layers with similar memory usage. We observed the following: i) memory usage primarily driven by activations, which store the outputs at each layer for computing gradients during backpropagation, while the memory consumed by weights negligible, therefore we focus on activation memory for self-ensembling: ti) in both methods, activations share similar memory consumption and are predominantly concentrated in the initial layers, which generally capture crucial information from the input; tii) certain groups of adjacent layers, specifically layers 1-15, 16-28, 29-44, and 45-52, show similar sizes of activations. Based on this analysis. we group layers with similar memory usage into submodules (i.e., layers 1-15 for submodule 1, 16-28 for submodule 2, 29-44 for submodule 3, and 45-52 for submodule 4) for subsequent early exits and only update the heads at early exits, freezing the submodules to improve memory usage. Indeed, this process does not require backpropagation to the submodules, thus eliminating the need for activation memory. worth highlighting that this analysis shows similar patterns for different model architectures, as discussed in Appendix D
3.2 Early-exits
To further optimize memory usage and manage changing distribution shifts, we introduced early-exit inference combined with self-ensembling submodules. For a given pre-trained model, we sequentially pass inputs through cascaded submodules and infer confidence level, calculated from the model's

---

Flie-tuninn memory
12-
Mamory MMB)
We ght
Activation
high weight memory
activatio mamory
memory
12
18
34
36
46
B5) Modktating per- Ryeer memory usage
Layer
Weight
Activation
Memtoy
achvation
low memory
Layer Indox
Figure 3: Memory usage of different layers during test-time adaptation. Note that the weight memory for modulation (bottom Figure) is < 10 KB, which is negligible. However, both methods show significant activation memory usage. All experiments were conducted using ResNet50 on the CIFAR- 10 dataset with batch size 1.
prediction, for the predictions each stage. If the confidence level is high and exceeds predefined threshold, which is hyper-parameter during TTA (discussed in Appendix D). good prediction achieved, allowing the process to exit at that submodule without passing through the remaining ones. Model updates only occur for the preceding submodules. Conversely, the confidence level remains low, indicating difficulty in prediction to pass through the subsequent submodules for further inference until exits from an appropriate submodule with high confidence. This approach saves memory for easy samples that do not require extensive processing, while effectively handling varying distribution shifts. Different samples are processed differently, and updates optimized according the varying levels of distribution shifts in the samples. Our work is the first to introduce early exits for on-device TTA in edge devices, enhancing both inference efficiency and model accuracy with data distribution shifts.
potential high distribution shift, the instance continues
Specifically, assume K submodules are derived, denoted by θ {{ Each of the submodules is further associated with exits/classifiers, denoted as + = {Ø+ 1 km1 using a linear layer. Given the 7th instance, the output for each submodule is first computed
exp
Σ j-1 exp (2))
where 2 e C denotes the latent representation learned from the hthh submodule, and p represents the predicted probability for classes for the classifier of the 4th submodule after the softmax layer. Given the predicted probability p we can estimate the confidence level using the entropy follows:
C p log 1ogp"
H(x;) = ΣEf
-
A low entropy indicates low uncertainty in the prediction, thus high confidence level. If H H(x;)) is smaller than the threshold 7* for each submodule k this facilitates an early exit of the instance and also the adaptation of the submodules preceding this Athh submodule. Otherwise, further process the instance with the subsequent submodules.
3.3 WS normalization
To entirely omit the usage of normalization layers and enable accurate and memory efffrre-efficient TTA on MCUs, we introduce Weight Standardization (WS) as a replacement for traditional normalization layers 3 7 g for the first time in TTA. Particularly, we propose the inclusion of a WS layer preceding the linear layer attached to each submodule. Therefore, the exits/classifiers of each submodule 6² consist of WS layer and linear layer for C' class classification. Our method adapts only ø² i.e., the WS statistics and one linear layer during the testing phase, while keeping the other components 0 frozen. This avoids model collapse with small batch sizes and minimizes memory usage.
Unlike traditional normalization methods like Batch Normalization (BN) and Group Normalization (GN), which primarily focus recentering and rescaling activations, Weight Standardization (WS)
5

---

targets the smoothing effects on weights. This approach results in more favorable loss landscape and improved parameter updating. Additionally, WS guarantees batch-agnostic normalization, not impacted by the batch size. The WS operation is defined as:
W Pa
W
o +E
Here, Mw and ow represent the mean and standard deviation of the weights calculated over all channels and dimensions, computed as:
(wss 1w)
and Wayj represents the element of the weight matrix for the ish row and juh column. This normaliza- tion applied directly to the early exit CNN layers o². Importantly, WS introduces no additional parameters within the CNN layers and facilitates TTA across different batch sizes. During inference, WS is computed solely from the weights, eliminating the need to accumulate batch statistics on the fly, thus achieving batch-agnostic adaptation. This allows it to be applicable to small batch sizes, thus further ensuring memory and computational efficiency during TTA and making TinyTTA suitable for deployment on MCUs. From an implementation viewpoint, for MCU deployment, we introduce new CNN layer incorporating the WS normalization in our TInyTTA Engine (see Section (..5) to avoid using batch normalization layers, which are not supported by most constrained hardware.
3.4 Training and Inference
To facilitate the learning of submodules 8 and early-exit model parameters Φ², we introduce the concurrent training of all models on the training dataset. Given the source data, we pass them through both the pre-trained model and the submodules combined with early exit branches and compute the losses to optimize the models. The first loss, L1, aims to infer accurate predictions from the submodules and early-exit modules, which can be formulated as the eross-yn (CE) for all classes. The second loss, L2, aims to achieve reliable self enfembisen by aligning the outputs from the submodules to the same as the pre-trained models. The process can be formulated as:
L = CE L2= = 12x 2/
(5)
where Σ represents the features learned from the submodules and zy represents the one from the pre-trained model. The final loss consists of these two losses:
min L = XC1 + (1- 2)
(6)
where À controls the trade-off between the two losses.
The training process is performed offline, without additional training costs on the device. Then, at inference time during the TTA phase, TinyTTA operates without utilizing any source data, as in the standard TTA setting (6] The model initially calculates the entropy of the first submodule's output and compares it with a predefined entropy threshold. Based on this comparison, the exit determines whether the feature outputs should proceed to the subsequent submodules of the self-ensemble model. The process continues if the entropy is greater than the threshold; otherwise, it halts. is worth noticing that after training, only the submodules and early exits are deployed on the device. Once deployed, only carly exit branches are updated on-device, while the rest the model remains frozen, ensuring both high TTA accuracy and low memory usage.
3.5 Tiny TTA Engine
To enable TTA on edge devices like MCUs, we propose TinyTTA Engine as illustrated in Figure 4 The TIA Engine pipeline encompasses several steps. Initially, once the model is loaded, its forward graph constructed. During compile-time, operations are optimized through fusion, and an automatic differentiation (autodiff) process constructs the backward graph needed for updating. Following autodiff, the backward graph optimized via a fuse and quantize procedure, aiming to mitigate resource limitations during execution.

---

Complte time
Execution time
fuse
fuse op
fuse op
freeze Qvantize
freeze & quanttiee
freeze quantizo
Online sensor reading
Load
autodlff
Norm
fp32
1p32
quantiee graph
fo32
Linear
CNN
forward graph
backward graph (b) Autodiff
backwardd graph
WS
(a) TinyTTA processed model
(c) Ensemble graph
[d) On-device TTA
Figure 4: The TInYTTA Engine operates two phases: compile time and execution time. During compile time, (a) given a reprocessed model, it (b) fuses backbone operations to enhance efficiency, then enables backpropagation on TinyTTA exits. Subsequently, (c) it freezes and quantizes the backbone before integration with Tiny TTA exits. Finally, the model is loaded onto the MCU for (d) on-device TTA during execution time.
The cornerstone of TInYTTA Engine lies in its ability to perform backpropagation on-device. We build TinyTTA Engine upon the widely used MCU library TFLM by developing floating-point based backpropagation algorithm. TinyTTA Engine specifically caters to the backward computation needs of prevalent DNN operators such ReLU, FullyConnected. Sof tmax, Maxpool, Avgpool, Conv, and DepthwiseConv. To further optimize memory efficiency, we have developed a layer-wise update strategy reduce memory utilization alongside a dynamic memory allocation mechanism leveraging heap memory. Specifically, the interpreter-basee framework of TinyTTA Engine enables memory reduction by dynamically choosing to save intermediate activations only if a layer needs updates (refer to Appendices C.2 and C.3
The TinyTTA Engine code is publicly available at
httts:///itthbe com//h..coomm.... cww/cmmmohhha jtcp:/hjjjaaaa
4 Experimental Setup
Distribution shifted datasets. For comparison with baselines, our experiments were conducted using four popular corrupted datasets, including domain shift datasets CIFAR [23] and CIFAR
[24] and two domain generalization datasets, namely OfficeHome 25 and PACS 26 More details
about the datasets can be found in Appendix B
Implementation details. To provide comprehensive evaluation of TinyTTA, we conducted exper- iments on both MPUs via a Raspberry Pi Zero 2W and MCUs using an STM32H747. Details of the hardware and MCU implementation can be found in Appendix C The evaluation of TinyTTA on MPUs, which are equipped with relatively large memory, allows for a comparison with other state-of-the-arr (SOTA) methods, thereby offering an in-depth understanding of TinyTTA's algorith- mic performance. For the MPU experiments, we employed three SOTA mobile models. specifically EfficientNet (7), MobileNetV2 (28) and RegNet [29]. In particular, we utilized MobileNetV2 with scale factor of 0.5 (MobileNerV2. _05),, EfficientNet with scaling factor of (Efficienntet b1), and RegNet with a computational cost of 200 million floating-poine operations (RegNet-2200m For the MCU evaluation, we utilized the SOTA model MCUNet [5), which is the only model capable of running within the 512 KB memory constraint of MCUs. More details are discussed Appendix D
Baselines. We compared TinyTTA with several SOTA methods, including: i) TENT (6) which optimizes the affine parameters of batch normalization layers through entropy minimization; ii) COTTA [13, which updates all model parameters using a consistency loss between student and teacher models and stochastically restores the pre-trained model; iit) memory-effiie efficcient baseline EATA 77. which employs sample selection criterion for identifying non-redundant samples and updates the model by minimizing entropy loss; iv) the most memory-efficient method EcoTTA [ which freezes the original network parameters, uses AdaptBN (30 normalization layers and updates attached meta networks for memory-efficiae test-time adaptation.
Evaluation metrics. The same evaluation metrics are utilized across all datasets, encompassing accuracy, memory usage, latency, and energy consumption. Memory usage is quantified using TLnyTL (3I) for the Raspberry Pi Zero 2W and with TFLite Tools for MCUs, with a focus on the tensor arena size. As both the MPU and MCU necessitate loading libraries for TTA, latency is

---

MCUNet
Efficienbeet
MobieNell
RegNet
MC MCUNetsTinyTTA
Emcennteet-TiyTTA
MobieN et-TIyTTA
RegNet-TinyTTA
70
Accuracy
60
513
31A
$1.
53
50
41.0
1OC
CIOAR100C
OfficeHome
PACS
Figure 5: Comparison of model performance across four datasets, demonstrating the accuracy improvements when models adapted using TinyTTA. Across all datasets, TinyTTA consistently boosts accuracy with respect to not adapting the model.
recorded per batch. This measurement is taken after the model stabilizes following 10 warm-up iterations of test samples in TTA. Subsequently, the latency is averaged over 100 batches.
5
Results
This section addresses the following questions: (1) How does TinYTTA improve performance compared the source model without adaptation? (2) How does TinyTTA perform compared to baseline methods on MPUs? (3) How efficient is TinyTTA typical MCUs? (4) What effect do self-ensemble, early-exits, and WS have?
5.1 TInyTTA vs. source models on MPUs
We first compare the performance between TinyTTA and the source model without adaptation MPUs, as shown in F The performance was assessed using four distinct corrupted datasets and four different model architectures. Notably, TTA is evaluated with batch size of one, a challenging scenario for adaptation. Most existing TTA methods generally fail due to unstable statistics associated with single batch for normalization layers. For each combination of model and dataset, TinyTTA demonstrates an average improvement of 4.3%. The most significant enhancement is observed with the EfficientNet model on the CIFAR dataset, where accuracy increases from 49. 1 1% 53.3%. Even under the constraint of adapting to single batch of data during test time, Tiny consistently outperforms models without adaptation, highlighting its potential in real-world scenarios where limited data is available for adaptation.
5.2 TInyTTA vs. SOTA baselines on MPUs
We further compared Tiny TTA with other SOTA basclines across four different datasets and four model architectures on MPUs. The results shown in Figures ba to 6d
For MCUNet, as shown in Figure 6a TinyTTA consistently outperforms other TTA methods across all datasets. Specifically, our method achieves accuracy of 64.3%, 51.6%, 58.1 58.1%, and 63.6% on CIFAR1OC, CIFAR 100C, OfficeHome, and PACS datasets, respectively. In comparison, the second-best performing method, ЕCOTTA, only attains accuracy of 13. %, 5.5%, 6.2%, and 6.3% on the same datasets, significantly deteriorating the performance compared to the source model without adaptation. This is due to the small batch size that causes unstable statistics for adaptation. Notably, on CIFAR 100C, OfficeHome, and PACS datasets, TinyTTA achieves substantial accuracy improvements of 46.6% 6,, 52. 1..1% %. and 57.6% over the second-best method, respectively. In terms of memory usage, Tiny TTA exceptionally efficient, requiring 1.2x less memory than ECoTTA, 2.2x less than EATA, 2.3x less than TENT (Modulating). 4.2x less than TENT (Finc-tuning), and 6.0x less than CoTTA on average across all datasets on MPUs.
For EfficientNet, we observe similar performance trend, as depicted in Figure 6b In terms of accuracy, TinyTTA consistently outperforms other TTA methods across all datasets. On CIFAR10C dataset, TinyTTA achieves accuracy of 67.5%, which is 48.5% higher than the second- best method, COTTA Similarly, on CIFAR 100C, OfficeHome, and PACS datasets, TinyTTA also shows 47.3%, 53.7%, and 56. 7% relative improvements over the second-best method, respectively.

---

(Neesktrn))
Firebnrg!! COTOA
TENT (Mockiding) TENT Pistarng
No Sooooa (N82)
100
13 5
tha Fa
CI CIFRRISNC
Omzztoone ACC
CIARRIC
CH CHARLOOC
Ofccetann PAC3
CIAARI CIFAR100C
Offcetooee PACS
CF IRAC
Offcetieing PACS
MCUNet
EfficientNer
EoOTTA
BEATA MEIITE TEET (Neeukong RZGTEETT Firchargg! MINCOTTA
Tep ZLOTTA EUN CATA TENT IReaa zagl som TEn Pindang Catta
100
Accurccy
Accuenty
Minery (iB)
$
Fm
c# Soc. OF ARLINC
Offcctemns
PAc5
CFARICC
CI CIFRRTOL
Offcettone PACS
CIFARICC CIFAR1SIC
alicetont PACS
CIFAR10C
CR CRARI00C
alicetort 9A0s
MobileNet
(d) RegNet
Figure 6: TTA methods performance comparison of four edge models over four datasets. (a) Tiny (ours) is the only method capable of performing TTA under MCU constraints while maintaining high accuracy. (b)-(d) TinyTTA (ours) achieves the best performance using minimum memory. Results are tested on severity level
Regarding memory usage, our method exhibits the lowest memory footprint among all the compared methods. On the CIFAR 10C dataset, TinyTTA requires 5.65 MB of memory, which is 1.2x lower than ECOTTA (7.06 MB), 2.2x lower than EATA (12.59 MB), 2.3x lower than TENT (Modulating) (13.02 MB), 4.2x lower than TENT (Fine-tuning) (24.0 MB), and 6.0x lower than CoTTA (33.88 MB). The memory usage trend remains consistent across other datasets.
For MobileNet and RegNet in Figures 6c and 6d the results also show a similar trend as MCUNet and EfficientNet in terms of both accuracy and memory usage.
We also compared the latency and energy consumption of the different TTA methods on the Raspberry Pi Zero 2W MPU using the CIFARIOC dataset, which consists of 50,000 samples. The results Table [ highlight the efficiency of Tiny'TA in terms of both latency and energy consumption. Specifically, TinyTTA achieves an inference time of 0.22 seconds per sam- ple (11,000 seconds in total) with a total en- ergy consumption of 6.11 Wh. Compared to EATA, for example, Tiny reduces both la- tency and energy consumption by 12%. On the other hand, CoTTA shows the highest latency (312,500 seconds in total) and energy consump- tion (173.61 Wh), while TENT (both fine-tuning and modulating), though more efficient than COTTA, is still outperformed by Tiny Over- all, TTA offers the best balance between accuracy, memory, speed, and energy efficiency, making it ideal for edge AI applications.
Table 1: Latency (in seconds) and energy consumption (in Joule) per image for vari- ous methods on CIFARIOC on Raspberry Pi Zero 2W MPU.
Method
Latency (sec)
Energy (J)
CoTTA
6.25
12.50
TENT (Finetune)
0.51
1.02
TENT (Modulating)
0.51
1.02
EATA
0.25
0.50
FCCTTA
0.37
0.75
TinyTTA (Ours)
0.22
0.44
5.3 TinyTTA on MCUs
We now show the performance of TInyTTA extremely resource-constraince edge devices, specifi- cally MCUs with only 512 KB of on-chip -iinuoor--ignnicanttt smaller than those of MPUs like the Raspberry Pi Zero 2W. Notably, TinyTTA is the only method capable of performing TTA under the STM32H747 MCU's 512 KB memory constraint while achieving superior accuracy compared to other methods. This makes TinyTTA a promising solution for resource- cowuurcc-coostraineee devices requiring efficient and accurate TTA. We examine two scenarios: with and without TinyTTA ("Inference Only") analyze effectiveness and system overheads. The results are shown in Table
In addition to the superior performance over "Inference Only", as shown in Section 5.2 TinyTTA
reduces overall latency (50.7 ms vs. 55,8 ms) and energy consumption (11.5 mJ vs. 12.7 mJ) per

---

sample thanks to the early exit mechanism, despite requiring on-device model updates. Regarding memory usage, TinyTTA incurs small overheads of 40.2 KB for SRAM and 85.1 KB for Flash. Specifically, the additional SRAM is needed for updated parameters after adaptation, while the early exits re- quire additional heads, taking up stor- age on Flash.
Table 2: MCU deployment of the baseline and TinyTTA on STM32H747 using MCUNet and CIFARIOC.
System
Accuracy
SRAM
Flash
Latency
Energy
Notably, our TinTTT Engine enables fast (low latency) and efficient (low
Inference Only TinyTTA (update)
60.2%
8288B 123KB
290KB
55 85.8ms
12.7mJ
64.3%
375KB
50.7ms
11.5m.J
energy) on on-device adaptation with only a few tens of KBs of extra memory on extremely resource- limited loT devices.
5.4 Ablation Study
Figure fillustrates the trade-offs between accuracy and memory usage (in MB) across different configura- tions of our method for four datasets using MCUNet. The configuration represented by * (TinyTTA us- ing all components: self-ensemble, early-exits, and WS) achieves the best balance, with the highest ac- curacy and optimal memory usage. In contrast, (TinyTTA without WS) shows drop in accuracy by approximately 12-15% and no change in memory us-
70
Belf-eneemblls Early-Exits + WS
Clart0 Ciaar 100
60
Office
50
Assuracy
40-
20
20-
10
age. The (TinyTTA without early exits and WS) configuration reduces accuracy by about 44.5-54.5% and increases memory usage by 5x, highlighting the critical role of early exits. Finally, (without all components) exhibits the lowest accuracy, decreasing by around 40.3-50.3% and the highest memory foot- print increased by 5-6x, underscoring the necessity of integrating all components. The ablation study clearly demonstrates that our method, when leveraging all components, achieves the optimal trade-off for practical applications.
Memory Usage
Figure 7: Ablation study. * represents TinyTTA using all components. represents without WS. represents without early-exits and WS, and means without all compo- nents.
6 Conclusion
This paper presented TinYTTA, which, for the first time, enables TTA on resource-limited edge devices such as MCUs. By introducing self-ensemble framework, early-exit policy, and weight standardization, our method overcomes the limitations of traditional TTA methods, enabling efficient yet competitive on-device TTA edge devices. We also offer a new library, TinyTTA Engine, to unlock the development of TTA for various applications on MCUs.
Limitations and Future Work
TinyTTA has only been tested with a batch size one in the setting to be deployed on a single type of MCU with image data. Additionally, TinyTTA requires retraining the model via self-ensemble. However, the underlying TinyTTA Engine offers a robust foundation for extending its capabilities. We anticipate that with further development, our method and the TinyTTA Engine library could be adapted for a wider array of applications, encompassing various types of MCUs and diverse data modalities beyond just images. Future research will involve investigating the potential of TinyTTA to process different types of data, such as video and inertial measurement unit (IMU) data, thereby broadening its usability across different real-world applications.
Broader Impacts
Deploying TTA on microcontrollers is particularly useful for applications that need quick decision- making and operate in -imoiee, changing environments. Unlike traditional model inference,
10

---

where the model's weights stay the same, TTA requires the model to update its weights based on shifts input data distribution. This makes it challenging maintain accuracy while being deployable on MCUs. Our TinyTTA framework addresses this issue, enabling TTA on MCUs for the first time.
7
Acknowledgment
This work is supported by ERC through Project 833296 (EAR), and Nokia Bell Labs through donation. We also acknowledge financial support from the PNRR MUR project PE0000013-FAIR (Italy) CUP H23C22000860006.
References
[1] Muyang Lin, Ziyang Zhang, Xiaoxiang Gao, Yizhou Bian, Ray S Wu, Geonho Park, Zhiyuan Lou, Zhuorui Zhang, Xiangchen Xu, Xiangjun Chen, et al. A fully integrated wearable ultrasound system to monitor deep tissues in moving subjects. Nature Biotechnology, 42(3):488 457, 2024.
[2] Artúr István Károly, Péter Galambos, József Kuti, and Imre J Rudas. Deep learning in robotics: Survey on model structures and training strategies. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 51(1):266-279 2020.
[3] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In The Fleventh International Conference on Learning Reprexentations, 2023.
[4] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Dehenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robusthench: a standardized adversarial robustness benchmark. In Proceedings of the Neural Information Processing Systems Track Datasets and Bench- marks, 2021.
[5] Ji Lin, Ligeng Zhu, Wei-Ming Chen, Wei-Chen Wang, Chuang Gan, and Song Han. On-device training under 256kb memory. Advances in Neural Information Processing Systems, 35:22941-22955 2022.
[6] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell, Tent: Fully test- time adaptation by entropy minimization. In 9th International Conference on Learning Representations, 2021.
[7] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International conference on machine learning, pages 16888-16905. PMLR, 2022.
[8] Skyler Seto, Barry-John Theobald, Federico Danieli, Navdeep Jaitly, and Dan Busbridge. Realm: Robust entropy adaptive loss minimization for improved single-sample test-time adaptation. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pages 2062-2071, 2024.
[9] Ji Lin, Wei-Ming Chen, Yujun Lin, john cohn, Chuang Gan, and Song Han. Mcunet: Tiny deep learning on iot devices. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1171 1122.. Curran Associates, Inc., 2020.
[10] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and Training of Neural Networks for Efficient Integer- Arithmetic-Only Inference. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[11] Robert David, Jared Duke, Advait Jain, Vijay Janapa Reddi, Nat Jeffries, Jian Li, Nick Kreeger, Ian Nappier, Meghna Natraj, Tiezhen Wang, et al. Tensorflow lite micro: Embedded machine learning for tinyml systems. Proceedings of Machine Learning and Systems, 3:800-811 1, 2021.
[12] Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batch training with batch-channel normalization and weight standardization. arXiv preprint arXiv: 10520, 2019.
[13] Junha Song. Jungsoo Lee, In So Kweon, and Sungha Choi. Ecotta: Memory-efficieen efficient continual test-time adaptation via deff-iistlled regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19920-11929, 2023.
[14] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self supervision for generalization under distribution shifts, In International conference on machine learning, pages 9229-92222 9248. PMLR, 2020.

---

[15] Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. TTN: A domain-shift aware batch normal- ization in test-time adaptation. In The Eleventh International Conference on Learning Representations, 2023.
[16] Taesik Gong, Jongheon Jeong, Taewon Kim, Yowon Kim, Jinwoo Shin, and Sung-Ju Lec. Note: Robust continual test-time adaptation against temporal correlation. Advances in Neural Information Processing Systems, 35:27253-27266, 2022.
[17] Tacsik Gong, Yewon Kim, Tacckyung Lee, Sorn Chottananurak, and Sung-Ju Lec. Sotta: Robust test-time adaptation on noisy data streams. Advances in Neural Information Processing Systems, 36, 2024.
[18] Junyuan Hong, Lingjuan Lyu, Jiayu Zhou, and Michael Spranger. Mecta: Memory economie continual test-time model adaptation. The Eleventh International Conference on Learning Representations. 2022.
[19] Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei W Koh, and Chelsea Finn. Wild-time: A benchmark of in-the-wild distribution shift over time. Advances in Neural Information Processing Systems, 35:10309- 10324, 2022.
[20] Bowen Zhao, Chen Chen, and Shu-Tar Tao Xia. Delta: Degradation -rree fully test sist-time adaptation. In The Eleventh International Conference on Learning Representations, 2022.
[21] Surat Teerapittayanon. Bradley McDanel, and H. T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 23rd International Conference on Pattern Recognition, ICPR 2016, pages 2464 2469. IEEE, 2016.
[22] Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97 of Proceedings of Machine Learning Research, pages 3301 -31100 PMLR, 2019.
[23] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[24] Collin Burns and Jacob Steinhardt. Limitations of post-hoe feature alignment for robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2525 2233,, 2021.
[25] Hemanth Venkateswara. Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018-5027, 2017.
[26] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference computer vision, pages 5542-5550, 2017.
[27] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional ncural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Con- ference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California. USA, volume 97 of Proceedings of Machine Learning Research, pages 6105-6114. PMLR, 2019.
[28] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chicl Chen. Mobilenetv2: Inverted residuals and linear bottlcnecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018
[29] Iliaa Radosavovic, Raj Prateek Kosaraju. Ross Girshick. Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 10428-10458 2020.
[30] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in neural information processing systems, 18:111539-111 11551, 2020.
[31] Han Cai, Chuang Gan, Ligeng 7hu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems, 33:1 1285-11297 2020.
[32] Colby Banbury, Chuteng Zhou, Igor Fedorov, Ramon Matas, Urmish Thakker, Dibakar Gope, Vijay Janapa Reddi, Matthew Mattina, and Paul Whatmough. Micronets: Neural network architectures for deploying tinyml applications on commodity microcontrollers. Proceedings of machine learning and systems, 3:517-532, 2021.
[33] Viet Anh Trinh, Hassan Salami Kavaki, and Michael I. Mandel. Importantaug: A data augmentation agent for speech. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022, pages 8592- 8596. IEEE, 2022.
12

---

Prodicted classees
1000
Pred classes
1000
Prodicted classees
1002
1000
1000
400
500
600
800
000
603
600
400
600
Prod
403
200
10000 20030 50030 40020 Oniine samplss
1.0000 20000 32000 40002 Onine samples
10000 20000 50:00 40000 Onina samples
10000 20000 30000 40000 Onhnne samples
(a) Modulating at level 5
(b) Modulating at level 3
(c) Fine-tuning level 3 (d) Fine-tuning at level
Figure 8: Comparison between modulating and fine-tuning using online test-time entropy minimiza-
tion. (a-b) For larger domain shifts (c.g., severity level 5), the modulating model is more prone to collapse and predict fewer classes. (c-d) For smaller domain shifts (e.g.. severity level 3), modulating more robust than fine-tuning. All experiments are conducted on ImageNet-C for fog noise and ResNet50. Refer to Section B for more details about severity levels.
A Appendix/supplemental material
Entropy Minimization. In test-time adaptation, the objective is to fine-tune the model parameters during the testing phase to better align with the test data distribution without the labels. One technique used for this purpose is entropy minimization [6]. which aims make the model's predictions more confident on the test data. For a classification model, the entropy of the model's 's prediction for data point is calculated using the formula:
C H(6) = - pa log( c=l
where Hÿ)) represents entropy, C' is the number of classes, and Pa. is the predicted probability of the data point belonging to class c.
Modulating and Finetune TTA. Our empirical analyses in Figures 8d and 8c demonstrate the performance of fine-tuning TTA for samples with severity levels 3 and 5, respectively. In contrast, Figures 8a and 8b illustrate the outcomes of modulation-basee methods for the same severity levels. For fine-tuning TTA, while it can adapt to severe data corruption (e.g.. severity level 5) by tuning the entire model, struggles with milder domain shifts (e.g., severity level 3). leading to model collapse where it predicts only a limited range of classes. This suggests that fine-tuning is more effective for significant domain shifts but becomes prone collapse under minor shifts, resulting in limited class predictions. On the other hand, modulation-based TTA methods exhibit different pattern. With milder severity (e.g., level 3) shown Figure 8bb the modulation- based method accurately predicts samples across various classes, indicated by the dispersed points representing the 1000 classes. However, under severe data corruption (e.g., level 5), modulation- to predict only limited number of classes with sequential test samples, illustrated in Figure 8a by the black line showing overlapping predicted classes for a single class. This suggests a paucity in existing methods, indicating that different mechanisms might be required to address varying levels of distribution shifts within a universal framework, which Tiny TTA achieves.
methods tend
B Datasets
CIFARIOC and CIFARI00C feature 15 types of corruption, categorized into four main groups: noise, blur, weather, and digital, each with five severity levels. The OfficeHome dataset com- prises four domains d e (art, produc, real). containing 15,588 examples with dimen- sions (3, 224, 224) and 65 classes. Similarly, the PACS dataset comprises four domains d (art, cartoons, photos, sketches), containing 9,991 examples with dimensions (3, 224, 224) and classes.
13

---

C Implementation
We now introduce the hardware and software implementation of TinyTTA, Following this, we will discuss one of the core tasks of TinYTTA, which is the MCU implementation.
C.1 Hardware
The self-ensemble training phase of our framework was conducted on Linux server equipped with an Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz and NVIDIA Quadro RTX 6000 GPU, where we pre-trained the shared backbone and multiple heads. For the deployment phase, the system components
implemented two types of edge devices. The first MPU was Raspberry Pi Zero 2W featuring an ARM Cortex-A53 processor with distinct memory specifications. The second MCU used is the STM32H747XI, which possesses dual-core (ARM Cortex M4 and M7) architecture. We only use the ARM Cortex M7 core, reflecting a typical single-core MCU design. This choice constrained the SRAM and embedded Flash capacities to 512 KB and 1 MB for STM32H747XI, respectively. These two platforms are representative of a broad spectrum of devices commonly used in edge computing. Specifically, as the MCU is based on the ARM Cortex-M architecture, TinyTTH can be seamlessly deployed across range of similar ARM Cortex-M-base MCUs, demonstrating the versatility of our approach in resource-constraine environments.
C.2 Software
The pre-training stage of TinYTTA, executed standard Linux server, utilized PyTorch version 1.10.2. For the meta-learner module, Python, NumPy, and TensorFlow Lite Micro (TFLM) were employed due to their compatibility and efficient integration. TFLM was specifically selected for its effectiveness and flexibility in deployment across diverse hardware platforms and pre-trained models without re- gegerattting the code that embeds architectures, weights, and execution graphs and logic. The efficacy of Tiny TTA was evaluated by analyzing the required system resources to enable TTA regarding memory usage of SRAM and Flash, end-to-end latency, and energy consumption. The process of transitioning PyTTrcch model to these MCUs with TensorFlow Lite involved a multi-stage conversion: initially ONNX format, then TensorFlow, and finally to TensorFlow Lite, utilizing appropriate converters at each step. The execution of the TensorFlow Lite model on the MCUs was facilitated by TFLM and Mbed OS.
TensorFlow Lite for Microcontrollers (TFLM) | offers a platform tailored for executing machine learning models tiny devices, circumventing the necessity for conventional operating systems or standard C/C++ libraries. TFLM's interpreter- based methodology has its advantages, providing adaptability wide set of edge devices and pre-trained models with minimal runtime overhead for deep neural network computations. However, TFLM's principal shortcoming lies in its incapacity for on-device training, confining models to static state post-deployment for solely inference purposes. Recognizing the advantages of TFLM's minimal runtime overhead and high level flexibility, we developed our TinyTTA Engine framework by augmenting new functionality, on-device adaptation during runtime, to TFLM.
C.3 MCU Implementation
We deployed TinyTTA
TTA on the STM32H747XI The deployment process involves loading the necessary libraries from Mbed os and acquiring the GCC ARM toolchain and the mbed-cli i dev tool. For model management, the TFLM ffamework is set up the project environment, alongside mbed-os components. TFLite Tools are employed to approximate rough SRAM requirements. The model is then converted into C array format suitable for MCU deployment.
El
In addition, we enabled on-device adaptation of Tiny Engine by implementing a backward pass computation and extending this functionality top of the floating-point-asse operators (mentioned in Section 3.5) that previously supported only forward computation in the TFLM framework. In detail, we implemented the gradient calculation regarding weights, biases, and activations C for
14

---

each operator. Furthermore, we developed a layer-wise update logic that allows TinYTTA Engine to select which layer to update during on-device TTA further reduce the memory overhead on SRAM and the required computational costs, as gradient calculation for weights can be omitted for non-updated layers.
Lastly, the increased binary size derived from our backpropagation implementation in TinyTTa is merely 8.6 KB is implemented on top of the existing operator codebase and third-party libraries. This shows that TinyTTA enables on-device TTA on MCUs for the first time, with minimal storage overhead.
C.4 TIAYTA vs. TinyEngine
As our TinyTTA Engine enables on-device training for TTA, in this section we discuss the state-of- the-art on-device training framework for MCUs, known TinyEngine [5]. Specifically, TinyEngine focuses on on-device training (with labeled data), employing automatic differentiation operations at compile-time and utilizing code generation to minimize runtime overhead. The framework statically pre-determines the layers and channels to be updated before deployment, allowing these updates to be executed at runtime. While this approach is efficient, imposes constraints on post-deployment adaptability, as any model update requires recompilation for the target device. Furthermore, due to the code generation process, the model's architecture, weights, and layer dimensions hard-coded into the binary file, which necessitates replacing the entire executable to modify the model.
In contrast, our TInyTTA Engine focuses test-time adaptation with unlabeled data. It enables dynamic adaptation during inference by allowing early exits at submodules, specifically for high- entropy samples, thereby enabling reliable TTA. To adapt TinyEngine for TTA, the only practical solution to use TENT [6, which fine-tunes the model via entropy minimization. our evaluation, compared TinyEngine using TENT a Raspberry Pi Zero 2W (with batch size 1) TTAY measuring performance in terms of accuracy. The experimental setup is the same as described in Appendices B and C. and the results are summarized in Figure 9
Trydrrgia
TryfTt
TayE
"BNTTA
Tredrrgo
TrySrrgn
0.06
2.00
6.21
2.29
3.30
PAC3
6463
CRAASEC
PAcs
PAC7
(a) MCUNet
(b) EfficientNet
(c) MobileNet
(d) RegNet
Figure 9: Comparison of accuracy between TinyEngine (using TENT) and TinyTTA (ours) on different datasets and model architectures.
The results demonstrate that TinyTTA, powered by the TinyTTA TTA Engine, significantly outperforms TinyEngine across all datasets and model architectures due to its ability to dynamically exit high- entropy samples without the need for recompilation, making it more practical solution for real- world deployment, where data distributions may evolve over time.
D
Model size
Compared with ResNet50, which the majority of TTA works focused, models targeting mobile devices are much smaller in terms of model parameters. Specifically, MobileNerV2_ __x5= Efficient- Net_bi and RegNet-200m have parameters totaling 23.71 MB, 24 MB, and 8.85 MB, respectively. In comparison, ResNet50 is approximately 40 times larger than MCUNet in terms of the number of parameters, which only has 2.6 MB. The MCUNet model, optimized for 256KB SRAM and IMB Flash, was pre-trained on ImageNet and fine-tuned on CIFAR-10 10 and CIFAR-100 datasets for 50
15

---

Memory Usage per Layer MobiieNer22
Weight
Monrory (M8)
Mermory Usage Layer in EfficientNet
Weggt
02
00
64
72
70
88
90
100
104
108
112
Layer
Figure 10: Memory usage of different layers for MobileNetV2 and EfficientNet.
epochs, following the methodology outlined in [1. To further enhance the model's adaptability, we fine-tuned MCUNet for an additional 15 layers, enabling one-batch online adaptation.
As shown in Figure I0 similar to ResNer50 (cf. Figure edge models such as MobileNetV2 and EfficientNet exhibit similar layerwise memory usage patterns, where adjacent layers show similar memory usage. For example, in MobileNetV2, we can group layers with similar memory usage into submodules (i.c., layers 1-11 for submodule 12-20 for submodule 21-40 for submodule and 41-52 for submodule 4) for subsequent carly exits. Similarly, for EfficientNet, we can group layers 1-26. 27-42, 43-81, and 82-115 for each submodule. Other per-layer model memory profiling can be accessed via related works, such as MCUNetV3 [5]. where layers are grouped as 1-11, 18, 19-32, and 33-42. RegNet, the layers are grouped as 1-6, 7-10, 11-23, and 24-42.
Regarding the computational overhead introduced by early-exit branches, we analyzed the increase in model size and the number of parameters for the early exits of three different models across various datasets. Note that the last exit layer retains the same memory and parameters as the original model. Table summarizes these results.
Table 3: Memory overhead and number of parameters for early exits across models and datasets.
1st Exit
2nd Exit
3rd Exit
Model
Dataset
Mem (MB)
Params (KB)
Mem (MB)
Params (KB)
Mem (MB)
Params (KB)
CIFARIOC
2.70
0.03
8.46
0.07
17.29
MobileNet
CIFAR100C
11.3
0.10
0.17
43,30
OfficHomme
0.03
8.03
0.07
19.07
0.13
33.19
PACS
0.01
2.41
0.03
7.88
0.06
16.42
CIFARIOC
0.02
5.19
0.17
44.17
0.52
137.10
EfficientNet
CIFAR100C
18.24
87.46
197.67
OfficHHome
0.05
13.17
0.27
70.62
0.66
174.11
PACS
0.02
4.76
0.16
42.73
0.52
135.08
CIFARTOC:
0.10
25.09
0.53
140.22
0.53
140.22
RegNet
CIFAR100C
0.66
173.43
OfficeHome
0.13
33.51
0.61
160.51
0.61
160.51
PACS
0.09
24.63
0.53
139.11
0.53
139.11
Results show that the addition early-exit branches results in relatively low memory overhead and parameter increases for all models. Specifically, i) for MobileNet, the overhead ranges from 0.01 MB 0. 1. MB across different datasets, which is negligible compared to typical memory sizes (e.g.. 512 MB larger memory configurations): ii) EfficientNer exhibits slightly higher memory and parameter increase due its more complex architecture, but still within acceptable limits for most conurec-conttriieed devices, ranging from 0.02 MB to 0.75 MB depending on the dataset and exit
16

---

point; iii) In comparison, RegNet has a memory overhead ranging from 0.10 MB to 0.66 MB, which still remains low relative the overall system memory available on most edge devices.
E
Comparison of WS and GN
Figure presents the performance comparison of TinyTTA using different normalization layers, specifi- cally Weight Standardization (WS) and Group Normal- ization (GN) 8 across varying levels of distribution shifts in the CIFAR-IOC dataset (L1 to L5). The results averaged over three separate runs with different ran- dom seeds. TinYTTA with WS consistently outperforms TinyTTA with GN on all datasets. For instance, on dataset LI. Tiny TinTTTA+ WS achieves an accuracy of 83.2 compared 73.4 for TinyTTA+Gh. This performance gap persists across the other datasets, with TinyTTA+ TS maintaining higher accuracy levels (78.3, 73.6, 68.7, and 64.3) com- pared to TinYTTA+GN (67.2 2, 63.1, 57.9, and 53.2) on L2, L3, L4, and L5, respectively. The observed differences suggest that WS more effective than GN in maintaining model accuracy under distribution shifts, likely due to its ability to better standardize the weights during the adaptation process, thus enhancing the robustness and generalization capability of TinyTTA resource-constraiiee MCUs.
100
TinyTTA+H
TGn
75
Accuracy
L1
L2
L3
L4
L5
Figure 11: Ablation study all levels (L) of distribution shift. Tests are per- formed on CIFARIOC.
F
Impact of Changing Distribution Shifts
When examining the results across different datasets for TinYTTA, shown in Figure 11 it is evident that the severity of distribution shifts significantly impacts the model's performance. As the datasets progress from L1 to L5, there is noticeable decline in accuracy, indicating increasing levels of difficulty in adapting to the distribution shifts. Specifically, Tiny achieves an accuracy of 83.2 on L1, but this performance drops 78.3 on L2. 73.6 6 on L3, 68.7 7 on L4, and finally 64.3 on L5. This downward trend suggests that as the severity of the shifts increases, the model finds progressively harder maintain high accuracy. Despite this challenge, TinyTTA demonstrates relatively strong ability to adapt, maintaining a higher level of accuracy compared to methods using other normalization strategies. This highlights the robustness of WS in handling varying degrees of distribution shifts, making it suitable choice for deployment in resource-constraine microcontroller environments where consistent performance is crucial.
G Choice of Entropy Thresholds
The entropy thresholds represent the level of distribution shift in data during adaptation made by each model. A higher entropy threshold indicates a lower level of confidence in the model's predictions, while lower entropy threshold suggests more confident predictions. The grid search results indicate the optimal entropy thresholds for four models MCUNet, EfficientNet. MobileNet, and RegNet- across four datasets: CIFARIOC, CIFARIOOC OfficeHome, and PACS. For the CIFAR 10C dataset, which comprises 10 classes, the entropy thresholds are 1.12, 1.34, 1.53, and 1.26 for MCUNet, EfficientNet, MobileNet, and RegNet, respectively. For CIFAR 1 10C, containing 100 classes, the thresholds are 3.26, 2.58, 3.1, and 2.31. For the OfficeHome dataset, which has 65 classes, the thresholds are 2.53, 2.16, 2.28, and 2.44. For PACS, the thresholds are 1.58, 1.43, 1.14, and 1.33. These values suggest that entropy thresholds vary significantly depending on the model and the dataset, reflecting the need for model-specific and dataset-specific tuning to achieve optimal performance.
H Experimental Results on Distribution-Shitted Audio Data
In this section, we present the results of our experimental evaluation on distritution audio data, aiming to demonstrate the adaptive capabilities of TinyTTA with data modalities beyond
17

---

images. Specifically, we utilized a pre-trained MicroNets model [32]. which was originally trained on the Speech Commands V2 dataset, achieving an accuracy 86%. This dataset consists of 35 keywords, including "yes," "no," and "forward," among others. To evaluate the robustness of the model real-world scenarios, we tested on the Musan Keywords Spotting test dataset [33], which includes the same 35 speech commands but under various real-world noise conditions such as dial tones, fax machine noises, car idling, thunder, wind, footsteps, rain, and animal noises. The hyperparameters setting for TinyTTA is as follows: a learn- ing rate x 10-5, a batch size of the SGD optimizer with momentum of 0.9, and self-ensemble of early exit layers at depths [3, 5, 7]. Figure [ shows the comparative results of different TTA methods. Specifically, the pre-trained model ex- perienced a significant performance drop, losing approximately 33% accuracy when applied to the distribution-shifted Musan dataset without any adaptation. This highlights the challenge of adapting models real-world noisy environments. Instead, TinyTTA achieved an 8% improvement in accuracy compared no adaptation, demonstrating strong resilience various types of environmental noise. Finally, TinyTTA outperformed all other tested methods, achieving the highest accuracy of 0.61. Notably, the second- -est method, CoTTA, achieved only 0.23 accuracy, underscoring the effectivvness of TinyTTA in main- taining performance under substantial distribution shifts, also with distribution-shifeee audio data.
0.6-
No Adaptatoon.....
Accumcc
0.2
0.0
caTTa
TEAT (Fimmoneg
TENn (Medibttgg)
Eata
ECOTTA
Tmy mond)
Figure 12:
Performance compari- son of MicroNets model on the Mu- san Keywords Spotting test dataset using different TTA methods.
I Comparison with Updating Bias Only
In this section, we explore the effects of fine-tuning only the bias, as implemented in TinyTL. [31, applied the TTA setting. We implemented bias-only fine-tuning strategy via entropy minimization and compared it against adjusting the exits, as proposed in our TinyTTA. Specifically, during TTA. only the model's biases are updated, with no retraining of other parameters. Results in Figure show that fine-tuning only the bias is insufficient for robust TTA performance, whereas integrating early exit branches and updating them in our TinyTTs demonstrates stable performance scenarios involving distribution shifts across different datasets and models.
Diy
Edit
Oney
Dny
pa5s
CP COAKE
edC3
pacs
PACE
(a) MCUNet
(b) EfficientNet
(c) MobileNet
(d) RegNet
Figure 13: Comparison of bias-only fine-tuning vs. exits adjustment across different models and datasets.
J Comparison with non-TTA methods
To better provide understanding of the advantages of implementing TTA in resource-constraing environments, we present the comparison of TinyTTA with non-TTA methods. In particular, we implemented new baseline using Test-Time Training [4] (TTT) with self-supervised rotation classification. We used SGD with momentum (0.9), a learning rate of le 5, an augmentation size of 20, batch size of 1, and À 0.9. Tests performed on the Raspberry Pi Zero 2W.
The experimental results, as shown in Figure 14 demonstrate significant improvement in accuracy across all models and datasets when using TTA compared to applying Test-Time Training. For example, on the CIFAR 1OC dataset, TinyTTA boosts MCUNet's accuracy from 16% 64% and EfficientNet's 's from 18% to 68%. This trend is consistent across all datascts, with TTA providing
18

---

MCUNGHTT ICUUMe-TRyYTTA
EffcenteeHH111
RegheteT1 RegNeciTiySTA
ETRcenakecHiiyyTTT
Mondetiet-T my
60
Accuracy
40
20-
CIFAR1OC
100C
Officetoome
PACS
Figure 14: Comparison of Test-Time Training (TTT) and TinyTTA in terms aceuracy on various models with batch size of one. TinyTTA significantly improves accuracy across all models and datasets.
MCUNe-IT
EffcentkeHHTT ETKciettkttTiiTTTA
RegleeHTT
MobioNatt-
RegHNeisTHyTTAA
Memoryy (669)
20-
10-
CIFAR1OC
CIF 100C
Officel ome
PACS
Figure 15: Comparison of Time Training (TTT) and TinyTTA in terms of memory usage (in MB) on various models with a batch size one. TinYTAA significantly improves accuracy across all models and datasets.
accuracy gains between 40-50% for all models. Interestingly, TinyTTA also reduces memory usage compared to TTT on the Raspberry Pi Zero 2W across all models and datasets, shown in Figure [5] highlighting TinyTTA efficiency not only improving accuracy but also minimizing resource consumption, making it particularly suitable for resource-conssrained environments.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims in the abstract and introduction should accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, the authors discuss the limitations in discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
Not applicable, as this paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, the authors provide sufficient information about their methodology, including the experimental setup and data analysis procedures the Experimental Setup and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Yes, the paper provides sufficient instructions to reproduce the main experi- mental results, described in the supplemental material. The TinyTTA Engine code is fully publicly available at:
com/h h- jiaaiTETTTE
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
Soaiidee/Codeuumissttt
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, the paper provides detailed description of the training and test protocols, including data splits, hyperparameter settings, and the optimization algorithm used in Experimental Setup and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes. The paper includes information about the statistical significance of the experiments in the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етτоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes. The paper provides sufficient details on the computer resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aarrii..ccooouuu.l.aaaaaa
Answer: [Yes]
Justification: Yes. The research conducted in the paper conforms to the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes. The paper discussed the potential beneficial of societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Topic not fit.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Yes. The paper credits asset creators, states licenses and terms, and ensures proper respect.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersai thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
datasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The details of the data, including the URLs, specifics of data generation, and model development, are provided in the Experimental Setup section and the Appendix.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No human involved.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "vHo3CTqvjX", "reviewer": "Reviewer_yW6M", "summary": "In this work, the authors focus on enabling test time adaption on resource limited edge devices. To achieve that, the authors first to train a self-ensemble network where the sub-networks are partitioned according to memory usage. After that, the authors further adopt WS normalization to improve adaption capacity under batch size as one.  In the experiments, the proposed methods show better accuracy with higher efficiency compared to prior test time adaption works.", "questions": "1. The motivation to modulate the pre-trained model according to memory usage is confused.  In the Fig.3, it shows that initial layers occupy much higher memory usage than later layers. However, the authors group the first several layers as first submodule that should be always activated during test time adaption on device, leading to less memory reduction. Will it better to avoid pass through some of the initial layers and active more later layers? \n\n2. One main concern is that the training process happened on device is unclear.  \n\n    (1) The self-ensemble learning has higher training cost. In line 206, the authors mention that \"After training, only the submodules and early exits will be deployed on MCUs.\" It seems that the proposed method first do self-ensemble learning offline which suffers from extra training cost. \n\n    (2) Since the work targets on test time adaption where the source data should be unavailable. How does the authors to train the self-ensemble networks?\n\n    (3) In line 220, the authors said \"The cornerstone of TinyTTA lies in its ability to perform backpropagation on-device\". What is the difference compared to [1]? Which part of the model will be trained on device?  \n\n   [1] On-device training under 256kb memory.\" NeurIPS 2022 \n\n3. The Fig.3 is unclear.  From my understanding, modulation aims to partition the entire model into different groups based on activation memory. Why does activation memory change per layer, and there is no weight memory, compared to \"fine-tuning per-layer memory usage\"? \n\n4. Writing needs to be improved:\n\n   (1) In line 97, \"practical\" should be \"impractical\"\n\n   (2) Line 170 is incomplete\n\n   (3) In line 175, the authors said \"to entirely omit usage of normalization layers and ....  on MCUs\". Isn't WS normalization is a normalization layer used on MCUs?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Although self-ensemble learning and WS normalization is not new, it seems to first use on-device test time adaption.\n\n1. It's interesting to see that adopting WS normalization can achieve better accuracy compared to other normalization layers on the setting of batch size as 1.", "weaknesses": "1. The motivation to modulate the pre-trained model according to memory usage is confused.  \n\n2. The training process on device is unclear.\n\n3. The paper writing needs be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful comments. Please see below our responses.\n\n> Q1: In Fig.3, initial layers occupy much higher memory than later layers. Is it better to avoid pass through some of the initial layers and active more later layers?\n\nA1: Memory usage primarily concerns activations, which store the outputs at each layer. This storage is essential during backpropagation, as computing gradients requires retaining both the outputs and gradient values for each node in each layer. However, we only update the Weight Standardization (WS) layer in the heads at early exits and freeze the submodules after self-ensembling (cf. Fig 2). This process does not require backpropagation to the submodules, thus eliminating the need for activation memory. Consequently, it does not increase memory usage.\n\nAdditionally, avoiding the initial layers is not feasible because these layers capture crucial information from the input. Skipping them would result in incomplete information being learned, leading to degraded performance. We will further clarify this in Appendix A, “Modulating and Finetune TTA”. \n\n>Q2: The authors mention \"After training, only the submodules and early exits will be deployed on MCUs.\" It seems that the method first do self-ensemble offline which suffers from extra training cost.\n\nA2: The self-ensemble model is trained offline, so there is no additional training cost on the device. The training procedure is discussed in Appendix C.2. We will include a heading for the first paragraph as 'Pre-training of Self-ensemble' and further clarify this point.\n\n>Q3: Since source data should be unavailable. How does the authors train the self-ensemble networks?\n\nA3: The self-ensemble networks are trained offline, so we still use the source data, the same as related work in the literature [2]. However, once deployed on the device, the adaptation using TinyTTA does not have access to the source data, adhering to the standard TTA pipeline and the same configuration as in [2]. We will clarify this point in Section 3.1.\n\n>Q4: What is the difference compared to [1]? \n\nA4: We discussed [1] TinyEngine in Appendix C.4. Specifically, TinyEngine focuses on on-device training (with labeled data), whereas TinyTTA Engine (ours) focuses on on-device test-time adaptation (with unlabeled data). TinyEngine pre-determines the layers and channels to be updated before deployment statically (i.e., as a binary file), executing these updates at runtime. In comparison, TinyTTA is dynamic during inference towards exits in submodules, enabling the exiting of high-entropy samples for reliable TTA. To enable TinyEngine for TTA, the only viable solution is using TENT [3] to finetune with entropy minimization. To this end, we compared TinyEngine (dubbed as TE) using TENT on a Raspberry Pi Zero 2W, using batch size 1, with TinyTTA (ours) in terms of accuracy. All configurations are the same as in Appendices B and C. The results are below:\n\n|**Method**|**CIFAR10C**|**CIFAR100C**|**OfficeHome**|**PACS**|\n|----------|------------|-------------|--------------|--------|\n|TE-MCUNet|0.13|0.06|0.07|0.06|\n|**TinyTTA (Ours)-MCUNet**|**0.64**|**0.52**|**0.58**|**0.64**|\n|TE-EfficientNet|0.19|0.11|0.09|0.07|\n|**TinyTTA (Ours)-EfficientNet**|**0.68**|**0.53**|**0.62**|**0.66**|\n|TE-MobileNet|0.18|0.05|0.05|0.06|\n|**TinyTTA (Ours)-MobileNet**|**0.65**|**0.53**|**0.60**|**0.63**|\n|TE-RegNet|0.15|0.12|0.07|0.08|\n|**TinyTTA (Ours)-RegNet**|**0.64**|**0.51**|**0.54**|**0.57**|\n\nWe can make the following observations:\n- Powered by the TinyTTA Engine, TinyTTA generally performs stably as it allows for dynamically exiting high-entropy samples.\n- TE is unable to perform stable TTA across all datasets. \nWe will update the paper in Appendix C.4.\n\n>Q5: Which part of the model will be trained on device?\n\nA5: After deployment, as shown in Fig. 2, only the exits will be updated on-device, while the remaining parts of the model are frozen, ensuring both high TTA accuracy and low memory usage. \n\n>Q6: Why does activation memory change per layer, and no weight memory in Fig 3, compared to \"fine-tuning per-layer memory usage\"?\n\nA6: The primary memory usage for activations is determined by the size of the last output tensor of each layer, essentially storing each layer’s outputs. Since each layer’s output shape is different, their activation memory will accordingly be different. The weight memory is very small (a few KBs) compared to the activation memory. The weight memory usage for modulating TTA, i.e., the change of two parameters, Scale (γ) and Shift (β), is relatively small and not visible in Fig 3. \n\nConsider a single CNN layer with an input image size of 224x224x3 and a batch size of 1. The input activations alone would be around 1 (batch size) * 224 * 224 * 3 (channels) ≈ 1.8 million values, and the output activations after a convolutional layer with 64 filters would be 224 * 224 * 64 (channels/filters)≈ 34.1 million values. In contrast, the convolutional layer weights would only be 64 (channels) * 3 * 3 * 3 (kernel size) ≈1,728 values. This indicates that the sheer volume of activations, especially in layers producing large feature maps, leads to significantly higher memory consumption compared to the relatively small number of weights. We will clarify this in Appendix A.\n\n> Q7: The authors said \"to entirely omit usage of normalization layers ...\". Isn't WS normalization a normalization layer?\n\nA7: We discussed how to deploy Weight Standardization (WS) normalization in lines 189-190 and Fig 4. Specifically, WS will be applied within the CNN exit layer (i.e., a new CNN layer which was introduced by TinyTTA Engine during deployment) to avoid using batch normalization layers. We will further clarify this in Section 3.3.\n\n[1] On-device training under 256kb memory.\" NeurIPS 2022 \\\n[2] EcoTTA: Memory-Efficient Continual Test-time Adaptation via Self-distilled Regularization, CVPR 2023 \\\n[3] Tent: Fully test-time adaptation by entropy minimization, ICLR 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rekepdzcsZ", "reviewer": "Reviewer_DHsU", "summary": "This work presents a test-time adaptation framework for tiny deep neural networks. Specifically, the proposed framework partitions a specific model based on the memory usage of each layer, clusters adjacent layers with similar memory usage into a submodule, and adds an early exit header for each module. To avoid using batch normalization, the authors adopt weight standardization for the early exit header layer. Only the early exit header layer and the corresponding weight standardization parameters are updated during test-time adaptation. The authors also developed an MCU library to support the aforementioned test-time adaptation. The framework can support low-end IoT devices with only 512KB of memory.", "questions": "1. Will you open-source the TinyTTA engine?\n\n2. Is there any real-world case to show that updating the model via backpropagation, instead of simply switching some modes, makes a significant difference for tiny models?", "limitations": "The author discussed the limitations but did not address the potential negative societal impact of their work. This should be fine because, in my opinion, this work does not have any negative societal impact.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Real device deployment: It is great to see that the proposed framework can facilitate the deployment of tiny neural networks on low-end IoT devices with only 512 KB of memory.\n\n2. Well-motivated: The analysis of the memory usage of existing test-time adaptation techniques clearly highlights the drawbacks of previous methods, making the idea of partitioning models based on memory usage quite straightforward.\n\n3. Impressive results: The significantly better accuracy versus memory usage compared to baseline test-time adaptation on four different models is very impressive.", "weaknesses": "1. Lack of discussion on design choices: For example, why can't the \"fine-tune bias only\" technique from TinyTL [28] be used in test-time adaptation? What is its performance compared to only fine-tuning the early exit header proposed in this work? Why is there a \"lack of support for normalization layers on MCUs\"? Since the authors have developed their own MCU library, why can't the normalization layer be added to the library?\n\n2. Limited experiments: As the authors mentioned in the Conclusion section, this work only targets image data. Thus, it is unclear whether the design in this work can be generalized to other applications. For example, will only updating the early exit header be sufficient for other applications?\n\n3. Lack of details on the TinyTTA Engine: Since the algorithm is not entirely novel (i.e., adding multiple early exit headers and weight standardization are not proposed by the authors, but the authors may be the first to use them in test-time adaptation for tiny models), the TinyTTA Engine itself seems to be the key factor ensuring these techniques work efficiently and effectively on real devices. More details and insights from the implementation of the TinyTTA Engine would be greatly appreciated by the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for all your positive comments. Please see below our response to the specific weakness and questions.\n>Q1: Lack of discussion on design choices: why can't the \"fine-tune bias only\" technique from TinyTL [28] be used in test-time adaptation? What is its performance compared to only fine-tuning the early exit header proposed in this work?\n\nA1: TinyTL [28] needs labels for incoming samples to adapt the model, whereas TTA typically handles situations where only samples are available without labels. This is more practical in real-world settings, as providing labels is often challenging due to unforeseen noise and environmental factors. We now conducted \"fine-tune bias only\" via entropy minimization and compared it with adjusting the exits. Specifically, during TTA, only biases are allowed to be updated. The results are below:\n\n|**Method**|**CIFAR10C**|**CIFAR100C**|**OfficeHome**|**PACS**|\n|----------|------------|-------------|--------------|--------|\n|Bias only-MCUNet|0.15|0.09|0.11|0.07|\n|**Exits-MCUNet**|**0.64**|**0.52**|**0.58**|**0.64**|\n|Bias only-EfficientNet|0.13|0.15|0.18|0.09|\n|**Exits-EfficientNet**|**0.68**|**0.53**|**0.62**|**0.66**|\n|Bias only-MobileNet|0.16|0.13|0.15|0.11|\n|**Exits-MobileNet**|**0.65**|**0.53**|**0.60**|**0.63**|\n|Bias only-RegNet|0.18|0.13|0.19|0.17|\n|**Exits-RegNet**|**0.64**|**0.51**|**0.54**|**0.57**|\n\nBased on the experiment, we can observe that: \\\n(1) Adjusting bias alone could not achieve reliable TTA performance. \\\n(2) TinyTTA is relatively stable across four datasets and models. \\\nWe consider that the results are related to the characteristics of TTA, which aims to align data distribution shifts by adjusting the mean and variance. Adjusting the bias alone is insufficient to maintain reliable performance across different datasets and conditions. We will incorporate these new results in Appendix E under a new heading: \"Comparison with Updating Bias Only.\".\n\n>Q2: Why is there a \"lack of support for normalization layers on MCUs\"? Since the authors have developed their own MCU library, why can't the normalization layer be added to the library?\n\nA2: Normalization layers are designed to work with mini-batches of data. However, due to limited memory resources, MCUs typically only allow a single batch of data. As such, normalization layers can technically be added to libraries for MCUs. In practice, the norm layer and the convolutional layer operations are combined into a single convolutional layer operation in order to save the computation and memory as in [1,2]. We will emphasize this further in the paper at Appendix A “Modulating and Finetune TTA.”\n\n>Q3: Limited experiments: As the authors mentioned in the Conclusion section, this work only targets image data. Thus, it is unclear whether the design in this work can be generalized to other applications. For example, will only updating the early exit header be sufficient for other applications?\n\nA3:We conducted a different real-world distribution-shifted audio data modality experiment using a pretrained model on MicroNets [1], which was trained on Speech Commands V2 with 86% accuracy. This dataset contains 35 keywords such as “yes,” “no,” “forward,” etc. We tested the model on the Musan Keywords Spotting test dataset [2], which includes 35 speech commands under real-world noises such as dial tones, fax machine noises, car idling, thunder, wind, footsteps, rain, and animal noises. The setting aims to adapt the pretrained speech command model to real-world scenarios. TinyTTA parameters are: learning rate (lr) = 1e-5, batch size of 1, the SGD optimizer with a momentum of 0.9, and self-ensemble of early exit layers at [3, 5, 7]. The results are as follows:\n|**Method**|**Accuracy**|\n|----------|------------|\n|No Adaptation|0.53|\n|CoTTA|0.21|\n|TENT (Finetuning)|0.05|\n|TENT (Modulating)|0.11|\n|EATA|0.07|\n|ECoTTA|0.23|\n|TinyTTA (Ours)|0.61|\n\nBased on the experiment, we can observe that: \\\n(1) The pretrained model could experience a performance drop of 33% in distribution shift settings. \\\n(2) TinyTTA improved accuracy by 8% over the baseline, showing strong resilience to various noises. \\\n(3) TinyTTA achieved the highest accuracy of 0.61, significantly outperforming other methods (the highest among the state-of-the-art techniques is CoTTA with 0.23).\n\n>Q4: Lack of details on the TinyTTA Engine: Since the algorithm is not entirely novel.\n\nA4: Our approach uniquely combines self-ensembling (frozen after fine-tuning), early exits, and WS normalization. This innovative design significantly improves both inference speed and accuracy. Additionally, our method brings novelty in hardware deployment by introducing the TinyTTA Engine, enabling the first on-device test-time adaptation with unlabeled data. We will provide more details in Appendix C including details of backpropagation, operators, layerwise update strategy, and dynamic memory allocation.\n\n>Q5: Will you open-source the TinyTTA engine?\n\nA5: Yes. As stated in line 683, The TinyTTA Engine Code will be made fully publicly available upon acceptance.\n\n>Q6: Is there any real-world case to show that updating the model via backpropagation, instead of simply switching some modes, makes a significant difference for tiny models?\n\nA6: In realistic scenarios of TTA, we do not have knowledge of the given target domain. Hence, it is difficult to switch to the right model suitable for the target domain. Additionally, note that our target hardware consists of MCUs with extremely limited storage, typically at most 1 MB. Even if we have knowledge of the target domain for a stream of data, we can only store up to 2-3 tiny models (refer to Table 1 for the memory and storage requirements of a single model). We will clarify this in Appendix A.\n\n[1] Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. CVPR2018 \\\n[2] TensorFlow Lite Micro. MLSys 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z4095PvjUO", "reviewer": "Reviewer_2zcM", "summary": "The paper \"TinyTTA: Efficient Test-time Adaptation via Early-exit Ensembles on Edge Devices\" presents a framework designed to enable test-time adaptation (TTA) on resource-constrained IoT devices. TinyTTA utilizes a self-ensemble and batch-agnostic early-exit strategy to adapt to data distribution shifts efficiently with smaller batch sizes, thereby reducing memory usage and improving latency. The approach is validated on Raspberry Pi Zero 2W and STM32H747, demonstrating significant improvements in accuracy, memory efficiency.", "questions": "- Could you possibly add more details about computational overhead of self-ensembles and early exits.\n- What is the motivation for using entropy thresholding in early exit? \n- While you evaluated TinyTTA on four benchmark corruption datasets, these are based on synthetic noise. - How does TinyTTA perform on datasets with real-world distribution shifts or noise?\n- While you mention improvements in energy efficiency, a detailed analysis is not provided. Can you provide a breakdown of energy consumption for different components of TinyTTA, and compare it with baseline methods\n-how do the optimal entropy thresholds in Appendix G differ and impact the performance of the system?", "limitations": "The authors provided a limitations section, and to the best of my understanding, the paper does not have any negative societal impacts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors address the challenges of deploying TTA on devices with stringent memory and computational constraints, which is a less explored area\n2. use of self-ensemble networks and early-exit strategies for TTA on edge devices allowing for adaptive inference based on confidence levels, optimizing both memory usage and computational efficiency.\n3. uses Weight Standardization (WS) as a substitute for traditional normalization layers, specifically tailored for microcontrollers with strict memory limitations.\n4. The paper provides a comprehensive evaluation of the framework across multiple datasets and compares its performance with several state-of-the-art methods. The evaluation covers all important metrics like accuracy, memory usage, latency etc.\n5. The paper is well-structured and written in a clear way explaining all concepts and methodologies. Key concepts such as self-ensemble networks, early-exit strategies, and weight standardization are explained with sufficient clarity\n6. Introduces TinyTTA Engine, a MCU library that enables on device TTA.", "weaknesses": "main:\n- The paper only presents results on vision data, what about other type of data?\n- Implementation Details needed for reproducibility are missing and the code is not provided. E.g, There are not details about how submodules are created for the models utilized in the paper and hyperparameters used during training which hinders the reproducibility. \n- While the results of the experiments are effectively demonstrated in the figures, the readability of the figures can be improved. Specifically, Figure 5 is hard to read. The caption of this figure is also not explanatory enough. Consider briefly summarizing the results in the caption.\n- The comparison was only limited to other TTA methods. Including non-TTA methods might provide a baseline to better understand the advantages of implementing TTA in resource-constrained environments\n- The experiments primarily focus on a specific type of MCU (STM32H747) and one MPU (Raspberry Pi Zero 2W). Is TinyTTA generalizable across other platforms as well?\n- Section 3.1. self-ensemble network, second paragraph: \"(iii) certain groups of adjacent layers, specifically\n145 layers 2-15, 16-28, 29-44, and 45-52, show similar sizes of activations. Based on this analysis, we\n146 group layers with similar memory usage into submodules for subsequent early exits to improve\n147 memory usage: i.e., layers 0-15 for submodule 1, 16-28 for submodule 2, 29-44 for submodule 3, and\n148 45-52 for submodule 4. \". \nThis is\n   -  very model specific: does this generalize to other models?\n   - overly details. Please move such details to a table or so. \n- Section 3.1. second paragraph and also Figure 3: this is well known and, for example, also stated in the MCUNet paper.\n- Details on the early exit are not clear: what layers etc. do the authors use?\n- The ablation study is incomplete: needs to also show early exit with and without model updates. \n- Table 1 should get another line: \"Inference-only with EE\". \n-The paper does not explore the sensitivity of TinyTTA to various hyperparameters.\n-The paper lacks a detailed analysis of adaptation time and energy consumption.\n-The paper does not discuss the challenges and trade-offs involved in deploying the framework\n\nminor: \n* last paragraph of the introduction: 512 KB of SRAM stated twice\n* Section 3.2 Early Exists: \"For a given pre-trained model ,\" -> remove space before comma\n* Figure 5: I suggest using the similar color for the same models, e.g.., similar color for MCUNet and MCUNet+TinyTTA\n* Figure 6: font size of the figures is too small\n* Figure 7: please add units % and KB", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for all your positive comments. Please see below our responses.\n\n>Q1: Other type of data other than vision data?\n\nA1: We tested on the Musan Keywords Spotting audio dataset [2] using a pretrained MicroNets [1] (86% accuracy on Speech Commands V2), which includes 35 speech commands with real-world noises, as below:\n|**Method**|**Accuracy**|\n|----------|------------|\n|No Adaptation|0.53|\n|CoTTA|0.21|\n|TENT (Finetune)|0.05|\n|TENT (Modulating)|0.11|\n|EATA|0.07|\n|ECoTTA|0.23|\n|**TinyTTA (Ours)**|**0.61**|\n\nWe observed:\n- A 33% performance drop in the pretrained model under distribution shifts.\n- TinyTTA achieved the highest accuracy of 0.61 (8% improvement).\n\n>Q2: Implementation Details.\n\nA2: As in line 683, we will provide the source code of TinyTTA Engine. We will further elaborate on Appendix C. \n\n>Q3: Non-TTA method?\n\nQ3:We implemented a new baseline using Test-Time Training (TTT) [3] with self-supervised rotation classification, using SGD (momentum 0.9, lr 1e-5), augmentation size 20 and batch size 1. Memory is tested on a Raspberry Pi Zero 2W.\n\n| **Method** | **Model** | **Accuracy↑** | | | | **Memory (MB)↓** | | | |\n|------------|----------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|\n| | | CIFAR10C | CIFAR100C | OfficeHome | PACS | CIFAR10C | CIFAR100C | OfficeHome | PACS |\n| TTT | MCUNet | 0.16 | 0.05 | 0.07 | 0.07 | 0.41 | 1.35 | 1.32 | 1.33 |\n| | EfficientNet | 0.18 | 0.06 | 0.06 | 0.12 | 12.81 | 37.43 | 37.21 | 37.71 |\n| | MobileNet | 0.17 | 0.07 | 0.08 | 0.10 | 11.43 | 36.27 | 36.71 | 36.58 |\n| | RegNet | 0.15 | 0.12 | 0.06 | 0.07 | 12.28 | 14.33 | 14.45 | 14.20 |\n| TinyTTA (Ours) | MCUNet | **0.64** | **0.52** | **0.58** | **0.64** | **0.2** | **0.73** | **0.71** | **0.72** |\n| | EfficientNet | **0.68** | **0.53** | **0.62** | **0.66** | **5.65** | **16.94** | **16.97** | **16.91** |\n| | MobileNet | **0.65** | **0.53** | **0.60** | **0.63** | **5.58** | **16.74** | **16.79** | **16.77** |\n| | RegNet | **0.64** | **0.51** | **0.54** | **0.57** | **6.13** | **6.25** | **6.28** | **6.22** |\n\n- TinyTTA outperforms TTT by ~50% across four datasets and models.\n- TTT requires, on average, double the memory compared to TinyTTA.\n\n>Q4: The experiments focus on specific type of MCU and MPU. Generalizeable to other chips?\n\nA4: Yes, we selected these two platforms as they represent a range of devices. Specifically, since our MCU is based on Core M processors, TinyTTA can be directly deployed on ARM Cortex-M-based MCUs. We also chose one of the smallest Raspberry Pi to ensure that other Pis can also run TinyTTA. Further clarification will be provided in Appendix C.1.\n\n>Q5: Ablation study of early exit w/ and w/o model updates.\n\nA5: The accuracy of the model w/o updates is discussed in Fig. 5. The memory usage (MB) w/o updates is discussed below.\n\n| Model | CIFAR10C | | CIFAR100C | | OfficeHome | | PACS | |\n|-------|----------|-----------|------------|-----------|------------|-----------|-------|-----------|\n| | w/o | w/ | w/o | w/ | w/o | w/ | w/o | w/ |\n| MCUNet | 0.189 | 0.2 | 0.726 | 0.73 | 0.69 | 0.71 | 0.711 | 0.72 |\n| EfficientNet | 4.94 | 5.65 | 15.78 | 16.94 | 15.99 | 16.97 | 15.93 | 16.91 |\n| MobileNet | 5.47 | 5.58 | 16.43 | 16.74 | 16.56 | 16.79 | 16.54 | 16.76 |\n| RegNet | 5.81 | 6.18 | 4.44 | 6.25 | 4.47 | 6.28 | 4.41 | 6.22 |\n\n- The memory usage w/ and w/o adaptation is very similar.\n- TinyTTA generally requires limited memory to perform on-device TTA. \nWe will update Fig. 7 and Section 5.4.\n\n>Q6: Computational overhead of self-ensembles and early exits?\n\nA6: Self-ensembles are conducted offline (line 465). We analysed the overhead in memory and the number of parameters of the early exits:\n\n|Model|Dataset|1st Exit||2nd Exit||3rd Exit||\n|---|---|---|---|---|---|---|---|\n|Model|Dataset|Memory (MB)|Params (KB)|Memory (MB)|Params (KB)|Memory (MB)|Params (KB)|\n|MobileNet|CIFAR10C|0.01|2.70|0.03|8.46|0.07|17.29|\n| |CIFAR100C|0.04|11.43|0.10|25.83|0.17|43.30|\n| |OfficeHome & PACS|0.03|8.03|0.07|19.07|0.13|33.19|\n|EfficientNet|CIFAR10C|0.02|5.19|0.17|44.17|0.52|137.10|\n| |CIFAR100C|0.07|18.24|0.33|87.46|0.75|197.67|\n| |OfficeHome & PACS|0.05|13.17|0.27|70.62|0.66|174.11|\n|RegNet|CIFAR10C|0.10|25.09|0.53|140.22|0.53|140.22|\n| |CIFAR100C|0.15|38.86|0.66|173.43|0.66|173.43|\n| |OfficeHome & PACS|0.13|33.51|0.61|160.51|0.61|160.51|\n\nAdding new prediction heads results in minimal memory increase, e.g., MobileNet's overhead is 0.01 MB to 0.17 MB, negligible compared to 512 MB MPU memory.\nWe will extend this in Appendix D.\n\n>Q7: Motivation for using entropy thresholding? Sensitivity of hyperparameters?\n\nA7: Entropy thresholding, as used in many TTA methods like [4], avoids high-entropy, less reliable samples to maintain TTA performance. Hyperparameters, including the entropy threshold, are determined post hoc [3] and discussed in Appendix G. We will add a table of layer exits and other hyperparameters in Appendix D.\n\n>Q8: Energy consumption compared baseline methods?\n\nA8: We now compared the latency and energy consumption on Raspberry Pi zero 2 W MPU using CIFAR10C:\n\n| Method |  | CIFAR10C (50000 images)| |\n|--------|--------------------------|------------------|--------------|\n| | | Latency (seconds) | Energy (Wh) |\n| CoTTA | | 312,500 | 173.61 |\n| TENT (Finetune) | | 25,500 | 14.17 |\n| TENT (Modulating) | | 25,500 | 14.17 |\n| EATA | | 12,500 | 6.94 |\n| ECoTTA | | 18,850 | 10.47 |\n| TinyTTA (Ours) | | 11,000 | 6.11 |\n\n- TinyTTA has an inference time of 0.22 seconds per sample and energy consumption of 0.122 mWh, showing high efficiency.\n- TinyTTA outperforms baselines, reducing latency by 12% (1500 seconds) and energy consumption by 12% (0.83 Wh) compared to EATA.\n\n\n[1] MicroNets. MLSys 2021 \\\n[2] Importantaug. ICASSP 2022 \\\n[3] Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. ICML 2020 \\\n[4] Efficient Test-Time Model Adaptation without Forgetting. ICML 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ILT8qIecpH", "reviewer": "Reviewer_bEbZ", "summary": "The paper \"TinyTTA: Efficient Test-time Adaptation via Early-exit Ensembles on Edge Devices\" presents a combination of test-time Adaptation of pre-trained models and early-exist strategies for the efficient inference of Deep Learning models on edge devices. The authors introduce the ideas of test-time Adaptation (i.e., some parameters of the model are changed during deployment by, e.g., entropy minimization) and model splitting by introducing \"early exists\" nodes in the model. Most notably, the authors introduce a novel weight normalization that does not rely on batch statistics (e.g. BatchNorm) but re-weights and re-scales the weights of the model directly. Finally, the authors briefly present their novel inference engine, TinaTTA, which leverages the author's ideas into a usable framework that is used for the experiments. The authors showcase some impressive results on established image datasets (CIFAR10C, CIFAR100C, OfficeHome, PACS) and known small devices (Raspberry et al. two and a dual-core embedded Arduino). In particular, the present method has low latency and low energy consumption while often offering a much better accuracy than existing methods.", "questions": "1) What is the overhead in memory/number of parameters by adding new prediction heads? Did you explore this? \n2) How did you analyze the activations for grouping the models into sub-modules? Did you apply some principle here, or was it an ad-hoc grouping? \n3) What do you mean when you write, \"activations consume much more memory compared to weights\"?", "limitations": "The authors discuss limitations in the evaluation of their method (image data only, only one MCU), but the evaluation is generally well-done and, hence, quite thorough.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The overall idea of bringing test time adaptation through early exists too small devices is good\n- The TinyTTA engine and the experimental evaluation seem impressive and well done", "weaknesses": "Overall, this is a very engineering-focused paper without much methodological impact. Therefore, it can be questioned whether it fits the NeurIPs conference. More specifically: \n- The authors propose self-ensembling, which basically means splitting the network into smaller submodules. The idea is very straightforward and not really studied in depth in the paper. It is unclear how to split the network (the authors mention using activation as a guiding tool, but it is unclear how they do it exactly). Similar ideas have already been discussed in the literature (see next comment), which are not mentioned in the paper. \n- The authors propose early-exists, which is not a novel technique but is well-known in the literature already [1,2,3]. As far as I remember, this technique has already been used since Yolo v5 (although mainly for improved training, and this might be debatable) [4]. Unfortunately, the Related Work section focuses on test-time adaptation (Which is good. I learned a lot here, thanks!) but misses out on the related work on the early-exit networks. Hence, it is unclear to me to what extent self-ensembling and early-exists is really new or just a re-brand of existing methods\n- While the evaluation is generally good, it does not highlight the memory overhead of introducing early exists. At the very least, the authors have to introduce some classification heads for each exit, which requires additional parameters. I could not find a mention of this in the paper, but maybe I missed it (see my question).\n- Minor issues with the paper \n\t- Citations can be improved: There are a few arxiv papers that have been published already. I suggest using dblp for high-quality references (Note: There might be more than these three; I stopped checking after three):\n\t\t- Tent: Fully Test-time Adaptation by Entropy Minimization is an ICLR 2021 paper I think\n\t\t- Towards Stable Test-Time Adaptation in Dynamic Wild World is an ICLR 2023 paper I think\n\t\t- RobustBench: a standardized adversarial robustness benchmark is a NeurIPs 2021 Paper I think\n\t- The authors mention data distributional shifts multiple times in the paper. I suggest to clarify this a bit more, since it is unclear to me if we look at shift in the label space or data space. \n\t- In section 3.1, the sentence \"[...] and approximating each submodule with the full model's capabilities\" is unclear. See my question.\n\t- In section 3.1, the merging of subsequent layers into submodules is not really clear. See my question.\n\t- The authors mention that \"activations consume much more memory compared to weights\", which is also unclear to me. See my question.\n\n[1] Why should we add early exits to neural networks? by Scardapane etal. 2020 https://arxiv.org/abs/2004.12814\n\n[2] BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks by Teerapittayanon etal. 2017 https://arxiv.org/abs/1709.01686\n\n[3] T-RECX: Tiny-Resource Efficient Convolutional neural networks with early-eXit by Ghanathe and Wilton 2023 https://arxiv.org/pdf/2207.06613\n\n[4] TPH-YOLOv5: Improved YOLOv5 Based on Transformer Prediction Head for Object Detection on Drone-captured Scenarios by Zhu etal. 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for all your positive comments. Please see below our response to the specific weakness and questions.\n\n>Q1: Unclear how to use activation memory to split the network. How did you analyze the activations for grouping the models into sub-modules? Did you apply some principle here, or was it an ad-hoc grouping?\n\nA1: Per-layer memory profiling is based on TinyTL[1], as discussed in lines 141-148 and 253-254. The grouping is post-hoc by visualizing the similar memory usage of adjacent layers as discussed in line 146. We will further elaborate this in Appendix D.\n\n>Q2: Proposed early-exists is not a novel technique but is well-known in the literature. What extent self-ensembling and early-exists is really new or just a re-brand of existing methods?\n\nA2: We acknowledge that early-exit networks have been explored in the literature, primarily for inference efficiency. However, they have never been explored under data distribution shifts and for adaptation purposes. Our work is the first to introduce early exits for on-device test-time adaptation in edge devices, enhancing both inference efficiency and model accuracy with data distributional shifts. Specifically, our approach innovatively integrates weight standardization (WS) within the heads of early exits and designs these heads for subsequent modules. This ensures adaptation relies solely on WS in the heads, maintaining both inference speed and model accuracy with data distributional shifts. Moreover, our approach uniquely combines self-ensembling (frozen after fine-tuning), early exits, and WS normalization. This innovative design significantly improves both inference speed and accuracy. Additionally, our method brings novelty in hardware deployment by introducing the TinyTTA Engine, enabling the first on-device test-time adaptation with unlabeled data.\n\nWe will include more references on early-exit networks in the Related Work section and further clarify this novelty in Section 3.2.\n\n>Q3: While the evaluation is generally good, it does not highlight the memory overhead of introducing early exists.\n\nA3: We now conducted experiments to explore the overhead in memory and the number of parameters by adding new prediction heads. The results are below:\n\n| Dataset | Model | 1st Exit Memory (MB) | 1st Exit Params (K) | 2nd Exit Memory (MB) | 2nd Exit Params (K) | 3rd Exit Memory (MB) | 3rd Exit Params (K) |\n|---------|-------|----------------------|---------------------|----------------------|---------------------|----------------------|---------------------|\n| CIFAR10C | MobileNet | 0.0103 | 2.70 | 0.0323 | 8.46 | 0.0660 | 17.29 |\n| CIFAR100C | MobileNet | 0.0436 | 11.43 | 0.0985 | 25.83 | 0.1652 | 43.30 |\n| OfficeHome & PACS | MobileNet | 0.0306 | 8.03 | 0.0728 | 19.07 | 0.1266 | 33.19 |\n| CIFAR10C | EfficientNet | 0.0198 | 5.19 | 0.1685 | 44.17 | 0.5230 | 137.10 |\n| CIFAR100C | EfficientNet | 0.0696 | 18.24 | 0.3336 | 87.46 | 0.7540 | 197.67 |\n| OfficeHome & PACS | EfficientNet | 0.0502 | 13.17 | 0.2694 | 70.62 | 0.6642 | 174.11 |\n| CIFAR10C | RegNet | 0.0957 | 25.09 | 0.5349 | 140.22 | 0.5349 | 140.22 |\n| CIFAR100C | RegNet | 0.1482 | 38.86 | 0.6616 | 173.43 | 0.6616 | 173.43 |\n| OfficeHome & PACS | RegNet | 0.1278 | 33.51 | 0.6123 | 160.51 | 0.6123 | 160.51 |\n\nAccording to the results of the above table, we observe the following:\n- Adding new prediction heads results in a minimal memory increase. For instance, MobileNet shows an overhead ranging from 0.01 MB to 0.17 MB (with parameters ranging from 2.7K to 43.30K), which is negligible compared to the 512 MB MPU memory.\n- Different models exhibit varying overheads. EfficientNet shows the highest increase in both memory (up to 0.75 MB) and parameters compared to MobileNet and RegNet.\n- Overall, compared to the 512 MB MPU memory deployed, the overhead remains relatively small. We will extend this discussion in Appendix D.\n\n>Q4: Citations can be improved: There are a few Arxiv papers that have been published already.\n\nA4: We will thoroughly review our paper and update our citations using DBLP to ensure high-quality references. \n\n>Q5: The authors mention data distributional shifts multiple times in the paper. I suggest to clarify this a bit more, since it is unclear to me if we look at shift in the label space or data space.\n\nA5: The shift occurs in the data space. As illustrated in Fig. 2, for an image classification task with an image as the input, the leftmost cat image shows a small distribution shift, as it still clearly depicts the cat; in this case, the severity is 1. Gradually, in the second and third images, the cat’s image becomes noisier and harder to discern with the human eye, corresponding to severities of 3 and 5, respectively. We will further clarify these distributional shifts in the data space in Section 3.\n\n>Q6: What do you mean when you write, \"activations consume much more memory compared to weights\"?\n\nA6: Activations consume much more memory than weights in neural networks because the value of activations need to be stored for every layer during backpropagation, especially when using large batch sizes and deep networks. For instance, consider a single CNN layer with an input image (e.g., CIFAR100C) size of 224x224x3 and a batch size of 1. The input activations alone would be around 1 (batch size) * 224 * 224 * 3 (channels) ≈ 1.8 million values, and the output activations after a convolutional layer with 64 filters would be 224 * 224 * 64 (channels/filters)≈ 34.1 million values. In contrast, the convolutional layer weights would only be 64 (channels/filters) * 3 * 3 * 3 (kernel size) ≈1,728 values. This indicates that the sheer volume of activations, especially in layers producing large feature maps, leads to significantly higher memory consumption compared to the relatively small number of weights. We will clarify this in Appendix A.\n\n[1] TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning. NeurIPS 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers and meta reviewers,

We appreciate all the positive comments of our work:
- Reviewer yW6M: First time using WS and self-ensemble for on-device test time adaptation.
- Reviewer DHsU: Well-motivated, impressive results, and real device deployment on low-end IoT devices with only 512 KB of memory.
- Reviewer 2zcM: Comprehensive evaluation, well-structured and written, and novel TinyTTA Engine.
- Reviewer bEbZ: First TTA for small devices; TinyTTA Engine is impressive and well evaluated.

We have addressed all the comments by providing more clarifications and new results:
- Reviewer yW6M: We have clarified memory usage in initial layers, self-ensemble training, on-device updating components, activation memory, and WS normalization. New experiment: Comparison with on-device training.
- Reviewer DHsU: We have clarified fine-tuning bias only, normalization layers on MCUs, TinyTTA engine, and switching some models instead of TTA. New experiments: real-world different data modality. Comparison with "fine-tune bias only."
- Reviewer 2zcM: We have clarified implementation details and the motivation of entropy thresholding. New experiments: real-world different data modality, new non-TTA baseline, and ablation study of early exits.
- Reviewer bEbZ: We have clarified the principle to group activation, novelty of exits, citations, and activation memory. New experiment: memory of exits.

Detailed Q&As are listed below. We look forward to further discussions and feedback.

Meta-review: We received author rebuttals, and all reviewers discussed the paper based on the rebuttals.
This paper discusses a test-time adaptation approach (self-ensemble and batch-agnostic early-exit strategy), using resource-constrained edge devices such as Raspberry Pi Zero 2W and STM32H747.
Considering the reviews and author-reviewer discussions, I recommend accepting this paper for NeurIPS'24.

For positive aspects of this work, I agree with the reviewers to many of their comments such as performance evaluations for TTA on the extremely resource-constrained devices, offering an MCU library, and improved tradeoff between model accuracy and memory usage.

For downsides, Reviewers yW6M's concerns are mainly about writing quality. Reviewer DHsU pointed out other weakness (evaluations solely on image data). Those drawbacks seem addressed through the discussions though the authors still should address them in the paper.
Reviewer 2zcM shared similar concerns. From their discussion with the authors, it seems like that the authors' rebuttal addressed their questions, and they did not disagree to the answers. 
Last not least, weaknesses suggested by Reviewer bEbZ are fair points, and in fact this paper may look engineering-focused. I see that the key contribution of this work is not only introducing on-device TTA for extremely resource-constrained devices but also offering the library, which should be unique contribution and of community's interest.

Overall, I think that advantages of this work slightly outweigh the disadvantages. In other words, my recommendation for acceptance is conditioned on 1) the revisions that the authors promised below and 2) the public release of the library with detailed documentations for the community. If the authors are not ready for releasing the library with the proper documentations, I suggest that they start working on documentation as soon as possible.
Decision: Accept (poster)