Title: TinyTTA: Efficient Test-time Adaptation via Early-exit Ensembles on Edge Devices
Venue: NeurIPS 2024
Accepted: yes
Keywords: Test-time adaptation

efficiency

edge device

microcontroller
Abstract: The increased adoption of Internet of Things (IoT) devices has led to the generation of large data streams with applications in healthcare, sustainability, and robotics. In some cases, deep neural networks have been deployed directly on these resource-constrained units to limit communication overhead, increase efficiency and privacy, and enable real-time applications. However, a common challenge in this setting is the continuous adaptation of models necessary to accommodate changing environments, i.e., data distribution shifts. Test-time adaptation (TTA) has emerged as one potential solution, but its validity has yet to be explored in resource-constrained hardware settings, such as those involving microcontroller units (MCUs). TTA on constrained devices generally suffers from i) memory overhead due to the full backpropagation of a large pre-trained network, ii) lack of support for normalization layers on MCUs, and iii) either memory exhaustion with large batch sizes required for updating or poor performance with small batch sizes. In this paper, we propose TinyTTA, to enable, for the first time, efficient TTA on constrained devices with limited memory. To address the limited memory constraints, we introduce a novel self-ensemble and batch-agnostic early-exit strategy for TTA, which enables continuous adaptation with small batch sizes for reduced memory usage, handles distribution shifts, and improves latency efficiency. Moreover, we develop the TinyTTA Engine, a first-of-its-kind MCU library that enables on-device TTA. We validate TinyTTA on a Raspberry Pi Zero 2W and an STM32H747 MCU. Experimental results demonstrate that TinyTTA improves TTA accuracy by up to 57.6\%, reduces memory usage by up to six times, and achieves faster and more energy-efficient TTA. Notably, TinyTTA is the only framework able to run TTA on MCU STM32H747 with a 512 KB memory constraint while maintaining high performance.

TinyTTA: Efficient Test-Time Adaptation via Early-Exit Ensembles on Edge Devices

Abstract

The increased adoption of Internet of Things (IoT) devices has led to the generation of large data streams with applications in healthcare, sustainability, and robotics. In some cases, deep neural networks have been deployed directly on these resource-constrained units to limit communication overhead, increase efficiency and privacy, and enable real-time applications. However, a common challenge in this setting is the continuous adaptation of models necessary to accommodate changing environments, i.e., data distribution shifts. Test-time adaptation (TTA) has emerged as one potential solution, but its validity has yet to be explored in resource-constrained hardware settings, such as those involving microcontroller units (MCUs). TTA on constrained devices generally suffers from i ) memory overhead due to the full backpropagation of a large pre-trained network, ii ) lack of support for normalization layers on MCUs, and iii ) either memory exhaustion with large batch sizes required for updating or poor performance with small batch sizes. In this paper, we propose TinyTTA, to enable, for the first time, efficient TTA on constrained devices with limited memory. To address the limited memory constraints, we introduce a novel self-ensemble and batch-agnostic early-exit strategy for TTA, which enables continuous adaptation with small batch sizes for reduced memory usage, handles distribution shifts, and improves latency efficiency. Moreover, we develop the TinyTTA Engine, a first-of-its-kind MCU library that enables on-device TTA. We validate TinyTTA on a Raspberry Pi Zero 2W and an STM32H747 MCU. Experimental results demonstrate that TinyTTA improves TTA accuracy by up to 57.6%, reduces memory usage by up to six times, and achieves faster and more energy-efficient TTA. Notably, TinyTTA is the only framework able to run TTA on MCU STM32H747 with a 512 KB memory constraint while maintaining high performance.

1 Introduction

Deploying deep neural networks on IoT devices, such as microcontroller units (MCUs), holds significant potential in many applications requiring real-time data analysis and low-latency responses, such as real-time human health monitoring [1] and robotics [2]. However, a practical challenge in deploying these models on MCUs in real-world scenarios is their adaptation capability. The data encountered in real settings often exhibit distribution shifts due to unforeseen noise and environment. For instance, sensor data can be affected by natural fluctuations and distortions, including weather changes (e.g., fog and snow) and sensor-related issues (e.g., defocus blur) [3, 4]. Existing mitigation strategies, such as fine-tuning [5], typically necessitate updating the entire model, which may be

impractical for edge devices due to constraints on memory, energy, and computational resources. Consequently, developing effective adaptation techniques for edge devices to address such distribution shifts while maintaining efficiency is a critical and pressing need.

Test-time adaptation (TTA) emerges as an effective strategy to mitigate generalization issues [6]. By adapting a pre-trained model to unlabeled test samples, TTA aims to align the model with the current data distribution through entropy minimization . This technique reduces the model's prediction entropy without requiring extensive retraining [6]. However, conventional TTA approaches face several limitations that prevent them from being applicable to edge devices. Firstly, TTA typically requires updates to either the entire network or several layers. Even when updating only a few layers, it still incurs substantial memory overhead due to the need for backpropagation. This process requires storing activations for a large pre-trained model, therefore imposing substantial memory requirements. Secondly, few works have considered mixed distribution shifts and have uniformly updated networks for samples exhibiting varying distribution shifts. This approach not only results in varying accuracy for different levels of distribution shifts but also increases computational overhead, as samples with minor shifts may not, in theory, necessitate updating the entire network. Thirdly, the most effective TTA strategies often involve adapting the normalization layers [3, 7, 8]. However, in practice, these layers are usually fused with convolutional layers when developed for MCUs to reduce computational and memory demands [9-11]. This poses practical challenges as currently there is no available library support for normalization layers in such hardware environments. Lastly, the effectiveness of normalization-based TTA typically relies on large batch sizes, such as 64 instances per update [3], which significantly increases computational overhead. Instead, constrained hardware such as MCUs typically only allow a single batch of data due to limited memory resources. While smaller batch sizes could be more practical for adaptation in MCUs [5], existing approaches suffer from significant performance degradation under these conditions. As a result, there is still a lack of efficient TTA frameworks applicable to resource-constrained edge devices. This challenge is even more pronounced for extremely constrained devices, such as MCUs, which play a pivotal role in many applications.

In this paper, we present a novel TTA framework named TinyTTA, which, for the first time, addresses the challenges posed by the efficiency requirements of constrained devices to enable TTA. We show how our design choices allow the framework to apply even to the extreme edge, i.e., MCUs. To address the first challenge, we propose an approach based on self-ensemble and early exits. With self-ensembles, we innovatively group subsequent layers of the pre-trained network into submodules, approximating each submodule to emulate the full network's capability. The early-exit strategy, combined with these submodules, allows instances to exit from different submodules rather than passing through the entire pre-trained network at test time. For example, samples that are easily predicted with high confidence, which is calculated from the model output, can exit from earlier submodules, alleviating the overhead associated with traversing subsequent layers and reducing the need for further propagation through the entire network. This design also accommodates mixed distribution shifts, addressing the second challenge. Samples with minor distribution shifts can be easily predicted, making a shallow submodule sufficient for reliable prediction, thus facilitating an early exit. Conversely, samples with significant distribution shifts may necessitate traversing the entire network to achieve reliable adaptation. To address the third and fourth challenges, we propose a weight standardization (WS)-based [12] adaptation for TTA. This method facilitates batch-agnostic TTA by simulating the effects of normalization layers while alleviating the implementation complexity associated with those layers. Additionally, we construct an operator library, named TinyTTA Engine , to enable practical deployment on MCUs.

Weconducted extensive experiments to assess TinyTTA performance using four benchmark corruption datasets on two edge devices: the Microprocessor (MPU) Raspberry Pi Zero 2W with 512 MB DRAM and STM32H747 MCU with 512 KB of SRAM. Results show that TinyTTA improves TTA accuracy by up to 57.6%, reduces memory usage by up to six times, achieves faster latency, and is more energyefficient than conventional baseline methods on MPUs. Notably, TinyTTA is the only framework able to run on the MCU STM32H747 with a 512 KB memory constraint while maintaining high performance, opening the door, for the first time, to performing TTA on resource-limited devices.

2 Related Work

This section reviews the key methods for performing TTA, emphasizing their main limitations. Subsequently, we examine the literature on early exit mechanisms.

GLYPH<c=7,font=/AAAAAB+font000000002b065d8f>

GLYPH<c=8,font=/AAAAAB+font000000002b065d8f>

GLYPH<c=9,font=/AAAAAB+font000000002b065d8f>

GLYPH<c=12,font=/AAAAAB+font000000002b065d8f>

GLYPH<c=7,font=/AAAAAB+font000000002b065d8f>GLYPH<c=11,font=/AAAAAB+font000000002b065d8f>

GLYPH<c=13,font=/AAAAAB+font000000002b065d8f>GLYPH<c=17,font=/AAAAAB+font000000002b065d8f>GLYPH<c=29,font=/AAAAAB+font000000002b065d8f>GLYPH<c=18,font=/AAAAAB+font000000002b065d8f>GLYPH<c=22,font=/AAAAAB+font000000002b065d8f>GLYPH<c=3,font=/AAAAAB+font000000002b065d8f>GLYPH<c=16,font=/AAAAAB+font000000002b065d8f>GLYPH<c=23,font=/AAAAAB+font000000002b065d8f> GLYPH<c=20,font=/AAAAAB+font000000002b065d8f>

GLYPH<c=6,font=/AAAAAB+font000000002b065d8f>

GLYPH<c=10,font=/AAAAAB+font000000002b065d8f>GLYPH<c=6,font=/AAAAAB+font000000002b065d8f>GLYPH<c=6,font=/AAAAAB+font000000002b065d8f>

GLYPH<c=7,font=/AAAAAB+font000000002b065d8f>GLYPH<c=6,font=/AAAAAB+font000000002b065d8f>GLYPH<c=6,font=/AAAAAB+font000000002b065d8f>GLYPH<c=6,font=/AAAAAB+font000000002b065d8f>

GLYPH<c=7,font=/AAAAAB+font000000002b065d8f>GLYPH<c=10,font=/AAAAAB+font000000002b065d8f>GLYPH<c=6,font=/AAAAAB+font000000002b065d8f>GLYPH<c=6,font=/AAAAAB+font000000002b065d8f>

GLYPH<c=15,font=/AAAAAB+font000000002b065d8f>GLYPH<c=20,font=/AAAAAB+font000000002b065d8f>GLYPH<c=25,font=/AAAAAB+font000000002b065d8f>GLYPH<c=27,font=/AAAAAB+font000000002b065d8f>GLYPH<c=28,font=/AAAAAB+font000000002b065d8f>GLYPH<c=31,font=/AAAAAB+font000000002b065d8f>GLYPH<c=3,font=/AAAAAB+font000000002b065d8f>GLYPH<c=4,font=/AAAAAB+font000000002b065d8f>GLYPH<c=15,font=/AAAAAB+font000000002b065d8f>GLYPH<c=13,font=/AAAAAB+font000000002b065d8f>GLYPH<c=5,font=/AAAAAB+font000000002b065d8f>

GLYPH<c=15,font=/AAAAAB+font000000002b065d8f>GLYPH<c=27,font=/AAAAAB+font000000002b065d8f>GLYPH<c=19,font=/AAAAAB+font000000002b065d8f>GLYPH<c=30,font=/AAAAAB+font000000002b065d8f>GLYPH<c=24,font=/AAAAAB+font000000002b065d8f>GLYPH<c=17,font=/AAAAAB+font000000002b065d8f>GLYPH<c=29,font=/AAAAAB+font000000002b065d8f>GLYPH<c=23,font=/AAAAAB+font000000002b065d8f>GLYPH<c=26,font=/AAAAAB+font000000002b065d8f>GLYPH<c=21,font=/AAAAAB+font000000002b065d8f>

GLYPH<c=14,font=/AAAAAB+font000000002b065d8f>GLYPH<c=23,font=/AAAAAB+font000000002b065d8f>GLYPH<c=26,font=/AAAAAB+font000000002b065d8f>GLYPH<c=20,font=/AAAAAB+font000000002b065d8f>GLYPH<c=29,font=/AAAAAB+font000000002b065d8f>GLYPH<c=30,font=/AAAAAB+font000000002b065d8f>GLYPH<c=26,font=/AAAAAB+font000000002b065d8f>GLYPH<c=20,font=/AAAAAB+font000000002b065d8f>

Out of Memory

(a) Tent memory (ResNet50)

GLYPH<c=7,font=/AAAAAB+font000000002b066171>

GLYPH<c=8,font=/AAAAAB+font000000002b066171>

GLYPH<c=9,font=/AAAAAB+font000000002b066171>

GLYPH<c=11,font=/AAAAAB+font000000002b066171>

GLYPH<c=7,font=/AAAAAB+font000000002b066171>GLYPH<c=10,font=/AAAAAB+font000000002b066171>

GLYPH<c=12,font=/AAAAAB+font000000002b066171>GLYPH<c=16,font=/AAAAAB+font000000002b066171>GLYPH<c=28,font=/AAAAAB+font000000002b066171>GLYPH<c=17,font=/AAAAAB+font000000002b066171>GLYPH<c=21,font=/AAAAAB+font000000002b066171>GLYPH<c=3,font=/AAAAAB+font000000002b066171>GLYPH<c=15,font=/AAAAAB+font000000002b066171>GLYPH<c=22,font=/AAAAAB+font000000002b066171>GLYPH<c=31,font=/AAAAAB+font000000002b066171>GLYPH<c=19,font=/AAAAAB+font000000002b066171>

GLYPH<c=6,font=/AAAAAB+font000000002b066171>

GLYPH<c=8,font=/AAAAAB+font000000002b066171>GLYPH<c=6,font=/AAAAAB+font000000002b066171>GLYPH<c=6,font=/AAAAAB+font000000002b066171>

GLYPH<c=9,font=/AAAAAB+font000000002b066171>GLYPH<c=6,font=/AAAAAB+font000000002b066171>GLYPH<c=6,font=/AAAAAB+font000000002b066171>

GLYPH<c=10,font=/AAAAAB+font000000002b066171>GLYPH<c=6,font=/AAAAAB+font000000002b066171>GLYPH<c=6,font=/AAAAAB+font000000002b066171>

GLYPH<c=11,font=/AAAAAB+font000000002b066171>GLYPH<c=6,font=/AAAAAB+font000000002b066171>GLYPH<c=6,font=/AAAAAB+font000000002b066171>

GLYPH<c=7,font=/AAAAAB+font000000002b066171>GLYPH<c=6,font=/AAAAAB+font000000002b066171>GLYPH<c=6,font=/AAAAAB+font000000002b066171>GLYPH<c=6,font=/AAAAAB+font000000002b066171>

GLYPH<c=14,font=/AAAAAB+font000000002b066171>GLYPH<c=19,font=/AAAAAB+font000000002b066171>GLYPH<c=24,font=/AAAAAB+font000000002b066171>GLYPH<c=26,font=/AAAAAB+font000000002b066171>GLYPH<c=27,font=/AAAAAB+font000000002b066171>GLYPH<c=30,font=/AAAAAB+font000000002b066171>GLYPH<c=3,font=/AAAAAB+font000000002b066171>GLYPH<c=4,font=/AAAAAB+font000000002b066171>GLYPH<c=14,font=/AAAAAB+font000000002b066171>GLYPH<c=12,font=/AAAAAB+font000000002b066171>GLYPH<c=5,font=/AAAAAB+font000000002b066171>

GLYPH<c=14,font=/AAAAAB+font000000002b066171>GLYPH<c=26,font=/AAAAAB+font000000002b066171>GLYPH<c=18,font=/AAAAAB+font000000002b066171>GLYPH<c=29,font=/AAAAAB+font000000002b066171>GLYPH<c=23,font=/AAAAAB+font000000002b066171>GLYPH<c=16,font=/AAAAAB+font000000002b066171>GLYPH<c=28,font=/AAAAAB+font000000002b066171>GLYPH<c=22,font=/AAAAAB+font000000002b066171>GLYPH<c=25,font=/AAAAAB+font000000002b066171>GLYPH<c=20,font=/AAAAAB+font000000002b066171>

GLYPH<c=13,font=/AAAAAB+font000000002b066171>GLYPH<c=22,font=/AAAAAB+font000000002b066171>GLYPH<c=25,font=/AAAAAB+font000000002b066171>GLYPH<c=19,font=/AAAAAB+font000000002b066171>GLYPH<c=28,font=/AAAAAB+font000000002b066171>GLYPH<c=29,font=/AAAAAB+font000000002b066171>GLYPH<c=25,font=/AAAAAB+font000000002b066171>GLYPH<c=19,font=/AAAAAB+font000000002b066171>

STM32H747XI

512KB

Out of Memory

(b) Tent memory (MCUNet)

GLYPH<c=5,font=/AAAAAB+font000000002b066e5e>

GLYPH<c=6,font=/AAAAAB+font000000002b066e5e>

GLYPH<c=8,font=/AAAAAB+font000000002b066e5e>

GLYPH<c=11,font=/AAAAAB+font000000002b066e5e>

GLYPH<c=5,font=/AAAAAB+font000000002b066e5e>GLYPH<c=10,font=/AAAAAB+font000000002b066e5e>

GLYPH<c=13,font=/AAAAAB+font000000002b066e5e>GLYPH<c=20,font=/AAAAAB+font000000002b066e5e>GLYPH<c=27,font=/AAAAAB+font000000002b066e5e>GLYPH<c=21,font=/AAAAAB+font000000002b066e5e>GLYPH<c=23,font=/AAAAAB+font000000002b066e5e>GLYPH<c=3,font=/AAAAAB+font000000002b066e5e>GLYPH<c=18,font=/AAAAAB+font000000002b066e5e>GLYPH<c=24,font=/AAAAAB+font000000002b066e5e>GLYPH<c=30,font=/AAAAAB+font000000002b066e5e>GLYPH<c=22,font=/AAAAAB+font000000002b066e5e>

GLYPH<c=4,font=/AAAAAB+font000000002b066e5e>

GLYPH<c=5,font=/AAAAAB+font000000002b066e5e>GLYPH<c=4,font=/AAAAAB+font000000002b066e5e>

GLYPH<c=6,font=/AAAAAB+font000000002b066e5e>GLYPH<c=4,font=/AAAAAB+font000000002b066e5e>

GLYPH<c=7,font=/AAAAAB+font000000002b066e5e>GLYPH<c=4,font=/AAAAAB+font000000002b066e5e>

GLYPH<c=8,font=/AAAAAB+font000000002b066e5e>GLYPH<c=4,font=/AAAAAB+font000000002b066e5e>

GLYPH<c=9,font=/AAAAAB+font000000002b066e5e>GLYPH<c=4,font=/AAAAAB+font000000002b066e5e>

GLYPH<c=12,font=/AAAAAB+font000000002b066e5e>GLYPH<c=21,font=/AAAAAB+font000000002b066e5e>GLYPH<c=21,font=/AAAAAB+font000000002b066e5e>GLYPH<c=28,font=/AAAAAB+font000000002b066e5e>GLYPH<c=25,font=/AAAAAB+font000000002b066e5e>GLYPH<c=20,font=/AAAAAB+font000000002b066e5e>GLYPH<c=21,font=/AAAAAB+font000000002b066e5e>GLYPH<c=29,font=/AAAAAB+font000000002b066e5e>

GLYPH<c=17,font=/AAAAAB+font000000002b066e5e>GLYPH<c=22,font=/AAAAAB+font000000002b066e5e>GLYPH<c=26,font=/AAAAAB+font000000002b066e5e>GLYPH<c=16,font=/AAAAAB+font000000002b066e5e>GLYPH<c=22,font=/AAAAAB+font000000002b066e5e>GLYPH<c=27,font=/AAAAAB+font000000002b066e5e>GLYPH<c=9,font=/AAAAAB+font000000002b066e5e>GLYPH<c=4,font=/AAAAAB+font000000002b066e5e>

GLYPH<c=15,font=/AAAAAB+font000000002b066e5e>GLYPH<c=14,font=/AAAAAB+font000000002b066e5e>GLYPH<c=19,font=/AAAAAB+font000000002b066e5e>GLYPH<c=16,font=/AAAAAB+font000000002b066e5e>GLYPH<c=22,font=/AAAAAB+font000000002b066e5e>GLYPH<c=27,font=/AAAAAB+font000000002b066e5e>

Out of Memory

(STM32H747XI)

Out of Memory

Memory-intensive TTA. Existing TTA work can be categorized into fine-tuning TTA and modulating TTA. In particular, fine-tuning TTA methods, such as CoTTA [13] and Test-Time Training (TTT) [14], updates the entire model architecture, leading to intensive memory usage. Modulating TTA methods, instead, adjust normalization layers while keeping other components frozen, as seen in TENT [6], TTN [15], NOTE [16], and SoTTA [17]. However, as shown in Figure 1a and 1b, modulating TTA still exhibits memory usage similar to fine-tuning TTA, since the model still needs to backpropagate through the full model structure.

Memory-efficient TTA. To address the memory inefficiency of previous TTA methods, various memory-efficient TTA techniques have been proposed, primarily targeting GPUs. These include EATA [7], which filters out high-entropy data to minimize optimization cache, and MECTA [18], which stops backpropagation caching. However, their analyses remain theoretical and impractical, and memory usage has been shown to be similar to that of modulating TTA [13]. Even the most efficient TTA method, EcoTTA [13], which achieves memory efficiency on GPUs by distilling the pre-trained model via shallower models, still relies on updating normalization layers, which are not supported in MCUs.

TTA under diverse batch sizes. As shown in Figure 1c, the success of most TTA methods relies on large batch sizes, due to more reliable statistics provided for normalization adaptation. Several methods target batch size 1 for more efficient adaptation, by using different normalization techniques like group norm (GN) [8], layer norm (LN) [3], and adaptive norm (AdaptBN) [13]. However, all these methods still exhibit poor TTA performance and require updating many normalization layers, which is computationally expensive and impractical for MCUs.

TTA under changing distribution shifts. Existing TTA methods often overlook the dynamic nature of distribution shifts and apply the same adaptation principle across all samples. However, real-world data on MCUs, such as sensor readings, exhibit highly diverse distribution shifts over time [19]. To date, only a few studies have considered the presence of different levels of distribution shifts. For example, NOTE [16] and DELTA [20] employ modified normalization layers, while SAR [3] proposes sharpness-aware TTA. However, these methods still rely on normalization layers and thus are not applicable when the batch size is one for MCU deployment.

Early exit. Early exit strategies have been explored to reduce computational costs in various machine learning architectures, particularly for real-time and resource-constrained applications. Notably, BranchyNet [21] introduced early exit mechanisms for deep neural networks, enabling faster inference by adding exit points and allowing models to make predictions at intermediate layers instead of traversing the full model for every input. Similarly, Shallow-Deep Networks [22] aim to reduce inference costs by dynamically selecting the exit point based on input complexity. For these reasons, early exits in the TTA setting can offer significant computational benefits. However, most early exit methods have not been designed to address distribution shifts or batch size constraints typical of MCUs.

3 Efficient TTA via Early-exit Ensembles and TinyTTA Engine

TinyTTA is designed for continuous adaptation to various types of distribution shifts in data during inference on resource-constrained edge devices. This novel framework, as shown in Figure 2, aims

Online single sample

On-device TTA

Severity 1

cat bird dog

cat bird dog cat bird dog

TinyTTA

(a)(b)(c)(d)

(d) TinyTTA

Engine

MPU

Severity 3

Severity 5

MCU

(c) WS normalization

WS(

μ

,

𝜎

)  Linear

Exit

(b) Early-exits

Exit

(a) Self-ensemble

Exit

Exit

Block-wise Memory

Pretrained model

(e.g., MCUNet)

Frozen

Ensembled

Backward

CNN

Forward

to improve system memory efficiency during TTA, adaptability of the pre-trained model to diverse distribution shifts, and deployment feasibility on edge devices. TinyTTA comprises four modules: i ) a self-ensemble network that partitions the network into submodules, each approximating the capability of the full model; ii ) early-exits, which allow samples to be inferred through specific submodules based on predicted confidence levels, optimizing different submodules for memory and computational efficiency while managing varying levels of distribution shifts; iii ) WS normalization, integrated into each submodule to achieve batch-agnostic normalization; and iv ) TinyTTA Engine, an MCU library for on-device TTA.

3.1 Self-ensemble network

For edge devices such as MCUs, it has been observed that sharing operational layers and weights, for instance through TensorFlow Lite Micro (TFLM) [11], can significantly reduce on-chip memory usage. Instead of utilizing a pre-trained network in its entirety, partitioning it into submodules can offer substantial benefits in terms of memory usage. Consequently, we propose self-ensembling pre-trained models by dividing layers into submodules, grouping similar subsequent layers, and approximating each submodule with the full model's capabilities. A critical decision in this process is determining the number of partitions required and how to effectively divide the network into submodules. Evidence suggests that i ) adjacent layers in neural networks tend to exhibit high feature similarities and ii ) these layers often have comparable memory consumption [5]. This provides a basis for grouping layers with similar memory consumption together as submodules.

We initially conducted an analysis of memory usage during TTA across the layers of a pre-trained model, ResNet50. This analysis, as illustrated in Figure 3, compares memory usage for both activations and weights between fine-tuning (top figure) and modulation (bottom figure) TTA, aiming to identify layers with similar memory usage. We observed the following: i ) memory usage is primarily driven by activations, which store the outputs at each layer for computing gradients during backpropagation, while the memory consumed by weights is negligible, therefore we focus on activation memory for self-ensembling; ii ) in both methods, activations share similar memory consumption and are predominantly concentrated in the initial layers, which generally capture crucial information from the input; iii ) certain groups of adjacent layers, specifically layers 1-15, 16-28, 29-44, and 45-52, show similar sizes of activations. Based on this analysis, we group layers with similar memory usage into submodules (i.e., layers 1-15 for submodule 1, 16-28 for submodule 2, 29-44 for submodule 3, and 45-52 for submodule 4) for subsequent early exits and only update the heads at early exits, freezing the submodules to improve memory usage. Indeed, this process does not require backpropagation to the submodules, thus eliminating the need for activation memory. It is worth highlighting that this analysis shows similar patterns for different model architectures, as discussed in Appendix D.

3.2 Early-exits

To further optimize memory usage and manage changing distribution shifts, we introduced early-exit inference combined with self-ensembling submodules. For a given pre-trained model, we sequentially pass inputs through cascaded submodules and infer a confidence level, calculated from the model's

high activation memory

low memory

high weight memory

low memory

high activation memory

(a) Fine-tuning per-layer memory usage

(b) Modulating per-layer memory usage

12

Memory (MB)

Weight

Activation

8

4

0

0

2

4

6

8

10

14

16

18

20

22

24

26

28

30

32

34

363840

42

44

46

48

50

52

Layerindex

Memory (MB)

15

Weight

Activation

５

0

2

4

6

8

10

1214

16

18

20

22

24

26

28

30

32

34

36

38

40

42

44

46

48

50

52

Layerindex

prediction, for the predictions at each stage. If the confidence level is high and exceeds a predefined threshold, which is a hyper-parameter during TTA (discussed in Appendix E), a good prediction is achieved, allowing the process to exit at that submodule without passing through the remaining ones. Model updates only occur for the preceding submodules. Conversely, if the confidence level remains low, indicating difficulty in prediction or a potential high distribution shift, the instance continues to pass through the subsequent submodules for further inference until it exits from an appropriate submodule with high confidence. This approach saves memory for easy samples that do not require extensive processing, while effectively handling varying distribution shifts. Different samples are processed differently, and updates are optimized according to the varying levels of distribution shifts in the samples. Our work is the first to introduce early exits for on-device TTA in edge devices, enhancing both inference efficiency and model accuracy with data distribution shifts.

Specifically, assume K submodules are derived, denoted by Θ = { θ k } K k = 1 . Each of the submodules is further associated with exits/classifiers, denoted as Φ = { ϕ k } K k = 1 , using a linear layer. Given the i th instance, the output for each submodule is first computed as:

where z k i ∈ R C denotes the latent representation learned from the k th submodule, and p k i represents the predicted probability for C classes for the classifier of the k th submodule after the softmax layer. Given the predicted probability p k i , we can estimate the confidence level using the entropy as follows:

A low entropy indicates low uncertainty in the prediction, thus a high confidence level. If H ( x i ) is smaller than the threshold γ k for each submodule k , this facilitates an early exit of the instance and also the adaptation of the submodules preceding this k th submodule. Otherwise, we further process the instance with the subsequent submodules.

3.3 WSnormalization

To entirely omit the usage of normalization layers and enable accurate and memory-efficient TTA on MCUs, we introduce Weight Standardization (WS) as a replacement for traditional normalization layers [3, 7, 8] for the first time in TTA . Particularly, we propose the inclusion of a WS layer preceding the linear layer attached to each submodule. Therefore, the exits/classifiers of each submodule ϕ k consist of a WS layer and a linear layer for C class classification. Our method adapts only ϕ k , i.e., the WS statistics and one linear layer during the testing phase, while keeping the other components θ k frozen. This avoids model collapse with small batch sizes and minimizes memory usage.

Unlike traditional normalization methods like Batch Normalization (BN) and Group Normalization (GN), which primarily focus on recentering and rescaling activations, Weight Standardization (WS)

targets the smoothing effects on weights. This approach results in a more favorable loss landscape and improved parameter updating. Additionally, WS guarantees batch-agnostic normalization, as it is not impacted by the batch size. The WS operation is defined as:

Here, µ w and σ w represent the mean and standard deviation of the weights calculated over all channels and dimensions, computed as:

and w i,j represents the element of the weight matrix for the i th row and j th column. This normalization is applied directly to the early exit CNN layers ϕ k . Importantly, WS introduces no additional parameters within the CNN layers and facilitates TTA across different batch sizes. During inference, WS is computed solely from the weights, eliminating the need to accumulate batch statistics on the fly, thus achieving batch-agnostic adaptation. This allows it to be applicable to small batch sizes, thus further ensuring memory and computational efficiency during TTA and making TinyTTA suitable for deployment on MCUs. From an implementation viewpoint, for MCU deployment, we introduce a new CNN layer incorporating the WS normalization in our TinyTTA Engine (see Section 3.5) to avoid using batch normalization layers, which are not supported by most constrained hardware.

3.4 Training and Inference

To facilitate the learning of submodules θ k and early-exit model parameters ϕ k , we introduce the concurrent training of all models on the training dataset. Given the source data, we pass them through both the pre-trained model and the submodules combined with early exit branches and compute the losses to optimize the models. The first loss, L 1 , aims to infer accurate predictions from the submodules and early-exit modules, which can be formulated as the cross-entropy (CE) for all classes. The second loss, L 2 , aims to achieve reliable self-ensembling by aligning the outputs from the submodules to be the same as the pre-trained models. The process can be formulated as:

where ˜ z k represents the features learned from the submodules and z k represents the one from the pre-trained model. The final loss consists of these two losses:

where λ controls the trade-off between the two losses.

The training process is performed offline, without additional training costs on the device. Then, at inference time during the TTA phase, TinyTTA operates without utilizing any source data, as in the standard TTA setting [6]. The model initially calculates the entropy of the first submodule's output and compares it with a predefined entropy threshold. Based on this comparison, the exit determines whether the feature outputs should proceed to the subsequent submodules of the self-ensemble model. The process continues if the entropy is greater than the threshold; otherwise, it halts. It is worth noticing that after training, only the submodules and early exits are deployed on the device. Once deployed, only early exit branches are updated on-device, while the rest of the model remains frozen, ensuring both high TTA accuracy and low memory usage.

3.5 TinyTTA Engine

To enable TTA on edge devices like MCUs, we propose TinyTTA Engine as illustrated in Figure 4. The TinyTTA Engine pipeline encompasses several steps. Initially, once the model is loaded, its forward graph is constructed. During compile-time, operations are optimized through fusion, and an automatic differentiation ( autodiff ) process constructs the backward graph needed for updating. Following autodiff , the backward graph is optimized via a fuse and quantize procedure, aiming to mitigate resource limitations during execution.

forward graph

fuse op

autodiff

backward graph

quantize

graph

Compile time

backward graph

freeze &

quantize

freeze &

quantize

freeze &

quantize

Load

Online sensor

reading

(a) TinyTTA processed model

(b) Autodiff

(c) Ensemble graph

(d) On-device TTA

Execution time

exit

Norm

CNN

Linear

WS

fp32

fp32

fp32

fp32

fuse op

fuse op

exit

MCU

The cornerstone of TinyTTA Engine lies in its ability to perform backpropagation on-device. We build TinyTTA Engine upon the widely used MCU library TFLM by developing a floating-pointbased backpropagation algorithm. TinyTTA Engine specifically caters to the backward computation needs of prevalent DNN operators such as ReLU , FullyConnected , Softmax , Maxpool , Avgpool , Conv , and DepthwiseConv . To further optimize memory efficiency, we have developed a layer-wise update strategy to reduce memory utilization alongside a dynamic memory allocation mechanism leveraging heap memory. Specifically, the interpreter-based framework of TinyTTA Engine enables memory reduction by dynamically choosing to save intermediate activations only if a layer needs updates (refer to Appendices C.2 and C.3). The TinyTTA Engine code is publicly available at https://github.com/h-jia/TTE .

4 Experimental Setup

Distribution shifted datasets. For comparison with baselines, our experiments were conducted using four popular corrupted datasets, including domain shift datasets CIFAR10C [23] and CIFAR100C [24], and two domain generalization datasets, namely OfficeHome [25] and PACS [26]. More details about the datasets can be found in Appendix B.

Implementation details. To provide a comprehensive evaluation of TinyTTA, we conducted experiments on both MPUs via a Raspberry Pi Zero 2W and MCUs using an STM32H747. Details of the hardware and MCU implementation can be found in Appendix C. The evaluation of TinyTTA on MPUs, which are equipped with relatively large memory, allows for a comparison with other state-of-the-art (SOTA) methods, thereby offering an in-depth understanding of TinyTTA's algorithmic performance. For the MPU experiments, we employed three SOTA mobile models, specifically EfficientNet [27], MobileNetV2 [28], and RegNet [29]. In particular, we utilized MobileNetV2 with a scale factor of 0.5 (MobileNetV2_x05), EfficientNet with a scaling factor of 1 (EfficientNet_b1), and RegNet with a computational cost of 200 million floating-point operations (RegNet-200m). For the MCU evaluation, we utilized the SOTA model MCUNet [5], which is the only model capable of running within the 512 KB memory constraint of MCUs. More details are discussed in Appendix D.

Baselines. We compared TinyTTA with several SOTA methods, including: i ) TENT [6], which optimizes the affine parameters of batch normalization layers through entropy minimization; ii ) CoTTA [13], which updates all model parameters using a consistency loss between student and teacher models and stochastically restores the pre-trained model; iii ) a memory-efficient baseline EATA [7], which employs a sample selection criterion for identifying non-redundant samples and updates the model by minimizing entropy loss; iv ) the most memory-efficient method EcoTTA [13], which freezes the original network parameters, uses AdaptBN [30] normalization layers and updates attached meta networks for memory-efficient test-time adaptation.

Evaluation metrics. The same evaluation metrics are utilized across all datasets, encompassing accuracy, memory usage, latency, and energy consumption. Memory usage is quantified using TinyTL [31] for the Raspberry Pi Zero 2W and with TFLite Tools 1 for MCUs, with a focus on the tensor-arena size. As both the MPU and MCU necessitate loading libraries for TTA, latency is

CIFAR100C

OfficeHome

PACS

50

60

70

60.2

47.3

53.1

56.1

64.3

51.6

58.3

63.6

62.4

49.1

55.2

58.2

67.5

53.3

61.7

65.7

60.4

48.4

54.5

57.5

65.3

52.6

59.6

62.6

59.1

45.8

51.6

53.6

63.8

50.5

54.4

57.3

MCUNet

MCUNet+TinyTTA

EfficientNet

EfficientNet+TinyTTA

MobileNet

MobileNet+TinyTTA

RegNet

RegNet+TinyTTA

recorded per batch. This measurement is taken after the model stabilizes following 10 warm-up iterations of test samples in TTA. Subsequently, the latency is averaged over 100 batches.

5 Results

This section addresses the following questions: (1) How does TinyTTA improve performance compared to the source model without adaptation? (2) How does TinyTTA perform compared to baseline methods on MPUs? (3) How efficient is TinyTTA on typical MCUs? (4) What effect do self-ensemble, early-exits, and WS have?

5.1 TinyTTA vs. source models on MPUs

We first compare the performance between TinyTTA and the source model without adaptation on MPUs, as shown in Figure 5. The performance was assessed using four distinct corrupted datasets and four different model architectures. Notably, TTA is evaluated with a batch size of one, a challenging scenario for adaptation. Most existing TTA methods generally fail due to unstable statistics associated with a single batch for normalization layers. For each combination of model and dataset, TinyTTA demonstrates an average improvement of 4.3%. The most significant enhancement is observed with the EfficientNet model on the CIFAR100C dataset, where accuracy increases from 49.1% to 53.3%. Even under the constraint of adapting to a single batch of data during test time, TinyTTA consistently outperforms models without adaptation, highlighting its potential in real-world scenarios where limited data is available for adaptation.

5.2 TinyTTA vs. SOTA baselines on MPUs

We further compared TinyTTA with other SOTA baselines across four different datasets and four model architectures on MPUs. The results are shown in Figures 6a to 6d.

For MCUNet, as shown in Figure 6a, TinyTTA consistently outperforms other TTA methods across all datasets. Specifically, our method achieves an accuracy of 64.3%, 51.6%, 58.1%, and 63.6% on CIFAR10C, CIFAR100C, OfficeHome, and PACS datasets, respectively. In comparison, the second-best performing method, ECoTTA, only attains an accuracy of 13.1%, 5.5%, 6.2%, and 6.3% on the same datasets, significantly deteriorating the performance compared to the source model without adaptation. This is due to the small batch size that causes unstable statistics for adaptation. Notably, on CIFAR100C, OfficeHome, and PACS datasets, TinyTTA achieves substantial accuracy improvements of 46.6%, 52.1%, and 57.6% over the second-best method, respectively. In terms of memory usage, TinyTTA is exceptionally efficient, requiring 1.2 × less memory than ECoTTA, 2.2 × less than EATA, 2.3 × less than TENT (Modulating), 4.2 × less than TENT (Fine-tuning), and 6.0 × less than CoTTA on average across all datasets on MPUs.

For EfficientNet, we observe a similar performance trend, as depicted in Figure 6b. In terms of accuracy, TinyTTA consistently outperforms other TTA methods across all datasets. On the CIFAR10C dataset, TinyTTA achieves an accuracy of 67.5%, which is 48.5% higher than the secondbest method, CoTTA. Similarly, on CIFAR100C, OfficeHome, and PACS datasets, TinyTTA also shows 47.3%, 53.7%, and 56.7% relative improvements over the second-best method, respectively.

CIFAR100C

OfficeHome

PACS

0

20

40

60

80

CIFAR100C

OfficeHome

PACS

0

2

4

Memory (MB)

TinyTTA

EcoTTA

EATA

TENT (Finetuning)

CoTTA

(a) MCUNet

CIFAR100C

OfficeHome

PACS

0

20

40

60

80

CIFAR100C

OfficeHome

PACS

0

50

100

Memory (MB)

TinyTTA

EcoTTA

EATA

TENT (Finetuning)

CoTTA

(b) EfficientNet

CIFAR100C

OfficeHome

PACS

0

20

40

60

80

CIFAR100C

OfficeHome

PACS

0

50

100

Memory (MB)

TinyTTA

EcoTTA

EATA

TENT (Finetuning)

CoTTA

(c) MobileNet

CIFAR100C

OfficeHome

PACS

0

20

40

60

80

CIFAR100C

OfficeHome

PACS

0

20

40

Memory (MB)

TinyTTA

EcoTTA

EATA

TENT (Finetuning)

CoTTA

(d) RegNet

Regarding memory usage, our method exhibits the lowest memory footprint among all the compared methods. On the CIFAR10C dataset, TinyTTA requires 5.65 MB of memory, which is 1.2 × lower than ECoTTA (7.06 MB), 2.2 × lower than EATA (12.59 MB), 2.3 × lower than TENT (Modulating) (13.02 MB), 4.2 × lower than TENT (Fine-tuning) (24.0 MB), and 6.0 × lower than CoTTA (33.88 MB). The memory usage trend remains consistent across other datasets.

For MobileNet and RegNet in Figures 6c and 6d, the results also show a similar trend as MCUNet and EfficientNet in terms of both accuracy and memory usage.

We also compared the latency and energy consumption of the different TTA methods on the Raspberry Pi Zero 2W MPU using the CIFAR10C dataset, which consists of 50,000 samples.

The results in Table 1 highlight the efficiency of TinyTTA in terms of both latency and energy consumption. Specifically, TinyTTA achieves an inference time of 0.22 seconds per sample (11,000 seconds in total) with a total energy consumption of 6.11 Wh. Compared to EATA, for example, TinyTTA reduces both latency and energy consumption by 12%. On the other hand, CoTTA shows the highest latency (312,500 seconds in total) and energy consumption (173.61 Wh), while TENT (both fine-tuning and modulating), though more efficient than CoTTA, is still outperformed by TinyTTA. Over-

all, TinyTTA offers the best balance between accuracy, memory, speed, and energy efficiency, making it ideal for edge AI applications.

5.3 TinyTTA on MCUs

We now show the performance of TinyTTA on extremely resource-constrained edge devices, specifically MCUs with only 512 KB of on-chip memory-significantly smaller than those of MPUs like the Raspberry Pi Zero 2W. Notably, TinyTTA is the only method capable of performing TTA under the STM32H747 MCU's 512 KB memory constraint while achieving superior accuracy compared to other methods . This makes TinyTTA a promising solution for resource-constrained devices requiring efficient and accurate TTA. We examine two scenarios: with and without TinyTTA ('Inference Only') to analyze effectiveness and system overheads. The results are shown in Table 2.

In addition to the superior performance over 'Inference Only', as shown in Section 5.2, TinyTTA reduces overall latency (50.7 ms vs. 55.8 ms) and energy consumption (11.5 mJ vs. 12.7 mJ) per

sample thanks to the early exit mechanism, despite requiring on-device model updates. Regarding memory usage, TinyTTA incurs small overheads of 40.2 KB for SRAM and 85.1 KB for Flash.

Specifically, the additional SRAM is needed for updated parameters after adaptation, while the early exits require additional heads, taking up storage on Flash.

Notably, our TinyTTA Engine enables fast (low latency) and efficient (low

energy) on-device adaptation with only a few tens of KBs of extra memory on extremely resourcelimited IoT devices.

5.4 Ablation Study

Figure 7 illustrates the trade-offs between accuracy and memory usage (in MB) across different configurations of our method for four datasets using MCUNet. The configuration represented by ☀ (TinyTTA using all components: self-ensemble, early-exits, and WS) achieves the best balance, with the highest accuracy and optimal memory usage. In contrast, ∎ (TinyTTA without WS) shows a drop in accuracy by approximately 12-15% and no change in memory usage. The ▲ (TinyTTA without early exits and WS) configuration reduces accuracy by about 44.5-54.5% and increases memory usage by 5 × , highlighting the critical role of early exits. Finally, ⧫ (without all components) exhibits the lowest accuracy, decreasing by around 40.3-50.3%, and the highest memory footprint increased by 5-6 × , underscoring the necessity of integrating all components. The ablation study

★

Self-ensembles +

Early-Exits + WS

clearly demonstrates that our method, when leveraging all components, achieves the optimal trade-off for practical applications.

6 Conclusion

This paper presented TinyTTA, which, for the first time, enables TTA on resource-limited edge devices such as MCUs. By introducing a self-ensemble framework, an early-exit policy, and weight standardization, our method overcomes the limitations of traditional TTA methods, enabling efficient yet competitive on-device TTA on edge devices. We also offer a new library, TinyTTA Engine, to unlock the development of TTA for various applications on MCUs.

Limitations and Future Work

TinyTTA has only been tested with a batch size of one in the setting to be deployed on a single type of MCU with image data. Additionally, TinyTTA requires retraining the model via self-ensemble. However, the underlying TinyTTA Engine offers a robust foundation for extending its capabilities. We anticipate that with further development, our method and the TinyTTA Engine library could be adapted for a wider array of applications, encompassing various types of MCUs and diverse data modalities beyond just images. Future research will involve investigating the potential of TinyTTA to process different types of data, such as video and inertial measurement unit (IMU) data, thereby broadening its usability across different real-world applications.

Broader Impacts

Deploying TTA on microcontrollers is particularly useful for applications that need quick decisionmaking and operate in resource-limited, changing environments. Unlike traditional model inference,

where the model's weights stay the same, TTA requires the model to update its weights based on shifts in input data distribution. This makes it challenging to maintain accuracy while being deployable on MCUs. Our TinyTTA framework addresses this issue, enabling TTA on MCUs for the first time.

7 Acknowledgment

This work is supported by ERC through Project 833296 (EAR), and Nokia Bell Labs through a donation. We also acknowledge financial support from the PNRR MUR project PE0000013-FAIR (Italy) - CUP H23C22000860006.

References

Muyang Lin, Ziyang Zhang, Xiaoxiang Gao, Yizhou Bian, Ray S Wu, Geonho Park, Zhiyuan Lou, Zhuorui Zhang, Xiangchen Xu, Xiangjun Chen, et al. A fully integrated wearable ultrasound system to monitor deep tissues in moving subjects. Nature Biotechnology , 42(3):448-457, 2024.

Artúr István Károly, Péter Galambos, József Kuti, and Imre J Rudas. Deep learning in robotics: Survey on model structures and training strategies. IEEE Transactions on Systems, Man, and Cybernetics: Systems , 51(1):266-279, 2020.

Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In The Eleventh International Conference on Learning Representations , 2023.

Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , 2021.

Ji Lin, Ligeng Zhu, Wei-Ming Chen, Wei-Chen Wang, Chuang Gan, and Song Han. On-device training under 256kb memory. Advances in Neural Information Processing Systems , 35:22941-22954, 2022.

Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell. Tent: Fully testtime adaptation by entropy minimization. In 9th International Conference on Learning Representations , 2021.

Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International conference on machine learning , pages 16888-16905. PMLR, 2022.

Skyler Seto, Barry-John Theobald, Federico Danieli, Navdeep Jaitly, and Dan Busbridge. Realm: Robust entropy adaptive loss minimization for improved single-sample test-time adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2062-2071, 2024.

Ji Lin, Wei-Ming Chen, Yujun Lin, john cohn, Chuang Gan, and Song Han. Mcunet: Tiny deep learning on iot devices. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 11711-11722. Curran Associates, Inc., 2020.

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and Training of Neural Networks for Efficient IntegerArithmetic-Only Inference. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.

Robert David, Jared Duke, Advait Jain, Vijay Janapa Reddi, Nat Jeffries, Jian Li, Nick Kreeger, Ian Nappier, Meghna Natraj, Tiezhen Wang, et al. Tensorflow lite micro: Embedded machine learning for tinyml systems. Proceedings of Machine Learning and Systems , 3:800-811, 2021.

Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batch training with batch-channel normalization and weight standardization. arXiv preprint arXiv:1903.10520 , 2019.

Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi. Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11920-11929, 2023.

Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning , pages 9229-9248. PMLR, 2020.

Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. TTN: A domain-shift aware batch normalization in test-time adaptation. In The Eleventh International Conference on Learning Representations , 2023.

Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. Note: Robust continual test-time adaptation against temporal correlation. Advances in Neural Information Processing Systems , 35:27253-27266, 2022.

Taesik Gong, Yewon Kim, Taeckyung Lee, Sorn Chottananurak, and Sung-Ju Lee. Sotta: Robust test-time adaptation on noisy data streams. Advances in Neural Information Processing Systems , 36, 2024.

Junyuan Hong, Lingjuan Lyu, Jiayu Zhou, and Michael Spranger. Mecta: Memory-economic continual test-time model adaptation. In The Eleventh International Conference on Learning Representations , 2022.

Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei W Koh, and Chelsea Finn. Wild-time: A benchmark of in-the-wild distribution shift over time. Advances in Neural Information Processing Systems , 35:10309-10324, 2022.

Bowen Zhao, Chen Chen, and Shu-Tao Xia. Delta: Degradation-free fully test-time adaptation. In The Eleventh International Conference on Learning Representations , 2022.

Surat Teerapittayanon, Bradley McDanel, and H. T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 23rd International Conference on Pattern Recognition, ICPR 2016 , pages 2464-2469. IEEE, 2016.

Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019 , volume 97 of Proceedings of Machine Learning Research , pages 3301-3310. PMLR, 2019.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Collin Burns and Jacob Steinhardt. Limitations of post-hoc feature alignment for robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2525-2533, 2021.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5018-5027, 2017.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision , pages 5542-5550, 2017.

Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 6105-6114. PMLR, 2019.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4510-4520, 2018.

Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10428-10436, 2020.

Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in neural information processing systems , 33:11539-11551, 2020.

Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems , 33:11285-11297, 2020.

Colby Banbury, Chuteng Zhou, Igor Fedorov, Ramon Matas, Urmish Thakker, Dibakar Gope, Vijay Janapa Reddi, Matthew Mattina, and Paul Whatmough. Micronets: Neural network architectures for deploying tinyml applications on commodity microcontrollers. Proceedings of machine learning and systems , 3:517-532, 2021.

Viet Anh Trinh, Hassan Salami Kavaki, and Michael I. Mandel. Importantaug: A data augmentation agent for speech. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022 , pages 8592-8596. IEEE, 2022.

0

10000

20000

30000

40000

50000

0

200

400

600

800

1000

Predicted Classes

Modulating

:

collapse at level 5

1000

800

600

400

200

0

20000

40000

10000

30000

Predicted classes

Online samples

0

(a) Modulating at level 5

0

10000

20000

30000

40000

50000

0

200

400

600

800

1000

Predicted Classes

Modulating

:

no collapse level 3

1000

800

600

400

200

0

20000

40000

10000

30000

Predicted classes

Online samples

0

(b) Modulating at level 3

0

10000

20000

30000

40000

50000

0

200

400

600

800

1000

Predicted Classes

Finetune

:

collapse at level 3

0

1000

800

600

400

200

0

20000

40000

10000

30000

Predicted classes

Online samples

0

(c) Fine-tuning at level 3

0

10000

20000

30000

40000

50000

0

200

400

600

800

1000

Predicted Classes

Finetune

:

no collapse level 5

1000

800

600

400

200

0

20000

40000

10000

30000

Predicted classes

Online samples

(d) Fine-tuning at level 5

A Appendix/supplemental material

Entropy Minimization . In test-time adaptation, the objective is to fine-tune the model parameters during the testing phase to better align with the test data distribution without the labels. One technique used for this purpose is entropy minimization [6], which aims to make the model's predictions more confident on the test data. For a classification model, the entropy of the model's prediction for a data point is calculated using the formula:

where H ( ˆ p ) represents entropy, C is the number of classes, and ˆ p c is the predicted probability of the data point belonging to class c .

Modulating and Finetune TTA. Our empirical analyses in Figures 8d and 8c demonstrate the performance of fine-tuning TTA for samples with severity levels 3 and 5, respectively. In contrast, Figures 8a and 8b illustrate the outcomes of modulation-based methods for the same severity levels. For fine-tuning TTA, while it can adapt to severe data corruption (e.g., severity level 5) by tuning the entire model, it struggles with milder domain shifts (e.g., severity level 3), leading to a model collapse where it predicts only a limited range of classes. This suggests that fine-tuning is more effective for significant domain shifts but becomes prone to collapse under minor shifts, resulting in limited class predictions. On the other hand, modulation-based TTA methods exhibit a different pattern. With milder severity (e.g., level 3) as shown in Figure 8b, the modulation-based method accurately predicts samples across various classes, indicated by the dispersed points representing the 1000 classes. However, under severe data corruption (e.g., level 5), modulation-based methods tend to predict only a limited number of classes with sequential test samples, as illustrated in Figure 8a by the black line showing overlapping predicted classes for a single class. This suggests a paucity in existing methods, indicating that different mechanisms might be required to address varying levels of distribution shifts within a universal framework, which TinyTTA achieves.

B Datasets

CIFAR10C and CIFAR100C feature 15 types of corruption, categorized into four main groups: noise, blur, weather, and digital, each with five severity levels. The OfficeHome dataset comprises four domains d ∈ { art, clipart, product, real } , containing 15,588 examples with dimensions (3, 224, 224) and 65 classes. Similarly, the PACS dataset comprises four domains d ∈ { art, cartoons, photos, sketches } , containing 9,991 examples with dimensions (3, 224, 224) and 7 classes.

C Implementation

We now introduce the hardware and software implementation of TinyTTA. Following this, we will discuss one of the core tasks of TinyTTA, which is the MCU implementation.

C.1 Hardware

The self-ensemble training phase of our framework was conducted on a Linux server equipped with an Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz and an NVIDIA Quadro RTX 6000 GPU, where we pre-trained the shared backbone and multiple heads. For the deployment phase, the system components were implemented on two types of edge devices. The first MPU was a Raspberry Pi Zero 2W featuring an ARM Cortex-A53 processor with distinct memory specifications. The second MCU we used is the STM32H747XI, which possesses a dual-core (ARM Cortex M4 and M7) architecture. We only use the ARM Cortex M7 core, reflecting a typical single-core MCU design. This choice constrained the SRAM and embedded Flash capacities to 512 KB and 1 MB for STM32H747XI, respectively. These two platforms are representative of a broad spectrum of devices commonly used in edge computing. Specifically, as the MCU is based on the ARM Cortex-M architecture, TinyTTA can be seamlessly deployed across a range of similar ARM Cortex-M-based MCUs, demonstrating the versatility of our approach in resource-constrained environments.

C.2 Software

The pre-training stage of TinyTTA, executed on a standard Linux server, utilized PyTorch version 1.10.2. For the meta-learner module, Python, NumPy, and TensorFlow Lite Micro (TFLM) were employed due to their compatibility and efficient integration. TFLM was specifically selected for its effectiveness and flexibility in deployment across diverse hardware platforms and pre-trained models without re-generating the code that embeds architectures, weights, and execution graphs and logic. The efficacy of TinyTTA was evaluated by analyzing the required system resources to enable TTA regarding memory usage of SRAM and Flash, end-to-end latency, and energy consumption. The process of transitioning a PyTorch model to these MCUs with TensorFlow Lite involved a multi-stage conversion: initially to ONNX format, then to TensorFlow, and finally to TensorFlow Lite, utilizing appropriate converters at each step. The execution of the TensorFlow Lite model on the MCUs was facilitated by TFLM and Mbed OS.

TensorFlow Lite for Microcontrollers (TFLM) [11] offers a platform tailored for executing machine learning models on tiny devices, circumventing the necessity for conventional operating systems or standard C/C++ libraries. TFLM's interpreter-based methodology has its advantages, providing adaptability to a wide set of edge devices and pre-trained models with minimal runtime overhead for deep neural network computations. However, TFLM's principal shortcoming lies in its incapacity for on-device training, confining models to a static state post-deployment for solely inference purposes. Recognizing the advantages of TFLM's minimal runtime overhead and high level of flexibility, we developed our TinyTTA Engine framework by augmenting a new functionality, on-device adaptation during runtime, to TFLM.

C.3 MCUImplementation

Wedeployed TinyTTA on the STM32H747XI. The deployment process involves loading the necessary libraries from Mbed OS 2 and acquiring the GCC ARM toolchain and the mbed-cli dev tool. For model management, the TFLM 3 framework is set up in the project environment, alongside mbed-os components. TFLite Tools 4 are employed to approximate rough SRAM requirements. The model is then converted into a C array format suitable for MCU deployment.

In addition, we enabled on-device adaptation of TinyTTA Engine by implementing a backward pass computation and extending this functionality on top of the floating-point-based operators (mentioned in Section 3.5) that previously supported only forward-pass computation in the TFLM framework. In detail, we implemented the gradient calculation regarding weights, biases, and activations in C for

each operator. Furthermore, we developed a layer-wise update logic that allows TinyTTA Engine to select which layer to update during on-device TTA to further reduce the memory overhead on SRAM and the required computational costs, as gradient calculation for weights can be omitted for non-updated layers.

Lastly, the increased binary size derived from our backpropagation implementation in TinyTTA is merely 8.6 KB as it is implemented on top of the existing operator codebase and third-party libraries. This shows that TinyTTA enables on-device TTA on MCUs for the first time, with minimal storage overhead.

C.4 TinyTTA vs. TinyEngine

As our TinyTTA Engine enables on-device training for TTA, in this section we discuss the state-ofthe-art on-device training framework for MCUs, known as TinyEngine [5]. Specifically, TinyEngine focuses on on-device training (with labeled data), employing automatic differentiation operations at compile-time and utilizing code generation to minimize runtime overhead. The framework statically pre-determines the layers and channels to be updated before deployment, allowing these updates to be executed at runtime. While this approach is efficient, it imposes constraints on post-deployment adaptability, as any model update requires recompilation for the target device. Furthermore, due to the code generation process, the model's architecture, weights, and layer dimensions are hard-coded into the binary file, which necessitates replacing the entire executable to modify the model.

In contrast, our TinyTTA Engine focuses on test-time adaptation with unlabeled data. It enables dynamic adaptation during inference by allowing early exits at submodules, specifically for highentropy samples, thereby enabling reliable TTA. To adapt TinyEngine for TTA, the only practical solution is to use TENT [6], which fine-tunes the model via entropy minimization. In our evaluation, we compared TinyEngine using TENT on a Raspberry Pi Zero 2W (with batch size 1) to our TinyTTA, measuring performance in terms of accuracy. The experimental setup is the same as described in Appendices B and C, and the results are summarized in Figure 9.

CIFAR100C

OfficeHome

PACS

0.0

0.2

0.4

0.6

0.8

0.13

0.06

0.07

0.06

0.64

0.52

0.58

0.64

TinyEngine

TinyTTA

(a) MCUNet

CIFAR100C

OfficeHome

PACS

0.0

0.2

0.4

0.6

0.8

0.19

0.11

0.09

0.07

0.68

0.53

0.62

0.66

TinyEngine

TinyTTA

(b) EfficientNet

CIFAR100C

OfficeHome

PACS

0.0

0.2

0.4

0.6

0.8

0.18

0.05

0.05

0.06

0.65

0.53

0.60

0.63

TinyEngine

TinyTTA

(c) MobileNet

CIFAR100C

OfficeHome

PACS

0.0

0.2

0.4

0.6

0.8

0.15

0.12

0.07

0.08

0.64

0.51

0.54

0.57

TinyEngine

TinyTTA

(d) RegNet

The results demonstrate that TinyTTA, powered by the TinyTTA Engine, significantly outperforms TinyEngine across all datasets and model architectures due to its ability to dynamically exit highentropy samples without the need for recompilation, making it a more practical solution for real-world deployment, where data distributions may evolve over time.

D Model size

Compared with ResNet50, on which the majority of TTA works focused, models targeting mobile devices are much smaller in terms of model parameters. Specifically, MobileNetV2_ × 05 5 , Efficient-Net_b1 6 , and RegNet-200m 7 have parameters totaling 23.71 MB, 24 MB, and 8.85 MB, respectively. In comparison, ResNet50 is approximately 40 times larger than MCUNet in terms of the number of parameters, which only has 2.6 MB. The MCUNet model, optimized for 256KB SRAM and 1MB Flash, was pre-trained on ImageNet and fine-tuned on CIFAR-10 and CIFAR-100 datasets for 50

MemoryUsageperLayer inMobileNetV2

Layer ID

Weight

Activation

Memory(MB)

3

2

1

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

32

34

36

38

40

42

44

46

48

50

Layer ID

Memory Usage per Layer inEfficientNet

Weight

Activation

Memory(MB)

2

0

Q

4

8

12

16

20

24

28

32

36

40

44

48

52

56

60

64

68

72

76

80

84

88

92

96

100104108112

epochs, following the methodology outlined in [31]. To further enhance the model's adaptability, we fine-tuned MCUNet for an additional 15 layers, enabling one-batch online adaptation.

As shown in Figure 10, similar to ResNet50 (cf. Figure 3), edge models such as MobileNetV2 and EfficientNet exhibit similar layerwise memory usage patterns, where adjacent layers show similar memory usage. For example, in MobileNetV2, we can group layers with similar memory usage into submodules (i.e., layers 1-11 for submodule 1, 12-20 for submodule 2, 21-40 for submodule 3, and 41-52 for submodule 4) for subsequent early exits. Similarly, for EfficientNet, we can group layers 1-26, 27-42, 43-81, and 82-115 for each submodule. Other per-layer model memory profiling can be accessed via related works, such as MCUNetV3 [5], where layers are grouped as 1-11, 12-18, 19-32, and 33-42. In RegNet, the layers are grouped as 1-6, 7-10, 11-23, and 24-42.

Regarding the computational overhead introduced by early-exit branches, we analyzed the increase in model size and the number of parameters for the early exits of three different models across various datasets. Note that the last exit layer retains the same memory and parameters as the original model. Table 3 summarizes these results.

Results show that the addition of early-exit branches results in relatively low memory overhead and parameter increases for all models. Specifically, i ) for MobileNet, the overhead ranges from 0.01 MBto 0.17 MB across different datasets, which is negligible compared to typical memory sizes (e.g., 512 MB or larger memory configurations); ii ) EfficientNet exhibits a slightly higher memory and parameter increase due to its more complex architecture, but still within acceptable limits for most resource-constrained devices, ranging from 0.02 MB to 0.75 MB depending on the dataset and exit

point; iii ) In comparison, RegNet has a memory overhead ranging from 0.10 MB to 0.66 MB, which still remains low relative to the overall system memory available on most edge devices.

E Comparison of WS and GN

Figure 11 presents the performance comparison of TinyTTA using different normalization layers, specifically Weight Standardization (WS) and Group Normalization (GN) [3, 8], across varying levels of distribution shifts in the CIFAR-10C dataset (L1 to L5). The results are averaged over three separate runs with different random seeds. TinyTTA with WS consistently outperforms TinyTTA with GN on all datasets. For instance, on dataset L1, TinyTTA+WS achieves an accuracy of 83.2 compared to 73.4 for TinyTTA+GN. This performance gap persists across the other datasets, with TinyTTA+WS maintaining higher accuracy levels (78.3, 73.6, 68.7, and 64.3) compared to TinyTTA+GN (67.2, 63.1, 57.9, and 53.2) on L2,

L1

L2

L3

L5

0

25

50

75

100

TinyTTA+WS

TinyTTA+GN

L3, L4, and L5, respectively. The observed differences suggest that WS is more effective than GN in maintaining model accuracy under distribution shifts, likely due to its ability to better standardize the weights during the adaptation process, thus enhancing the robustness and generalization capability of TinyTTA on resource-constrained MCUs.

F Impact of Changing Distribution Shifts

When examining the results across different datasets for TinyTTA, as shown in Figure 11, it is evident that the severity of distribution shifts significantly impacts the model's performance. As the datasets progress from L1 to L5, there is a noticeable decline in accuracy, indicating increasing levels of difficulty in adapting to the distribution shifts. Specifically, TinyTTA achieves an accuracy of 83.2 on L1, but this performance drops to 78.3 on L2, 73.6 on L3, 68.7 on L4, and finally 64.3 on L5. This downward trend suggests that as the severity of the shifts increases, the model finds it progressively harder to maintain high accuracy. Despite this challenge, TinyTTA demonstrates a relatively strong ability to adapt, maintaining a higher level of accuracy compared to methods using other normalization strategies. This highlights the robustness of WS in handling varying degrees of distribution shifts, making it a suitable choice for deployment in resource-constrained microcontroller environments where consistent performance is crucial.

G Choice of Entropy Thresholds

The entropy thresholds represent the level of distribution shift in data during adaptation made by each model. A higher entropy threshold indicates a lower level of confidence in the model's predictions, while a lower entropy threshold suggests more confident predictions. The grid search results indicate the optimal entropy thresholds for four models-MCUNet, EfficientNet, MobileNet, and RegNet-across four datasets: CIFAR10C, CIFAR100C, OfficeHome, and PACS. For the CIFAR10C dataset, which comprises 10 classes, the entropy thresholds are 1.12, 1.34, 1.53, and 1.26 for MCUNet, EfficientNet, MobileNet, and RegNet, respectively. For CIFAR100C, containing 100 classes, the thresholds are 3.26, 2.58, 3.1, and 2.31. For the OfficeHome dataset, which has 65 classes, the thresholds are 2.53, 2.16, 2.28, and 2.44. For PACS, the thresholds are 1.58, 1.43, 1.14, and 1.33. These values suggest that entropy thresholds vary significantly depending on the model and the dataset, reflecting the need for model-specific and dataset-specific tuning to achieve optimal performance.

H Experimental Results on Distribution-Shifted Audio Data

In this section, we present the results of our experimental evaluation on distribution-shifted audio data, aiming to demonstrate the adaptive capabilities of TinyTTA with data modalities beyond

images. Specifically, we utilized a pre-trained MicroNets model [32], which was originally trained on the Speech Commands V2 dataset, achieving an accuracy of 86%. This dataset consists of 35 keywords, including "yes," "no," and "forward," among others. To evaluate the robustness of the model in real-world scenarios, we tested it on the Musan Keywords Spotting test dataset [33], which includes the same 35 speech commands but under various real-world noise conditions such as dial tones, fax machine noises, car idling, thunder, wind, footsteps, rain, and animal noises.

The hyperparameters setting for TinyTTA is as follows: a learning rate of 1 × 10 -5 , a batch size of 1, the SGD optimizer with a momentum of 0.9, and a self-ensemble of early exit layers at depths [3, 5, 7]. Figure 12 shows the comparative results of different TTA methods. Specifically, the pre-trained model experienced a significant performance drop, losing approximately 33% accuracy when applied to the distribution-shifted Musan dataset without any adaptation. This highlights the challenge of adapting models to real-world noisy environments. Instead, TinyTTA achieved an 8% improvement in accuracy compared to no adaptation, demonstrating strong resilience to various types of environmental noise. Finally, TinyTTA outperformed all other tested methods, achieving the highest accuracy of 0.61. Notably, the second-best method, CoTTA, achieved only 0.23 accuracy, underscoring the effectiveness of TinyTTA in maintaining performance under substantial distribution shifts, also with distribution-shifted audio data.

I Comparison with Updating Bias Only

In this section, we explore the effects of fine-tuning only the bias, as implemented in TinyTL [31], applied to the TTA setting. We implemented a bias-only fine-tuning strategy via entropy minimization and compared it against adjusting the exits, as proposed in our TinyTTA. Specifically, during TTA, only the model's biases are updated, with no retraining of other parameters. Results in Figure 13 show that fine-tuning only the bias is insufficient for robust TTA performance, whereas integrating early exit branches and updating them in our TinyTTA demonstrates stable performance in scenarios involving distribution shifts across different datasets and models.

CIFAR100C

OfficeHome

PACS

0.0

0.2

0.4

0.6

0.8

0.15

0.09

0.11

0.07

0.64

0.52

0.58

0.64

Exits

(a) MCUNet

CIFAR100C

OfficeHome

PACS

0.0

0.2

0.4

0.6

0.8

0.13

0.15

0.18

0.09

0.68

0.53

0.62

0.66

Exits

(b) EfficientNet

CIFAR100C

OfficeHome

PACS

0.0

0.2

0.4

0.6

0.8

0.16

0.13

0.15

0.11

0.65

0.53

0.60

0.63

Exits

(c) MobileNet

CIFAR100C

OfficeHome

PACS

0.0

0.2

0.4

0.6

0.8

0.18

0.13

0.19

0.17

0.64

0.51

0.54

0.57

Exits

(d) RegNet

J Comparison with non-TTA methods

To better provide an understanding of the advantages of implementing TTA in resource-constrained environments, we present the comparison of TinyTTA with non-TTA methods. In particular, we implemented a new baseline using Test-Time Training [14] (TTT) with self-supervised rotation classification. We used SGD with momentum (0.9), a learning rate of 1 e -5 , an augmentation size of 20, a batch size of 1, and λ = 0 . 9 . Tests are performed on the Raspberry Pi Zero 2W.

The experimental results, as shown in Figure 14, demonstrate a significant improvement in accuracy across all models and datasets when using TinyTTA compared to applying Test-Time Training. For example, on the CIFAR10C dataset, TinyTTA boosts MCUNet's accuracy from 16% to 64% and EfficientNet's from 18% to 68%. This trend is consistent across all datasets, with TinyTTA providing

CoTTA

TENT (Finetuning)

EATA

ECoTTA

TinyTTA (ours)

0.0

0.2

0.4

0.6

No Adaptation

0.21

0.05

0.11

0.07

0.23

0.61

CIFAR100C

OfficeHome

PACS

0

20

40

60

16

5

7

7

64

52

58

64

18

6

6

12

68

53

62

66

17

7

8

10

65

53

60

63

15

12

6

7

64

51

54

57

MCUNet+TTT

MCUNet+TinyTTA

EfficientNet+TTT

EfficientNet+TinyTTA

MobileNet+TTT

MobileNet+TinyTTA

RegNet+TTT

RegNet+TinyTTA

CIFAR100C

OfficeHome

PACS

0

10

20

30

40

50

Memory (MB)

0.41

1.35

1.32

1.28

0.2

0.73

0.71

0.63

12.81

37.43

37.21

35.53

5.65

16.94

16.97

14.81

11.43

36.27

36.71

34.33

5.58

16.74

16.79

14.32

12.28

14.33

14.45

12.57

6.13

6.25

6.28

6.21

MCUNet+TTT

MCUNet+TinyTTA

EfficientNet+TTT

EfficientNet+TinyTTA

MobileNet+TTT

MobileNet+TinyTTA

RegNet+TTT

RegNet+TinyTTA

accuracy gains between 40-50% for all models. Interestingly, TinyTTA also reduces memory usage compared to TTT on the Raspberry Pi Zero 2W across all models and datasets, as shown in Figure 15, highlighting TinyTTA's efficiency not only in improving accuracy but also in minimizing resource consumption, making it particularly suitable for resource-constrained environments.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, the main claims in the abstract and introduction should accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, the authors discuss the limitations in discussion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Not applicable, as this paper does not contain theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, the authors provide sufficient information about their methodology, including the experimental setup and data analysis procedures in the Experimental Setup and Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes, the paper provides sufficient instructions to reproduce the main experimental results, as described in the supplemental material. The TinyTTA Engine code is fully publicly available at: https://github.com/h-jia/TTE .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, the paper provides a detailed description of the training and test protocols, including data splits, hyperparameter settings, and the optimization algorithm used in Experimental Setup and Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Yes. The paper includes information about the statistical significance of the experiments in the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes. The paper provides sufficient details on the computer resources needed to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes. The research conducted in the paper conforms to the NeurIPS Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Yes. The paper discussed the potential beneficial of societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Topic is not fit.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Yes. The paper credits asset creators, states licenses and terms, and ensures proper respect.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The details of the data, including the URLs, specifics of data generation, and model development, are provided in the Experimental Setup section and the Appendix.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No human involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"TinyTTA: Efficient Test-time Adaptation via Early-exit Ensembles on Edge Devices\" presents a combination of test-time Adaptation of pre-trained models and early-exist strategies for the efficient inference of Deep Learning models on edge devices. The authors introduce the ideas of test-time Adaptation (i.e., some parameters of the model are changed during deployment by, e.g., entropy minimization) and model splitting by introducing \"early exists\" nodes in the model. Most notably, the authors introduce a novel weight normalization that does not rely on batch statistics (e.g. BatchNorm) but re-weights and re-scales the weights of the model directly. Finally, the authors briefly present their novel inference engine, TinaTTA, which leverages the author's ideas into a usable framework that is used for the experiments. The authors showcase some impressive results on established image datasets (CIFAR10C, CIFAR100C, OfficeHome, PACS) and known small devices (Raspberry et al. two and a dual-core embedded Arduino). In particular, the present method has low latency and low energy consumption while often offering a much better accuracy than existing methods.\n\nSTRENGTHS:\n- The overall idea of bringing test time adaptation through early exists too small devices is good\n- The TinyTTA engine and the experimental evaluation seem impressive and well done\n\nWEAKNESSES:\nOverall, this is a very engineering-focused paper without much methodological impact. Therefore, it can be questioned whether it fits the NeurIPs conference. More specifically: \n- The authors propose self-ensembling, which basically means splitting the network into smaller submodules. The idea is very straightforward and not really studied in depth in the paper. It is unclear how to split the network (the authors mention using activation as a guiding tool, but it is unclear how they do it exactly). Similar ideas have already been discussed in the literature (see next comment), which are not mentioned in the paper. \n- The authors propose early-exists, which is not a novel technique but is well-known in the literature already [1,2,3]. As far as I remember, this technique has already been used since Yolo v5 (although mainly for improved training, and this might be debatable) [4]. Unfortunately, the Related Work section focuses on test-time adaptation (Which is good. I learned a lot here, thanks!) but misses out on the related work on the early-exit networks. Hence, it is unclear to me to what extent self-ensembling and early-exists is really new or just a re-brand of existing methods\n- While the evaluation is generally good, it does not highlight the memory overhead of introducing early exists. At the very least, the authors have to introduce some classification heads for each exit, which requires additional parameters. I could not find a mention of this in the paper, but maybe I missed it (see my question).\n- Minor issues with the paper \n\t- Citations can be improved: There are a few arxiv papers that have been published already. I suggest using dblp for high-quality references (Note: There might be more than these three; I stopped checking after three):\n\t\t- Tent: Fully Test-time Adaptation by Entropy Minimization is an ICLR 2021 paper I think\n\t\t- Towards Stable Test-Time Adaptation in Dynamic Wild World is an ICLR 2023 paper I think\n\t\t- RobustBench: a standardized adversarial robustness benchmark is a NeurIPs 2021 Paper I think\n\t- The authors mention data distributional shifts multiple times in the paper. I suggest to clarify this a bit more, since it is unclear to me if we look at shift in the label space or data space. \n\t- In section 3.1, the sentence \"[...] and approximating each submodule with the full model's capabilities\" is unclear. See my question.\n\t- In section 3.1, the merging of subsequent layers into submodules is not really clear. See my question.\n\t- The authors mention that \"activations consume much more memory compared to weights\", which is also unclear to me. See my question.\n\n[1] Why should we add early exits to neural networks? by Scardapane etal. 2020 https://arxiv.org/abs/2004.12814\n\n[2] BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks by Teerapittayanon etal. 2017 https://arxiv.org/abs/1709.01686\n\n[3] T-RECX: Tiny-Resource Efficient Convolutional neural networks with early-eXit by Ghanathe and Wilton 2023 https://arxiv.org/pdf/2207.06613\n\n[4] TPH-YOLOv5: Improved YOLOv5 Based on Transformer Prediction Head for Object Detection on Drone-captured Scenarios by Zhu etal. 2021\n\nQUESTIONS:\n1) What is the overhead in memory/number of parameters by adding new prediction heads? Did you explore this? \n2) How did you analyze the activations for grouping the models into sub-modules? Did you apply some principle here, or was it an ad-hoc grouping? \n3) What do you mean when you write, \"activations consume much more memory compared to weights\"?\n\nLIMITATIONS:\nThe authors discuss limitations in the evaluation of their method (image data only, only one MCU), but the evaluation is generally well-done and, hence, quite thorough.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"TinyTTA: Efficient Test-time Adaptation via Early-exit Ensembles on Edge Devices\" presents a framework designed to enable test-time adaptation (TTA) on resource-constrained IoT devices. TinyTTA utilizes a self-ensemble and batch-agnostic early-exit strategy to adapt to data distribution shifts efficiently with smaller batch sizes, thereby reducing memory usage and improving latency. The approach is validated on Raspberry Pi Zero 2W and STM32H747, demonstrating significant improvements in accuracy, memory efficiency.\n\nSTRENGTHS:\n1. The authors address the challenges of deploying TTA on devices with stringent memory and computational constraints, which is a less explored area\n2. use of self-ensemble networks and early-exit strategies for TTA on edge devices allowing for adaptive inference based on confidence levels, optimizing both memory usage and computational efficiency.\n3. uses Weight Standardization (WS) as a substitute for traditional normalization layers, specifically tailored for microcontrollers with strict memory limitations.\n4. The paper provides a comprehensive evaluation of the framework across multiple datasets and compares its performance with several state-of-the-art methods. The evaluation covers all important metrics like accuracy, memory usage, latency etc.\n5. The paper is well-structured and written in a clear way explaining all concepts and methodologies. Key concepts such as self-ensemble networks, early-exit strategies, and weight standardization are explained with sufficient clarity\n6. Introduces TinyTTA Engine, a MCU library that enables on device TTA.\n\nWEAKNESSES:\nmain:\n- The paper only presents results on vision data, what about other type of data?\n- Implementation Details needed for reproducibility are missing and the code is not provided. E.g, There are not details about how submodules are created for the models utilized in the paper and hyperparameters used during training which hinders the reproducibility. \n- While the results of the experiments are effectively demonstrated in the figures, the readability of the figures can be improved. Specifically, Figure 5 is hard to read. The caption of this figure is also not explanatory enough. Consider briefly summarizing the results in the caption.\n- The comparison was only limited to other TTA methods. Including non-TTA methods might provide a baseline to better understand the advantages of implementing TTA in resource-constrained environments\n- The experiments primarily focus on a specific type of MCU (STM32H747) and one MPU (Raspberry Pi Zero 2W). Is TinyTTA generalizable across other platforms as well?\n- Section 3.1. self-ensemble network, second paragraph: \"(iii) certain groups of adjacent layers, specifically\n145 layers 2-15, 16-28, 29-44, and 45-52, show similar sizes of activations. Based on this analysis, we\n146 group layers with similar memory usage into submodules for subsequent early exits to improve\n147 memory usage: i.e., layers 0-15 for submodule 1, 16-28 for submodule 2, 29-44 for submodule 3, and\n148 45-52 for submodule 4. \". \nThis is\n   -  very model specific: does this generalize to other models?\n   - overly details. Please move such details to a table or so. \n- Section 3.1. second paragraph and also Figure 3: this is well known and, for example, also stated in the MCUNet paper.\n- Details on the early exit are not clear: what layers etc. do the authors use?\n- The ablation study is incomplete: needs to also show early exit with and without model updates. \n- Table 1 should get another line: \"Inference-only with EE\". \n-The paper does not explore the sensitivity of TinyTTA to various hyperparameters.\n-The paper lacks a detailed analysis of adaptation time and energy consumption.\n-The paper does not discuss the challenges and trade-offs involved in deploying the framework\n\nminor: \n* last paragraph of the introduction: 512 KB of SRAM stated twice\n* Section 3.2 Early Exists: \"For a given pre-trained model ,\" -> remove space before comma\n* Figure 5: I suggest using the similar color for the same models, e.g.., similar color for MCUNet and MCUNet+TinyTTA\n* Figure 6: font size of the figures is too small\n* Figure 7: please add units % and KB\n\nQUESTIONS:\n- Could you possibly add more details about computational overhead of self-ensembles and early exits.\n- What is the motivation for using entropy thresholding in early exit? \n- While you evaluated TinyTTA on four benchmark corruption datasets, these are based on synthetic noise. - How does TinyTTA perform on datasets with real-world distribution shifts or noise?\n- While you mention improvements in energy efficiency, a detailed analysis is not provided. Can you provide a breakdown of energy consumption for different components of TinyTTA, and compare it with baseline methods\n-how do the optimal entropy thresholds in Appendix G differ and impact the performance of the system?\n\nLIMITATIONS:\nThe authors provided a limitations section, and to the best of my understanding, the paper does not have any negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a test-time adaptation framework for tiny deep neural networks. Specifically, the proposed framework partitions a specific model based on the memory usage of each layer, clusters adjacent layers with similar memory usage into a submodule, and adds an early exit header for each module. To avoid using batch normalization, the authors adopt weight standardization for the early exit header layer. Only the early exit header layer and the corresponding weight standardization parameters are updated during test-time adaptation. The authors also developed an MCU library to support the aforementioned test-time adaptation. The framework can support low-end IoT devices with only 512KB of memory.\n\nSTRENGTHS:\n1. Real device deployment: It is great to see that the proposed framework can facilitate the deployment of tiny neural networks on low-end IoT devices with only 512 KB of memory.\n\n2. Well-motivated: The analysis of the memory usage of existing test-time adaptation techniques clearly highlights the drawbacks of previous methods, making the idea of partitioning models based on memory usage quite straightforward.\n\n3. Impressive results: The significantly better accuracy versus memory usage compared to baseline test-time adaptation on four different models is very impressive.\n\nWEAKNESSES:\n1. Lack of discussion on design choices: For example, why can't the \"fine-tune bias only\" technique from TinyTL [28] be used in test-time adaptation? What is its performance compared to only fine-tuning the early exit header proposed in this work? Why is there a \"lack of support for normalization layers on MCUs\"? Since the authors have developed their own MCU library, why can't the normalization layer be added to the library?\n\n2. Limited experiments: As the authors mentioned in the Conclusion section, this work only targets image data. Thus, it is unclear whether the design in this work can be generalized to other applications. For example, will only updating the early exit header be sufficient for other applications?\n\n3. Lack of details on the TinyTTA Engine: Since the algorithm is not entirely novel (i.e., adding multiple early exit headers and weight standardization are not proposed by the authors, but the authors may be the first to use them in test-time adaptation for tiny models), the TinyTTA Engine itself seems to be the key factor ensuring these techniques work efficiently and effectively on real devices. More details and insights from the implementation of the TinyTTA Engine would be greatly appreciated by the community.\n\nQUESTIONS:\n1. Will you open-source the TinyTTA engine?\n\n2. Is there any real-world case to show that updating the model via backpropagation, instead of simply switching some modes, makes a significant difference for tiny models?\n\nLIMITATIONS:\nThe author discussed the limitations but did not address the potential negative societal impact of their work. This should be fine because, in my opinion, this work does not have any negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors focus on enabling test time adaption on resource limited edge devices. To achieve that, the authors first to train a self-ensemble network where the sub-networks are partitioned according to memory usage. After that, the authors further adopt WS normalization to improve adaption capacity under batch size as one.  In the experiments, the proposed methods show better accuracy with higher efficiency compared to prior test time adaption works.\n\nSTRENGTHS:\n1. Although self-ensemble learning and WS normalization is not new, it seems to first use on-device test time adaption.\n\n1. It's interesting to see that adopting WS normalization can achieve better accuracy compared to other normalization layers on the setting of batch size as 1.\n\nWEAKNESSES:\n1. The motivation to modulate the pre-trained model according to memory usage is confused.  \n\n2. The training process on device is unclear.\n\n3. The paper writing needs be improved.\n\nQUESTIONS:\n1. The motivation to modulate the pre-trained model according to memory usage is confused.  In the Fig.3, it shows that initial layers occupy much higher memory usage than later layers. However, the authors group the first several layers as first submodule that should be always activated during test time adaption on device, leading to less memory reduction. Will it better to avoid pass through some of the initial layers and active more later layers? \n\n2. One main concern is that the training process happened on device is unclear.  \n\n    (1) The self-ensemble learning has higher training cost. In line 206, the authors mention that \"After training, only the submodules and early exits will be deployed on MCUs.\" It seems that the proposed method first do self-ensemble learning offline which suffers from extra training cost. \n\n    (2) Since the work targets on test time adaption where the source data should be unavailable. How does the authors to train the self-ensemble networks?\n\n    (3) In line 220, the authors said \"The cornerstone of TinyTTA lies in its ability to perform backpropagation on-device\". What is the difference compared to [1]? Which part of the model will be trained on device?  \n\n   [1] On-device training under 256kb memory.\" NeurIPS 2022 \n\n3. The Fig.3 is unclear.  From my understanding, modulation aims to partition the entire model into different groups based on activation memory. Why does activation memory change per layer, and there is no weight memory, compared to \"fine-tuning per-layer memory usage\"? \n\n4. Writing needs to be improved:\n\n   (1) In line 97, \"practical\" should be \"impractical\"\n\n   (2) Line 170 is incomplete\n\n   (3) In line 175, the authors said \"to entirely omit usage of normalization layers and ....  on MCUs\". Isn't WS normalization is a normalization layer used on MCUs?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}