Towards Robust and Efficient Cloud-Edge Elastic Model Adaptation via Selective Entropy Distillation
Authors: Yaofo Chen
Shuaicheng Niu
Yaowei Wang
Shoukai Xu
Hengjie Song
Mingkui Tan
Venue: ICLR 2024 poster (2024)
Keywords: model adaptation
cloud-edge model deployment
cloud-edge model collaboration
test-time adaptation
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose a Cloud-Edge Elastic Model Adaptation (CEMA) paradigm to conduct robust and effcient test-time model adaptation in a collaborative way..

Abstract: The conventional deep learning paradigm often involves training a deep model on a server and then deploying the model or its distilled ones to resource-limited edge devices. Usually, the models shall remain fixed once deployed (at least for some period) due to the potential high cost of model adaptation for both the server and edge sides. However, in many real-world scenarios, the test environments may change dynamically (known as distribution shifts), which often results in degraded performance. Thus, one has to adapt the edge models promptly to attain promising performance. Moreover, with the increasing data collected at the edge, this paradigm also fails to further adapt the cloud model for better performance. To address these, we encounter two primary challenges: 1) the edge model has limited computation power and may only support forward propagation; 2) the data transmission budget between cloud and edge devices is limited in latency-sensitive scenarios. In this paper, we establish a Cloud-Edge Elastic Model Adaptation (CEMA) paradigm in which the edge models only need to perform forward propagation and the edge models can be adapted online. In our CEMA, to reduce the communication burden, we devise two criteria to exclude unnecessary samples from uploading to the cloud, i.e., dynamic unreliable and low-informative sample exclusion. Based on the uploaded samples, we update and distribute the affine parameters of normalization layers by distilling from the stronger foundation model to the edge model with a sample replay strategy. Extensive experimental results on ImageNet-C and ImageNet-R verify the effectiveness of our CEMA.

Published as a conference paper at ICLR 2024
TOWARDS ROBUST AND EFFICIENT CLOUD-EDGE ELASTIC MODEL ADAPTATION VIA SELECTIVE
ENTROPY DISTILLATION
Yaofo Chen 12; Shuaicheng Niu Yaoweí Wang"", Shoukai Shoukii Xu', Hengjie Song', Mingkui Tan 145 South China University of Technologyy Pengcheng Laboratory² Nanyang Technological University Key Laboratory of Big Data and Intelligent Robot, Ministry of Education Pazhou Laboratory chenyaofo@gmaii com; mingkuitan@ edu.cn
ABSTRACT
The conventional deep learning paradigm often involves training a deep model server and then deploying the model or its distilled ones resource-limited edge devices. Usually, the models shall remain fixed once deployed (at least for period) due to the potential high cost of model adaptation for both the server and edge sides. However, in many real-world scenarios, the test environments may change dynamically (known as distribution shifts), which often results in de- graded performance. Thus, one has to adapt the edge models promptly to attain promising performance. Moreover, with the increasing data collected at the edge, this paradigm also fails to further adapt the cloud model for better performance. To address these, we encounter two primary challenges: 1) the edge model has limited computation power and may only support forward propagation; 2) the data transmission budget between cloud and edge devices is limited in latency- sensitive scenarios. In this paper, we establish Cloud-Edge Elastic Model Adap- tation (CEMA) paradigm which the edge models only need to perform forward propagation and the edge models can be adapted online. In our CEMA, to reduce the communication burden, we devise two criteria to exclude unnecessary samples from uploading to the cloud, iee., dynamic unreliable and low-informative sample exclusion. Based on the uploaded samples, we update and distribute the affine pa- rameters of normalization layers by distilling from the stronger foundation model to the edge model with a sample replay strategy. Extensive experimental results on ImageNet-C and ImageNet-R verify the effectiveness of our CEMA.
INTRODUCTION
Deep neural networks (DNNs) have witnessed remarkable breakthroughs in broad spectrum of applications from computer vision (He et al., 2016; Dosovitskiy et al., 2021) to natural language processing (Radford et al., 2018; Brown et al., 2020). In real-world applications, the traditional deployment pipeline of DNNs is as follows: 1) training a large/foundation model on a cloud server and 2) tistiliingcomppressnng the large/foundation model into smaller model to be deployed in edge devices for delay-sensitive applications. This pipeline has gained great success when test samples share the same distribution the training ones. However, in real-world edge devices, the environment may dynamically change and the distributions of test samples are different from training ones. Such distribution shifts often result from natural variations or corruptions, such as changes in lighting and sensor degradation (Hendrycks & Dietterich, 2019; Koh et al., 2021). In this case, models may exhibit significant performance drop (Wang et al., 2021; Zhang et al., 2022a).
To handle the distribution shift, previous methods seek to update the edge model, which can be roughly categorized into two groups: i) Offline generalization methods are executed on the cloud and then distribute updated models to the edge devices. Specifically, unsupervised domain adap- tation methods (Zhang et al., 2020; Liang al., 2020; Qiu et al., 2021; Lin et al., 2022) perform model adaptation collected test data in an offline manner. Domain generalization methods (Li
"qqual contribution. 'Corresponding author.

---

Published as a conference paper at ICLR 2024
Distribute Parameters
Edge
Model Adaptation
Edge
Limited
Guided
Edge Model
Resources
Foundation
Model
Edge Model
Model
Test Sample
Locul Edge TTA
Predictions
Massive
Test Sample
Resources
Cloud
Edge
Upload Sceuario Samples
Comparisons between the conventional Test-time Adaptation (TTA) (left) and our Cloud- Edge Elastic Model Adaptation (right). The conventional one locally performs adaptation only in the edge with limited resources. In contrast, our CEMA conducts model adaptation more efficiently in the edge, which offloads the heavy adaptation workloads to the cloud with massive resources.
Figure 1:
et al., 2018; Dou et al., 2019) pre-anticipate the possible test shifts at the training time, in which the possible shifts can be simulated by meta-learning scheme. However, they may yield inferior performance since it is hard to pre-anticipate all unknown shifts at training time. ii) Online gener- alization methods directly learn the shifts by adapting the model with test data. Recently, test-time training (Sun et al., 2020; Bartler et al., 2022) and fully test-time adaptation (TTA) methods (Wang et al., 2021; Niu et al., 2022; 2023) are newly devised to adapt a model to the test domain in an online manner, which are more practical in real-world applications. However, they may be computationally heavy to perform back- proppapppag,,inn, which may be unaffordable in resource- limited edge devices.
Besides, the foundation model in the cloud also should be continuously updated using the test sam- ples in the edges. To address the above issues, one can leverage both the cloud and the edge by uploading all the test samples the cloud for adaptation of both the foundation and edge models. However, it is still very challenging: 1) The data communication burden between the cloud and edges may be heavy. Since the communication overhead is mainly affected by the number of up- loaded samples. It not only decreases the adaptation efficiency in the cloud but also consumes the limited bandwidth the performance of the edge model on distribution-shif. test data is an open question. Typically, the cloud has much richer computational resources and budgets than edges. this case, the cloud able to support heavier computation and leverage more complex and stronger models for adaptation.
the cloud-edge system. 2) How to exploit the foundation model enhance
In this paper, we propose Cloud-Edge Flastic Model Adaptation (CEMA) paradigm that executes dynamic model adaptation in a cloud-edge collaborative way instead of inference with the fixed model. As shown in Figure 1, we delegate all adaptation workloads to the cloud and thus only require vanilla inference in edges. To reduce communication overhead, we exclude two types of samples from uploading to the cloud: 1) unreliable samples with high entropy identified by a dy- namic entropy thresholding scheme; 2) low-informative samples with low entropy identified by an unchanged thresholding scheme. Based on this, our CEMA greatly reduces the communication bur- den. To leverage rich knowledge in the foundation model, knowledge distillation for adaptation. To improve the data utilization efficiency of uploaded sam- ples, we devise replay buffer to store and reuse these samples. We distill the foundation model to the edge model based on both the newly uploaded samples and samples from the replay buffer. In this way, our CEMA achieves better performance than the vanilla adaptation.
it to guide the edge model via
Main novelty and contributions: 1) We establish a Cloud-Edgee Elastic Model Adaptation (CEMA) paradigm designed for efficient collaborative model adaptation. Our CEMA a general paradigm that is applicable to online adapt edge models to new dynamically changing environments. 2) We improve the adaptation performance of the edge model by performing a replay-based entropy distil- lation, which minimizes prediction entropy and the KL divergence between the edge model and the foundation model using a sample replay strategy. 3) We reduce communication costs by devising entropy- based criteria for excluding unreliable and wow-informative samples from being uploaded. Experimental results show CEMA lowers 60% communication cost than SOTAS on ImageNet-C.
CLOUD-EDGE COMMUNICATION-EFFIIC
MODEL ADAPTATION
Problem statement. In this paper, we focus on how to efficiently improve adaptation performance in the context of cloud-edge deployment under distribution-shifted scenarios. Let (-) denote model trained powerful cloud server on a set of training data. Instead of deploying the model gw(() in the cloud, would infer (-)(.) locally resource-limited edge device (egg., a surveil-
2

---

Published as a conference paper at ICLR 2024
Cloud (Massive Resources)
Entropy Loss (Equ. 5)
Replay Buffer
Foundation Model fe(.)
(f(xx))
Distillation Loss (Eqn. 6)
Uploaded Samples (xiiee
Edge Moddl (a()
(au(s))
Weights
Upload Samples (async)
Distribute Parameters (async)
Edges imited Resources)
Samples
Uplondad Samples
Samples (*x
Test Sample
Model (-1
Logits
Online Samples
Remove Samples via (xx)
An overview of our proposed CEMA. edge: after inference, each edge asynchronously uploads samples the cloud by excluding unreliable ones (based Shigh (x) in Eqn. (1)) and low- informative ones (based on 5iow(x) in Eqn. (3)). In cloud: 1) our CEMA improves the foundation model f.) with the uploaded samples via entropy minimization (Eqn. 5) and meanwhile stores uploaded samples into replay buffer. 2) With both the uploaded samples and the samples randomly sampled from the replay buffer, CEMA adapts the edge model gw with the guidance from the foundation model fo)-) via the knowledge distillation loss (Fqn. 6).
Figure 2:
lance camera an industrial park) for delay-sensitive applications. In the inference, (.) on edge devices may encounter out-of-distributinn test samples. These test samples are distribution-shine the training ones due to natural variations or corruptions, such as lighting/weather changes and unexpected noises resulting from sensor degradation. In this case, the model gar(-) may often be sensitive to these distribution shifts, potentially leading to significant performance degradation.
To tackle the distribution-shiff issue, many test-time adaptation (TTA) approaches (Sun et al., 2020; Wang et al., 2021) have been proposed to improve the model adaptation performance through pa- rameter updates. These methods become attractive since they do not require access to training data and adapt the model on the unlabeled test data via entropy minimization in self-supervised manner. However, the applicability of these approaches source constraints, such as memory limitations that hinder model updating. An alternative approach employ these methods adapt the model gar (-) in the cloud by centralizing the test data. Nev- ertheless, this suffers from two challenges. 1) Uploading all test samples incurs significantly heavy communication overhead. 2) Conventional TTA methods are typically designed for a single device and may be hard to fully exploit the resources of both the cloud and the edges. In this study, we seek address the issues by developing an efficient and effective cloud-edge based adaptation method.
edge devices is limited due to computational
2.1 EFFICIENT ADAPTATION FOR ROBUSTNESS AND COMMUNICATION ENHANCEMENT
In this paper, we devise Cloud-Edge Elastic Model Adaptation (CEMA) paradigm in which the adaptation task is decomposed to the cloud and edges based on their computational resources and budgets. The edge only requires performing a vanilla model inference, while the remaining adap- tation workloads are offloaded to the cloud (see Figure 2). Then, our CEMA selectively uploads a subset of samples, determined by our proposed entropy-basse based criteria (refer to Section 2.2). to the cloud for adaptation. This selective sample uploading strategy significantly reduces the communica- tion burden. Once the cloud adaptation process complete, the edge model updates its parameters from the cloud and then infers the next incoming test samples. Importantly, CEMA introduces no extra computational cost in edges and is applicable to resource-constrained edge devices.
In the cloud, we seek to adapt the edge model with uploaded test samples. Specifically, we seek to leverage a foundation model fe(-) with stronger capability and more parameters to guide the edge model for adaptation (refer to Section 2.3). Notably, the foundation model does not require access to training data and updates through unsupervised entropy minimization. To maximize data utilization for adaptation, devise replay buffer to store the uploaded samples. When transferring
3

---

Published as a conference paper at ICLR 2024
Algorithm Adaptation process in edge.
Algorithm 2 Adaptation process in cloud.
Require: Test samples [xest=(xx j=1= the edge
Require:
Test samples X'= ) n=1 the foundation model fo(-) and edge model (.)
model (t). parameters B, Emmx and Fimin-
1: for batch (xxxx) ) in Drest do
2: Calculate predictions y for all x E X via fe())
Update parameters Ō E 0 of the foun- dation model fo(-) via entropy mini- mization (Eqn. 5) with A.
3: Calculate S(x) via Eqn. (4) with Faax and minn-
4: Update the threshold Emax via Eqn. (2).
5: Upload samples {x (x)S())= xEX} to cloud.
Update parameters ú E of the edge model Iu (-) via knowledge distilla- tion (Eqn. 6) with X.
6: Update the parameters v E from the cloud.
7: end for
Ensure: The predictions (0) k= for all x E Drest-
3:
Distribute the parameters ő to edge.
knowledge from the foundation model to the edge model via knowledge distillation, exploit both the newly uploaded samples and the samples from the replay buffer for higher data utilization efficiency. This results in better performance of the edge model compared to vanilla adaptation. The pseudo-code involved in the edge and cloud is presented in Algorithms and 2, respectively.
2.2 SAMPLE FILTRATION FOR COMMUNICATION COST REDUCTION IN EDGE SIDE
To reduce the communication overhead between the cloud and edges, we propose a sample filtration strategy that removes high and low entropy test samples from being uploaded to the cloud. A recent study by Wang et al. (2021) proposes a model adaptation method that adapts on a batch of test samples by conducting entropy minimization. Minimizing entropy penalizes decisions high densities in the data distribution to improve accuracy for distinct classes (Grandvalet & Bengio, 2004), which has proven to be crucial constraint for domain adaptation (Saito al., 2019; Roy et al., 2019). Furthermore, Niu et al. (2022) has demonstrated that high entropy samples adversely affect the adaptation performance when entropy minimization is employed. The reason may be that the model adapts using the test samples without labels via entropy minimization, introducing considerable uncertainty when dealing with high-entropy samples during the adaptation process.
However, this method only filters test samples based on a static and pre-determined threshold. suffers two limitations: 1) The entropy of samples tends to decrease along with the adaptation. Therefore, only a part of the negatively impacting samples can be excluded. 2) It overlooks the fact that adaptation with extremely low-entropy samples is unnecessary. To address the above issues, we propose to 1) dynamically exclude the unreliable (high entropy) samples by adaptively adjusting the threshold accordance with the entropy of current samples. 2) exclude the low-informative (low entropy) samples. We design the entropy-based filtration criteria as it is an information-theoreet measure that represents uncertainty and information and has proven to be a simple yet effective strategy (Settles, 2010; Margatina et al., 2021; Ren et al., 2022). To this end, devise binary score S(x) to indicate whether a sample x should be uploaded. We only upload the test samples with 5xx)=1 and discard those with S(x)=0.
Dynamic identification on unreliable samples. Let (.) denote an indicator function. Following (Niu et al., 2022), we exploit a entropy threshold Emax to filter the high entropy samples as follows
Sampian w) Erucl
Sampler (E Evacl
100%
90%
Propertion
80%
Shigh (x) 1 [EE(x)) Fmna (x),
(I)
70%
where E(x;w) denotes the entropy of the prediction Iw (x) for the sample x. we perform adaptation through entropy minimization, the entropy of the test samples is likely to decrease. Consequently, a fixed threshold Emax would progressively exclude fewer and fewer samples during the adaptation. To sub- stantiate this, perform empirical study to veal the proportions of the test samples whose en- tropy is larger than fixed Fmax different stages
60%
0k-10k
10k-200
20k-30k 30k 30k-40k 40k-30k
Online Test Samples
Figure 3: Proportions of test samples with (x; w)> Emax (red) and =( (blue) during adaptation via entropy mini- mization on ImageNet-C.
4

---

Published as a conference paper at ICLR 2024
during the adaptation process. As shown in Figure 3, in the final stage, we filter out fewer samples (indicated by the red bar) compared to the initial stage. This trend could be attributed to the predic- tions becoming more certain (i.e., predictions with lower entropy) as the adaptation progresses.
In light of the aforementioned empirical study, it becomes feasible to filter out more high-entropy samples by dynamically decreasing the Emax. To this end, we seek to lower Emax according to the average entropy of the test samples after every adaptation batch. To be specific, in the adaptation can be calculated by
batch I, the entropy threshold E max
E avg Eivg
E
mux
EX-1 max
where Evg denotes the average entropy of all test samples in past t batches, À is saper-parametes.. Note that Bag can be obtained from the first batch of the test samples. Based on Eqn. (2), when the average entropy becomes smaller, Emax would be descended accordingly. In this way. we exclude more unreliable samples from uploading the cloud, thereby enhancing communication efficiency.
Identification on low-informative samples. In addition to high entropy samples, test samples with extremely low entropy are unnecessary for adaptation. Since they only contribute negligible gradient while minimizing the entropy loss. Following the similar scheme above, we employ threshold Emin to discard samples with entropy lower than Emii- We do not adopt dynamic variation strategy on Emi since the threshold that determines whether sample contributes negligible gradient does not depend on the average entropy of current test samples. Formally, Stow (x) can be written as
glow (x) = 1 (E(x:0)>F (x).
(3)
The overall binary score (xx can be calculated by
S(x) = ghigh (x). glow (x)
(4)
Note that the edge model only requires one regular forward propagation without the need for back- ward propagation gradient descent. Once the edge model (-v(-) makes predictions for test sam- ples, it asynchronously uploads the samples with S(x)=1 to the cloud using a background thread. We emphasize that uploading samples does not block the edge from inferring next incoming samples. In other words, the processes of inference and uploading can be executed simultaneously.
2.3 REPLAY-BASED KNOWLEDGE DISTILLATION FOR ADAPTATION IN CLOUD SIDE
Recent studies have demonstrated that larger models with great number of parameters often achieve better performance than small one on the out-of-distribution data (Hendrycks & Dietterich, 2019). Compared with the vanilla adaptation, it is possible to further improve the edge model (-(() in the cloud by distillation from high-performing practical that the foundation model has knowledge that covers all the test samples inferred by the edge model (see results and analysis in Table 8). Since cloud server has much more sufficient com- putational resources and budgets to support the heavier models. In this case, our proposed CEMA would adopt a foundation model to enhance the adaptation performance, which takes ad advantage of the rich computational resources of the cloud.
foundation model fo(-). Note that it is feasible and
However, it non-trivial distill the foundation model fo(-) the edge model (+) Note that vanilla distillation training requires a large number of samples (Chen et al., 2019; Yu et al., 2023). The amount of accessible test samples is limited as we exclude unreliable and low-informative sam- ples in the edge. To alleviate this issue, we devise replay buffer B to store the uploaded samples inspired by (Mnih et al., 2013). each distillation step, we first update the foundation model fe(.) via unsupervised entropy minimization in unsupervised manner on the uploaded samples. Through this adaptation process, the foundation model acquires additional knowledge of the cur- rent test data. Then, we boost the edge model gw (-) via the knowledge distillation guided by the foundation model on both the uploaded samples and the samples from the replay buffer.
Entropy minimization for foundation model adaptation. Upon receiving the batch of uploaded test samples {==(x;]W - first put them into the replay buffer B-B U X. Note that the capacity
5

---

Published as a conference paper at ICLR 2024
of the buffer is limited (see analysis and ablations in Table 16). We update the buffer with the newly uploaded test samples in a first-in and first-out manner. Subsequently, we update the foundation model over the batch &. Adaptation with a batch not only avoids a trivial solution (iee, assigning all probability to the most probable class (Wang al., 2021)) but also improves the parallel efficiency in the cloud server. Formally, we adapt fo) by minimizing the weighted entropy loss LENT(f((x))
min (x) > folulx) log Jo(y)x)
(5)
MEC
where H(x)=1/ exp (E(x;0)- and C denotes the model output space. In Eqn. (5), we opti- mize the loss weighted by (x) following (Niu et al., 2022) since we seek to encourage the model to focus on the low-entropy test samples during adaptation. In the poopaarppantiion process, it is worth noting that the gradient is not propagated through (x).
For efficient adaptation, we only update the affine parameters of normalization layers dEo while keeping the remaining layers fixed. The advantages are three folds: 1) This model updating strategy is efficient since it requires less memory footprint and computational resources. 2) Compared with distributing all the parameters to the edge, our CEMA only needs to transfer BN parameters and lowers downloading overhead (e.g., reduce 99.91 % parameters distributing burden in ResNet-18). 3) The model may preserve the previous knowledge since most parameters remain unchanged, leading to better adaptation stability and performance (see results in Table 17).
Knowledge distillation with replay bulfer for edge model adaptation. With the updated founda- tion model, we seek to adapt the edge model g (-) on both d' and set of test samples randomly sampled from the replay buffer B. Specifically, we align the predictions between the logits fa(.) and (-) via Kullback-Leible (KL) divergence LKI. In addition, we adapt (-) via cross-entropy loss LCE, in which the pseudo labels !/ are generated by the foundation model. Note that ý can be calculated by arg c (fo (z))). Formally, optimize gw (-) by employing both entropy minimization and knowledge distillation as follows,
min (x) [a (sw (x), fx(x)) + ( 9) + (ENT(G((x))))
where a and / are factors for balancing the losses. we use the KL divergence to align the prediction distribution of the foundation and edge models. and the CE loss to align the decision boundaries. These two kinds of losses make the edge model learn the knowledge in complementary manners from the foundation model. The combination of both losses outperforms the adoption of either one in isolation (See results and analyses in Table 12).
In edge model adaptation, we still only update the affine parameters of normalization layers new. After distilled adaptation of the edge model (-), we distribute the parameters V to edge devices to update (-). In particular, once the cloud has completed adaptation, we can perform the reception of parameters from the cloud in the background, typically via separate thread to avoid blocking the inference process. Upon full reception of the parameters, we would update the parameters of the edge model at the beginning of the next inference iteration.
3 EXPERIMENTS
In the following, we provide the details of the used datasets and implementation, more details are put in the supplementary materials. The code available at http://hwwh..oo/ccceeeooooo//aaa
Datasets and models. We evaluate our method and considered methods on ImageNet-C (Hendrycks & Dietterich, 2019). which is a distribution shift dataset by applying main category corruption (i.e., noise, blur, weather, and digital). with total of 15 diverse corruption types, to the ImageNet valida- tion set. Each corruption has different severity levels (iee. from level 1 to 5), which the higher severity level indicates the more severe distribution shift. We also verify our CEMA on ImageNet- R (Hendrycks et al., 2021), which contains 30,000 images with various artistic renditions of 200 ImageNet classes collected from Flickr. We adopt ResNet101 (He et al., 2016) as the foundation model and ResNet18 as the edge model in the main experiments. We explore more foundation models and edge models in the ablation studies (see results in Tables 13 and 14).
Implementation details. We set the entropy thresholds =0.4 x C as the initialized value following (Niu et al., 2022) and Fmii= 00.02 x In C, where C denotes the number of classes. Then

---

Published as a conference paper at ICLR 2024
Comparisons with state-of-the-art methods ImageNet-C (severity level 3 and 5) garding Accuracy ( (%). We adopt Resnet101 as the foundation model and ResNet18 as the edge model. denotes the TTA method that does not require any backward propagation and can be locally executed in edge devices. Note that our CEMA requires uploading fewer samples on average with severity levels (19.1k vs. 26. 1k) and (14.4k vs. 18.8k) than ETA.
Table 1:
Noise
Blur
Weather
Digital
Severity Level=3
Ceuss.
Impul.
Motion
Fog
Brit.
Elastic
Pixel
JPEG
Avg.
ResNet18 (baseline)
21.6
19.9
18.7
29.9
15.8
28.7
27.6
27.6
23.8
35.5
62.7
38.1
51.8
41.6
53.0
33.1
BN Adaptation!
42.3
39.8
40.0
37.5
31.4
45.1
44.3
40.8
36.2
53.9
65.0
58.2
60.2
58.0
57.7
47.4
ONDA!
40.0
38.9
37.5
29.5
27.5
43.8
41.9
40.2
35.2
54.6
65.1
56.1
59.7
58.6
57.6
45.9
LAME!
20.6
18.9
17.2
29.5
14.7
28.3
26.9
26.8
23.2
34,9
62.4
37.5
$1.3
41.1
52.5
32.4
PI.
48.1
48.0
46.1
41.1
39.7
51.3
49.9
47.3
39.8
58.6
64,9
59.2
62.5
60.8
59.4
51.8
Tent
47.2
47.1
45.1
40.0
38.2
50.4
49.4
46.7
40.1
58.1
64.9
59.0
62.5
60.5
59.2
51.2
COTTA
39,8
46.3
41.9
36.5
64.9
60.2
59.3
58.1
FTA
50.1
50.2
48.6
44.0
42.7
52.9
51.4
49.9
41.5
59.5
65.2
60.9
62.9
61.6
59.9
53.5
CEMA (Ours)
51.1
51.2
49.8
45.2
44.1
53.7
52.0
50.8
44.2
60.1
65.0
61.1
62.9
61.6
59.8
54.2
Soverity Level=5
Gauss,
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
ResNett8 & (baselinc)
1,5
2.3
1.5
11.4
8.7
11.1
17.6
10.6
16.2
14.0
51.5
3.4
16.5
23.3
30.7
14.7
BN Adaptation!
16.6
16.2
17.3
18.6
18.2
25.9
34.7
28.4
29.8
41.2
58.5
22.2
40.1
45.3
38.0
30.1
ONDA!
13.7
14.1
23.7
29.4
58.5
39.3
37.5
LAME
0.9
1.1
0.6
11.2
8.2
10.8
17.0
8.7
15.6
12.4
51.1
3.3
14.9
22.5
30.1
13.9
24.8
26.8
24.6
20.3
21.3
33.6
41.8
39.0
32.4
49.9
59.5
11.4
47.9
51.5
47.0
35.4
Tent
22.8
25.0
23.2
32.4
33.5
59.3
46.9
50.6
45.9
Col TA
15.2
16.2
15.7
11.8
14.9
26.9
36.9
31.2
29.9
43.6
59.2
17.0
40.9
47.2
39.3
29.7
26.8
29.7
27.6
22.6
22.7
37.1
42.4
37.6
60.1
26.1
49.8
53.3
48.5
CEMA (Ours)
29.8
32.2
30,3
25.3
26.8
39.3
45.3
43.7
38.7
52.8
60.1
32,9
50.8
54.0
49.3
40.8
the threshold Emax decreases based on Eqn. (2) with A=1.0. For the adaptation of the foundation and edge model, we use both an SGD optimizer with learning rate of 0.00025 and a momentum of 0.9. For the adaptation of the edge model, we set the batch size to 128, in which 32 samples are newly uploaded and the remaining 96 samples are randomly sampled from the replay buffer. The hyper-parameter a and B3 are both set to 3.
Compared methods. We compare
methods with the following kate-of-the of-t----tt TTA methods, including BN Adaptation (Schneider et al., 2020), ONDA (Mancini et al., 2018), LAME (Boudiaf et al., 2022), Pseudo Label (PL) (Lee et al., 2013), Tent (Wang et al., 2021), CoTTA (Wang al., 2022) and ETA (Niu et al., 2022). In the experiments, we assume the edge devices only have limited resources and thus are unable to perform backpropagation. All workloads in the above TTA methods invoked in backpropagation would be executed in the cloud by uploading test samples. In this way, we can compare the amount of data transmission between our method and the counterparts.
3.1 PERFORMANCE COMPARISONS ON IMAGENET-C
We compared our proposed CEMA with the considered methods in Table 1 in ImageNet- C (Hendrycks & Dietterich, 2019) with the severity levels and 5. We adopt a CNN-based model ResNet101 as the foundation model and ResNet18 the edge model in this experiment. We put more experimental results with the severity levels 1, 2, and 4 as well as the results on transformer- based models in the supplementary due the page limitation. From the results, our CEMA achieves the highest accuracy in most 15 different corruption types and the best average accuracy (e.g., 40.8% with the severity level 5). To be specific, CEMA outperforms Tent (29.8% vs. 22.8%) and ETA (29.8% vs. 26.8%) on the corruption Gaussian noise with the severity level 5. The reasons are twofold: 1) the proposed filtration strategy removes more harmful test samples; 2) our replay- based distillation scheme transfers distribution knowledge to the edge model. The results verify the effec- tiveness of the proposed sample filtration strategy and distilled adaptation method.
In Figure 4, we compare the average number of uploaded samples over 15 corruption types of our CEMA and the considered methods with ResNet18 as the edge model. Note that the compared methods PL, Tent and CoTTA need to upload the whole test samples (50k samples, 100%) to the cloud for adaptation. These methods introduce great communication overhead between the cloud and edges, which is inefficient in the context of cloud edge model deployment. ETA excludes some test samples with high prediction entropy but still requires uploading 26. lk (52%) and 19. (38%) samples in severity levels 3 and 5, respectively. With the proposed dynamic sample filtration

---

Published as a conference paper at ICLR 2024
Comparisons with state-of-the-art methods on ImageNet-R benchmark. We adopt ResNet101 as the foundation model and 8 as the edge model. 1 denotes the TTA method that does not require any backward prop-
Table 2:
Table 3: Comparisons with Tent and ETA with
mixture of 15 corruption types on ImageNet-G C.
Model
Accuracy (%)
#Uploaded samples
ResNeti8 (baseline)
33.1
Tent
30,5
750k (100%)
agation and can be executed in edge devices, which does not require uploading test samples.
ETA
44.5
384k (51%)
CEMA (Ours)
45.6
(38%) %
Table 4: Effect of sample filtration strategy.
Model
Acc. (5-)
#Uploaded samples
Method
Level
Level
ResNet18 (baselinc)
20.4
Acc. (%)
#Upload
Aec. (8)
#Upload
BN Adapt.
22.8
Shii"((] (xx wi Fixed Eus.
51.2
25,325
30.0
15,173
ONDA
22.7
+ Dynamic Eya. (x)
51.1
20,976
29.9
10,081
51.1
17,479
29.8
LAME
20.2
PI.
23.8
30,000 (100%)
Table 5: Effect of the replay buffer.
Tent
23.6
30,000 (100%)
Level
Level 5
ETA
26.2
7,457 (25%)
Replay Buffer
Acc. (%)
#Upload
Acc. (%)
#Upload
CoTTA
23.2
30,000 (100%)
47.4
17.654
25.0
9,084
CEMA (Ours)
27.9
5,944 (20%)
$1.1
17,479
29.8
9,889
AApp Upfonded Semples
308
51
UUpla Samples
51
a0k:
Accuncy
30
Accuncy
30
25k
26.11 L.
20k-
19.11
25.5L
14.4k
10g
108
46
0.5
0.9
0.02
0.05
0.05
Figure 4: Comparisons of the average number of uploaded test samples on ImageNet-C C
Figure 5: Effect of À
Figure 6: Effect of Emin in
Eqn. (2) with ResNet18 as edge model on ImageNet-C (Gaus- sian noise, severity level 3).
Eqn. (3) with ResNer18 as edge model on ImageNet-C (Gaus- sian noise, severity level 3).
with the severity levels and 5.
strategy, our CEMA further reduces the number of uploaded samples to 18.8k (37%) and 14.4k (29%) severity levels and 5. Compared with ETA, remove more high-entropy samples with a dynamic threshold and further exclude low-entropy samples. In this case, our proposed CEMA only needs upload fewer samples the cloud for adaptation, which demonstrates the superiority of our methods over existing methods in terms of the amount of data transmission.
3.2 PERFORMANCE COMPARISONS ON IMAGENET-R
In Table 2, we report the results of our CEMA and considered methods on a realistic out-of- distribution dataset ImageNet-R. This benchmark dataset collected from Flickr instead of gen- erated by some corruption algorithms (i.e., as ImageNet-C done) and filtered by Amazon MTurk annotators according to the class names in the original ImageNet. From the results, our CEMA achieves 27.9% accuracy on ImageNet-R (+4.1% 1% over Tent, +1.7% over the best counterpart ETA). As for the number of uploaded test samples, our CEMA only requires 5944 (20%) samples, which lower than ETA (7457, 25%) and much lower than Tent, PL and CoTTA (30,000, 100%). These results are consistent with those on ImageNet-C that our proposed CEMA yields the highest robust accuracy with the fewest uploaded samples. The results further demonstrate the effectiveness and potential of our method while applied to realistic test samples in real-world applications.
3.3 FURTHER EXPERIMENTS
In ablation studies, we consider two representative severity levels (i.e,, and 5). Severity levels 3 and 5 represent the medium-difficnt distribution shift and the most challenging distribution shift, spectively, By adopting these two levels, we effectively assess the performance of various adaptation algorithms. Moreover, the hyper-parameters derived from these levels demonstrate their adaptability
8

---

Published as a conference paper at ICLR 2024
and suitability for addressing other severity levels as well. For similar consideration, the settings are widely adopted by Tent (Wang et al., 2021) and ETA(Niu et al., 2022).
Due to the page limitation, We put more ablations in the supplementary materials, including the effects of 1) hyperparameters Emax. a and B, 2) components in distillation loss (Eqn. (6)), 3) different foundation models, 4) different edge models, 5) different filtration strategies 6) replay buffer size, 7) different parameters updating schemes and 8) different parameters distribution intervals.
Comparisons under mixed-and-shiffter distributions. We evaluate CEMA on mixed ImageNet-C in severity level 3 that consists of 15 different corruption types/distribution shifts (750k images in total). Note that it is common for the edge model to encounter test samples with mixed dis- tribution shifts in practice, In Table 3, compared with Tent and ETA, our CEMA outperforms them in both the accuracy (+ 15.15.1 % over Tent and +1.1% 1% over ETA) and the number of uploaded samples (-62% than Tent, - 13% than ETA). The results show the effectiveness of CEMA for applications in complex scenarios with distriiiittiitti samples.
Effect of À in Eqn. (2). To investigate the effect of X in Eqn. (2), we perform more experiments with different À E {0.6, 0.7, 0.8, 0.9, 1.0, 1.1]. From Figure 5, when À becomes larger, our CEMA would remove fewer test samples and upload more samples to the cloud, and the robust accuracy improves since we transfer the more contributed samples to the cloud for adaptation. The robust accuracy is highest (51.1%) when 1=1.0 and keep unchanged while 1>1.0. Considering larger À trade-off and
leads to more communication burden, we select A=1.0 for the efficiency-perfommae fix À to 1.0 for all other experiments. Experimental results in Tables 1, and 2 demonstrate that this fixed À works well, indicating that the À in our CEMA is not sensitive to different datasets.
Effect of Emin in Eqn. (3). We evaluate 0.03, 0.04, 0.05, 0.06]. Note that the larger Emin is, the more low-entropy samples are excluded. In this case, the out-o--distribution performance would decrease since we may remove some con- tributed samples during adaptation. From Figure 6, the accuracy drops when Emin is larger than 0.02. Therefore, we set the entropy threshold Emin to 0.02 across various datasets, including ImageNet-C (15 corruption types and 5 severity levels) and ImageNet-R. Extensive experiments show that our CEMA works well with the chosen hyperparameters on different datasets and various severity levels.
CEMA with different Emin. selected from {0.01, 0.02,
Effect of components in sample filtration strategy. We perform an ablation experiment in Table 4 to verify the components in the proposed sample filtration strategy. Compared with the baseline that removes test samples based on 5hiigh (x) with fixed Frmux (the same as ETA), introducing dynamic Fmax in Eqn. (2) achieves comparable accuracy (51.1% % vs. 51.2 51.2%) in severity level 3 with fewer up- loaded samples (20,976 vs. 25,325). When further removing low-entropy samples based on Slow (x) in Eqn. (3), the number of uploaded samples further decreases (e.g.. 25,325 + 17,479). The results demonstrate the effectiveness of our proposed sample filtration strategy. We further demonstrate the superiority of our filtration strategy over the random uploading strategy in Table 15.
Effect of the replay buffer. In Table 5, report the performance of our CEMA on ImageNet-C (Gaussian noise, severity level 3) with/without the replay buffer. Note that the replay buffer is able to provide more samples for distillation and boost the data utilization efficiency in our cloud edge adaptation. With the replay buffer, our CEMA achieves much higher accuracy (51.1% vs. 47.4%) on ImageNet-C (Gaussian noise, severity level 3). The reason is that we reuse samples from the replay buffer for knowledge distillation. This leads to more sufficient model updates when the foundation model transfers the knowledge to the edge model. The experimental results demonstrate the effectiveness of the proposed replay buffer in knowledge distillation.
CONCLUSION
In this paper, we devise a Cloud-Edge Elastic Model Adaptation (CEMA) paradigm that improves the model adaptation performance by leveraging both the cloud server and edge devices. In the adaptation, we highlight that our CEMA does not introduce any extra computational cost in the edge devices. Specifically, we devise sample filtration strategy to exclude unnecessary samples from the cloud for adaptation. It reduces the data transmission overhead between the cloud and edge and thus improves the adaptation efficiency. Besides, we adopt a powerful foundation model to guide the edge model for adaptation via the proposed applay-based knowledge distillation. Extensive experimental results several benchmark datasets demonstrate the effectiveness of our CEMA.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was partially supported by National Natural Science Foundation of China (NSFC) 61836003 (key project), National Natural Science Foundation of China (NSFC) 62072190, the Ma- jor Key Project of PCL PCL2023A08 and TCL Science and Technology Innovation Fund.
REPRODUCIBILITY STATEMENT
In this work, we implement our CEMA with different models on ImageNet-C and ImageNet-R datasets. Reproducing all the results in our method depends on the following three aspects:
1. DATASET.
The second paragraph of Section 3 and Appendix C.I provide the details of the adopted dataset and the download url.
MODELS. All adopted models (with the pre-trained weights) for test-time adaptation are publicly available. The download url is provided in Appendix C.2.
3. PROTOCOLS OF EACH METHOD. Appendix C.2 provides the implementation details of our CEMA and compared methods. The source code of our CEMA has been made publicly available.
REFERENCES
Aibek Alanov, Vadim Titov, and Dmitry P Vetrov. Hyperdomainnet: Universal domain adaptation for generative adversarial networks. Advances in Neural Information Processing Systems, pp- 29414-29426, 2022.
Alexander Bartler, Andre Bühler, Felix Wiewel, Mario Döbler, and Bin Yang. Mt3: Meta test- time training for self-supervised test-time adaption. In International Conference on Artificial Intelligence and Statistics, pp. 3080-3090, 2022.
William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018
Prashant Bhat, Elahe Arani, and Bahram Zonooz. Distill on the go: Online knowledge distillation in self-supervised learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 2678-2687, 2021.
Malik Boudiaf, Romain Müller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test- time adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 8334-
8343, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, pp. 1877-1901, 2020.
Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers. In AAA/ Conference on Artificial Intelligence, pp. 1430-3437, 2020a.
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In IEEE International Conference on Computer Vision, pp. 3513-3521, 2019.
Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasser- stein contrastive representation distillation. IEEE Conference on Computer Vision and Pattern Recognition, pp. 16296-1638 16305, 2021a.
10

---

Published as a conference paper at ICLR 2024
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 5008-5017, 2021b.
Ting Chen, Simon Korbliihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597-1607, 2020b.
Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael Franklin, Joseph E Gonzalez, and Ion Stoica. Clipper: A low-latency online prediction serving system. In USENIX Symposium on Networked Systems Design and Implementation, pp. 613-627, 2017.
Zeshuai Deng. Zhuokun Chen, Shuaicheng Niu, Thomas Li, Bohan Zhuang, and Mingkui Tan. Effi cient test-time adaptation for super-resolution with second-orde degradation and reconstruction. In Advances Neural Information Processing Systems, volume 36, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain general- ization via model-agnostic learning of semantic features. In Advances Neural Information Processing Systems, pp. 6447-6458, 2019.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183-1192 2017.
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoen- coders. volume 35, pp. 29374-29338 2022.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference Learning Representations, 2018.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Ad- vances in Neural Information Processing Systems, pp. 529-536, 2004.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A criti- cal analysis of out-of out-of-distribution generalization. In IEEE Conference on Computer Vision and Puttern Recognition, pp. 8340-8349, 2021.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv, abs/1 503.02231, 2015.
Neil Houlsby, Ferene Huszar, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv, abs/1 0tbs/111..57 12.5744, 2011.
Xuefeng Hu, Gokhan Uzunbas, Sirius Chen, Rui Wang, Ashish Shah, Ram Nevatia, and Ser- Nam Lim. Mixnorm: Test-time adaptation through online normalization estimation. arXiv,
abs/2110.11478. 2021.
Sergey Ioffe. Batch remormalization: Towards reducing minibatch dependence in batch- normalized models. In Advances in Neural Information Processing Systems, pp. 1945-1953, 2017.
Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for aogees-agnostcc do- main generalization. Advances Neural Information Processing Systems, 34:2427-2440, 2021.

---

Published as a conference paper at ICLR 2024
Lu Jiang, Deyu Meng, Shoou-I Yu, Zhen- Lan, Shiguang Shan, and Alexander G. Hauptmann. Self-paced learning with diversity. In Advances in Neural Information Processing Systems, pp- 2014.
Ansh Khurana, Sujoy Paul, Piyush Rai, Soma Biswas, and Gaurav Aggarwal. Sita: Single image test-time adaptation. arXiv, 0bss2211.00235,, 2021.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of wn-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664, 2021.
M. Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems, pp. 1189-1197, 2010.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In International Conference on Machine Learning Workshop, volume 3, pp. 896, 2013.
Li Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning generalize: Meta-learning for domain generalization. In AAAI Conference on Artificial Intelligence, pp. 3490-3497, 2018.
Jian Liang. Dapeng Hu, and Jiashi Feng. Do really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learming. pp. 6028-6039, 2020.
Hongbin Lin, Yifan Zhang, Zhen Qiu, Shuaicheng Niu, Chuang Gan, Yanxia Liu, and Mingkui Tan. Frotolype-guidee continual adaptation for class-inrementte unsupervised domain adaptation. In European Conference on Computer Vision, pp. 351 1-68. Springer, 2022.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick. James Hays, Pietro Perona, Deva Ramanan, Piotr Doll'a r, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. arXiv, abs/1405.033 405.0312, 2014.
Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexan- dre Alahi. Tit++: When does self-supervised test-time training fail or thrive? In Advances in Neural Information Processing Systems, volume 34, 2021.
Yufan Liu, Jiajiong Cao, Bing Li. Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 7096-7104, 2019.
Zhijian Liu, Zhanghao Wu, Chuang Gan, Ligeng Zhu, and Song Han. Datamix: Efficient privacy- preserving edge-cloud inference. In European Conference Computer Vision, pp. 578-595, 2020.
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 11966-1 11976, 2022.
Zheqi Lv, Wenqiao Zhang, Shengyu Zhang, Kun Kuang, Feng Wang, Yongwei Wang, Zhengyu Chen, Tao Shen, Hongxia Yang, Beng Chin Ooi, and Fei Wu. Duet: A tuning- free device-clous collaborative parameters generation framework for efficient device model generalization. Pro- ceedings of the ACM Web Conference, pp. 3077-3085, 2023.
Massimiliano Mancini, Hakan Karaoguz, Elisa Ricci, Patric Jensfelt, and Barbara Caputo. Kitting in the wild through online domain adaptation. In JEEE/RSJ International Conference Intelligent Robots and Systems, pp. 1103-1109, 2018.
Katerina Margatina, Giorgos Vernikos. Loic Barrault, and Nikolaos Aletras. Active learning by acquiring contrastive examples. In Proceedings of Conference on Empirical Methods in Natural Language Processing, pp. 650-663, 2021.
12

---

Published as a conference paper at ICLR 2024
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. arXiv, abs/1312.5502,,21 2013.
Zachary Nado, Shreyas Padhy, Sculley, Alexander Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv, abs/2006. 10963, 2020.
Mahdi Pakdaman Nacini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In AAAI Conference on Artificial Intelligence, pp. 2901- 2015.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International Conference Machine Learning, pp. 16888-16905, 2022.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In International Conference on Learning Representations, 2023.
Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, Li Lao, Fangwei Li, Vinu Rajashekhar, Sukriti Ramesh, and Jordan Soyke. Tensorflow-serring: Flexible, high-performance ml serving. arXiv, abs//771..0639, 2017.
Seyed Ali Osia, Ali Taheri, Ali Shahin Shamsabadi, Kleomenis Katevas, Hamed Haddadì, and Hamid Rabiee. Deep private-feature extraction. IEEE Transactions on Knowledge and Data Engineering, 32(1):54 66, 2018
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In JEEE Conference Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
Jizong Peng, Ping Wang. Christian Desrosiers, and Marco Pedersoli. Self-paced contrastive learn- ing for semi-supervised medical image segmentation with meta-labels. In Advances in Neural Information Processing Systems, pp. 16686-16699, 2021.
Viraj Prabhu, Shivam Khare, Decksha Kartik, and Judy Hoffman. Sentry: Selective entropy opti- mization via committee consistency for unsupervised domain adaptation. In IEEE International Conference Computer Vision, pp. 8558-8567, 2021.
Biao Qian, Yang Wang. Hongzhi Yin, Richang Hong, and Meng Wang. Switchable online knowl- edge distillation. In European Conference Computer Vision, volume 13671, pp. 449 466, 2022.
Zhen Qiu, Yifan Zhang, Hongbin Lin, Shuaicheng Niu, Yanxia Liu, Qing Du, and Mingkui Tan. Source-free domain adaptation via avatar prototype generation and adaptation. In International Joint Conference Artificial Intelligence, pp.
2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. In International Conference Learning Representations, 2018.
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 779-788, 2016.
Pei Ren, Xiuquan Qiao, Yakun Huang, Ling Liu, Calton Pu, and Schahram Dustdar. Fine- lineggrineed elastic partitioning for distributed dnn towards mobile web services in the 5g era. IEEE Trans- actions on Services Computing, 2021.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Yao Huang, Zhihui Li, Brij B. Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. ACM Computing Surveys, 54(9):180:1-1188 2022.
13

---

Published as a conference paper at ICLR 2024
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In International Conference on Learning Representations, 2015.
Subhankar Roy. Aliaksandr Siarohin, Enver Sangineto, Samuel Rota Bulò, Nicu Sebe, and Elisa Ricci. Unsupervised domain adaptation using feature-whitening and consensus loss. In IEEE Conference on Computer Vision and Puttern Recognition, pp. 9471-9480, 2019.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. In IEEE International Conference on Computer Vision, pp. 8049-8057, 2019.
Enver Sangineto, Moin Nabi, Dubravko Culibrk, and Nicu Sebe. Self paced deep learning for weakly supervised object detection. IEEE Transactions on Puttern Analysis and Machine Intelligence, 41 (3):712-725, 2019.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In Advances in Neural Information Processing Systems, volume 33, pp. 11539-11551, 2020.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
Burr Settles, Active learning literature survey. University of Wisconsin, 52, 07 2010.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time train- ing with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning, pp.
2020.
Evgenii Tsymbalov, Maxim Panov, and Alexander Shapeev. Dropout-based active learning for gression. In Analysis of Images, Social Networks and Texts, pp. 247-258, 2018
Han Vanholder. Efficient inference with tensorrt. In GPU Technology Conference, volume pp. 2, 2016.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Repre- sentations, 2021.
Ji Wang, Jianguo Zhang, Weidong Bao, Xiaomin Zhu, Bokai Cao, and Philip S Yu. Not just privacy: Improving performance of private deep learning in mobile cloud. In Proceedings of SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2407-2416 2018.
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 7191-7201, 2022.
Zhiquan Wen, Shuaicheng Niu, Ge Li, Qingyao Wu, Mingkui Tan, and Qi Wu. Test-time model adaptation for visual question answering with debiased self-supervisions. IEEE Transactions Mullimedia, 2023.
Min Xue, Huaming Wu, Ruidong Li, Minxian Xu, and Pengfei Jiao. Eosdnn: An efficient offloading scheme for dnn inference acceleration in local-edge-cloud collaborative environments. IEEE Transactions Green Communications and Networking, 6(1):248-264) 2021.
Zhendong Yang, Zhe Li, Yuan Gong, Tianke Zhang, Shanshan Lao, Chun Yuan, and Yu Li. Re- thinking knowledge distillation via cross- cross-entropy. arXiv, htbb///20..003999.. 2022a.
Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked gener- ative distillation. In European Conference Computer Vision, volume 13671, pp. 53-69, 2022b.
Han-Jia Ye, Su Lu, and De-Chun De-Chuan Zhan. Generalized knowledge distillation via relationship match- ing. IEFE Transactions Pattern Analysis and Machine Intelligence, 45(2);1817-1834, 2022.
14

---

Published as a conference paper at ICLR 2024
Shikang Yu, Jiachen Chen, Hu Han, and Shuqiang Jiang. Data-free knowledge distillation via feature exchange and activation region constraint. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 14266-24275 2023.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. In Advances in Neural Information Processing Systems, 2022a.
Yifan Zhang, Ying Wei, Qingyao Wu, Peilin Zhao, Shuaicheng Niu, Junzhou Huang, and Mingkui Tan. Collaborative unsupervised domain adaptation for medical image diagnosis. IEEE Transac- lions Image Processing, 29:7834-7844, 2020.
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Sele-super aggregation of diverse experts for test-agnostic long-tailed recognition. In Advances in Neural Information Processing Systems, volume 35, pp. 34077-340990 2022b.
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 4320-4328, 2018.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 11943-1 11952, 2022.
Bowen Zhao, Chen Chen, and Shu-Tao Xia. DELTA: Debiased Fully Test-time Adaptation. In International Conference Learning Representations, 2023.
Zhi Zhou, Xu Chen, En Li, Lickang Zeng, Ke Luo, and Junshan Zhang. Edge intelligence: Paving the last mile of artificial intelligence with edge computing. Proceedings of the IEEE, 107(8): 1738 1762, 2019.
15

---

Published as a conference paper at ICLR 2024
APPENDIX
CONTENTS
Related Work
17
More Discussions on CEMA
19
B.1 Transmission efficiency of CEMA
19
B.2 Adaptation throughput and required upload bandwidth
19
B.3 Availability in variable bandwidth scenarios
19
B.4 Adaptation and parameter updating mechanisms
19
B.5 Entropy- based criterion on overeonfidence models
19
More Implementation Details
21
C.I More details on datasets
21
C.2 More experimental protocols
21
D More Experimental Results on ImageNet-C
23
D.1 More comparisons with CNN-basee models ImageNet-C
23
D.2 More comparisons with Transformer-based models ImageNet-C
24
D.3 Availability with CLIP foundation models
24
More Ablation Results
27
E.1 Effect Emax in Eqn. (1)
27
E.2 Effect of a and / in Eqn. (6)
27
E.3 Effeet of knowledge distillation loss
27
E.4 Potential of strong foundation models.
28
E.5
Applicability different edge models.
28
E.6
Effectiveness of entropy- based criteria
.2..............................
29
E.7 Effect the replay buffer size
................................
29
E.8 Effect of different updating ways
30
E.9 Effect of updating interval in edge
30
E0
Comparisons with more sample identification strategies
30
E.11 More comparisons object detection
31
16

---

Published as a conference paper at ICLR 2024
A RELATED WORK
Test-time adaptation (TTA) (Sun et al., 2020; Wang et al., 2021) recently has shown great potential in handling distribution shifts between training and testing data, by directly adapting the pre-trained model on given test sample to learn the shifts. Specifically, test-time training (Sun et al., 2020) updates the model at test time via self-supervised image rotating prediction (Gidaris et al., 2018). After that, numerous methods (Zhang et al., 2022b; Wang et al., 2022; Bartler et al., 2022; Niu et al., 2022; 2023; Wen et al., 2023) have been further devised to improve and broaden the application scope of TTA. For example, TTT+ (Liu et al., 2021) and MT3 (Bartler et al., 2022) introduce self-supervised contrastive learning (Chen et al., 2020b) to provide supervision for model updating, which achieves better adaptation performance. Meanwhile, entropy-based methods (Wang et al., 2021; Niu et al., 2022; 2023; Zhang et al., 2022a) update the model by minimizing the unsupervised entropy of model outputs. However, the above methods rely on backpropagation at test time, which may be infeasible for resource-limited edge devices. To avoid this issue, one can update the model via backpropagation-ffes manner, such as BN statistics adaptation (Schneider et al., 2020; Hu al., 2021: Nado et al., 2020; Khurana et al., 2021), classifier adjustment (Iwasawa & Matsuo, 2021) 1) and reconstruction learning (Gandelsman et al., 2022; Deng et al., 2024). Though these methods improve the efficiency of TTA, they may yield inferior performance due to the insufficient model update.
In this work, we focus on deploying backpropagation-bant based model adaptation to cloud-edge systems to boost model adaptation performance. Considering the computational resources of the cloud and edges, our method only performs forward propagation without model updating on edge devices and does not introduce any extra computational cost. Instead, we adapt the model in the cloud by uploading only partial test samples from the edges. In this way, we allocate all the heavy test-time adaptation workloads to the cloud and reduce the computational cost of edges.
Knowledge distillation. Knowledge distillation (Hinton et al., 2015) is an effective method to obtain simple and efficient student models by transferring knowledge from complex teacher models. According to the types of extracted knowledge, knowledge distillation can be divided into logits- based distillation methods (Hinton et al., 2015; Yang al., 2022a; Zhao et al., 2022), feature- distillation methods (Romero et al., 2015; Chen et al., 2021a;b; Yang et al., 2022b) and relation- based distillation methods (Park et al., 2019; Liu et al., 2019; Ye al., 2022). The above methods are mostly based on the pre- pre-triine teacher models for offline distillation. Related to our method, online distillation methods (Zhang et al., 2018; Chen et al., 2020a; Bhat et al., 2021; Qian et al., 2022) can train teacher models and student models simultaneously in the absence of strong teachers. In this paper, we exploit knowledge distillation to leverage a stronger foundation model the cloud to boost the robustness of an edge model in the context of cloud-edge inference.
Collaborative cloud-edge inference. Conventional cloud-based model inference (Olston et al., 2017; Vanholder, 2016; Crankshaw et al., 2017) has an unacceptable responsive latency concern, which limits its applications in practice. To overcome these shortcomings, collaborative cloud-edge inference (Osia et al., 2018; Wang et al., 2018; Xue et al., 2021; Ren et al., 2021; Liu et al., 2020) leverages both the computational power on the edge and the cloud by dynamically allocating the workloads on them, which introduces plenty of advantages (Zhou et al., 2019), such low response latency and on-demand deployment. These methods mostly focus on the vanilla model inference task, i.e., model simply takes a test instance as input and outputs its predictions. Unlike these methods, we seek to improve the robustness in cloud-edge inference via test-time adaptation to alleviate the distribution shift issue. To this end, we propose divide the TTA task into several subtasks (including edge model inference and adaptation) and offload them efficiently between the cloud and edges.
Related to our CEMA, DUET (Lv et al., 2023) and Sxppereetworkbassee method (Alanov et al., 2022) seek to generate model parameters in the test time to address the distribution-shifeed issue with cloud-edge collaboration. However, DUET requires uploading all test samples to update the param- eter generator. While our CEMA excludes unnecessary samples from uploading to the cloud. Thus our CEMA may be applied to bandwidth-limited scenarios. Moreover, DUET updates the parameter generator in a supervised way, which requires source training data and corresponding ground-truth labels. While CEMA performs model adaptation in self-supervised manner only with unla- beled samples. Our CEMA may be applicable to more practical scenarios. The hyperNetwork- method requires prior specification of domain shift categories (i.e, represented by texts) when
17

---

Published as a conference paper at ICLR 2024
training its HyperDomainNet. While in our CEMA, the domain shifts unknown in the training stage. Consequently, the application of HyperDomainNet to our task presents inherent difficulties.
Informative sample identification seeks quantitatively measure the information contained in given sample and then develop sample-aware learning strategies, which has proven to be effective in many areas, such active learning (Ren et al., 2022; Gal et al., 2017; Sener & Savarese, 2018; Beluch et al., 2018; Tsymbalov et al., 2018 8) and domain adaptation (Zhang et al., 2020; Prabhu et al., 2021). For instance, BLAD (Gal et al., 2017) measures the sample's 's information by estimating the mutual information between model parameters and model predictions, and dropout-based meth- ods (Tsymbalov et al., 2018) calculate the variation over a set of model predictions with dropout to quantify the information. Sener & Savarese (2018) propose core-set approach to select samples that can mostly represent the whole dataset. SENTRY (Prabhu et al., 2021) and CoUDA (Zhang et al., 2020) measure the sample information via the prediction consistency of different data aug- mentations and networks, respectively.
In our CEMA, one key challenge is to reduce the data transmission costs in the context of efficient online test-time cloud-edge model adaptation. We achieve this by employing the idea of sample selection. To be specific, we devise an entropy-based thresholding technique to exclude partial samples from the model adaptation process. Here, we adopt the entropy as the sample's 's information measurement, it is easy use and efficient. The entropy can be calculated over a single sample and only involves one-time forward propagation, unlike prior methods that may rely on a whole dataset (Sener& & Savarese, 2018) less-efficient multiple forward propagations (Tsymbalov al., 2018; Prabhu et al., 2021). Nonetheless, compared with EATA (Niu et al., 2022) in which the authors exploit a static thresholding strategy to select unreliable samples, our sample selection strategy is different in the two aspects: i) we reveal that the suitable threshold may change continuously along with the online adaptation process and propose dynamic unreliable sample identification strategy; ii) we also introduce a low-informative sample selection strategy to identify samples that produce negligible gradients for model updating.
Self-paced learning. Motivated by the learning principle of humans, self-paced learning (Kumar et al., 2010) automatically reorders samples during training based on their difficulty. For instance, SPL (Kumar et al., 2010) and SP-MIL (Sangineto et al., 2019) iteratively select subset of the most reliable images for model updates. To enhance the diversity of the selected samples, SPLD (Jiang et al, 2014) pre-cluster the training data and encourages balance samples section from different clus- ters. Furthermore, SP-CON (Peng et al., 2021) jointly learns the important weights for each sample during training. Based on this, they conduct self-paced learning with the importance weights incor- porated within the loss. Despite both self-paced learning and CEMA improving learning efficiency by active sample selection, self-paced learning focuses learning robustness instead of mitigating computation cost. Consequently, it differs from CEMA in several aspects: 1) Self-paced learning initiates the learning process with an easy subset of samples, which includes even those of low in- formativeness. In contrast, our CEMA approach specifically focuses selecting samples that are both informative and reliable for adaptation. These two kinds of samples are beneficial the adap- tation process. 2) Self-paced learning offline selects the whole dataset for training the remaining samples become easier. While our CEMA dynamically adjusts the threshold to continually filter out less reliable samples. This is conducted online, focusing on maintaining communication efficiency.
18

---

Published as a conference paper at ICLR 2024
B MORE DISCUSSIONS ON CEMA
B.I TRANSMISSION EFFICIENCY OF CEMA
We would like to highlight that our CEMA reduces the uploading and downloading communication cost from two aspects. 1) Reducing uploading communication cost: We design entropy- -assed cri- teria to exclude unreliable and low-informative samples. It reduces 60% uploading communication overhead on ImageNet-C (Gaussian noise, severity level 3) benchmark. 2) Reducing downloading communication cost: We only update and transfer the affine parameters in BN layers instead of all the layers. Compared with the distribution of all the parameters in R (11.68 MB), our CEMA only needs to transfer 0.0096 MB parameters and lowers 99.91 1 downloading communica- tion overhead.
B.2 ADAPTATION TIROUGHIPUT AND REQUIRED UPLOAD BANDWIDTII
Taking ResNet101 as the foundation model and ResNet18 as the edge model, our CEMA at 220 images/second on NVIDIA A 1100 GPU. This is sufficient to support edge devices for real- time inference (60 images/second). On the edge side, we assume the edge model infers 60 images/second. Based on that our CEMA reduces 60% uploading communication overhead on ImageNet-C (Gaussian noise, severity level 3), it needs to upload around 24 images the cloud per second. Each image on ImageNet-CC is around 28 KB in disk after jpeg compression. In total, the edge device needs to upload around 24=282672 KB data the cloud per second.
B.3 AVAILABILITY IN VARIABLE BANDWIDTHI SCENARIOS
In practice, the bandwidth between the cloud and the edge may change constantly. Nevertheless, in certain contexts, such as surveillance systems in industrial parks, bandwidth not major concern since 1) our CEMA only requires relatively low communication overhead and 2) the cloud and edge devices are commonly interconnected through high-speed wired networks (typically with 100Mb/s bandwidth). Introducing the bandwidth variable in CEMA method would significantly increase the complexity, which may make the adaptation performance unstable.
We intend to address this issue by exploiting uploading queue. Specifically, we will upload test samples with the queue in the background thread, while ensuring that it does not block fore- ground model inference tasks. When the queue reaches its maximum capacity, will discard the test sample with the highest entropy in the queue. Through this mechanism, our CEMA is able dynamically adjust the number of uploaded samples according the available bandwidth. In this sense, it is equivalent to adjusting the hyperparameter À in Eqn. (2) according to the available bandwidth instead of manually pre-defining it. Thus, our proposed method is well-suited handle diverse and varying bandwidth conditions.
B.4 ADAPTATION AND PARAMETER UPDATING MECHANISMS
In our CEMA, we feed sample into the edge model and then determine whether this sample would be uploaded to the cloud based on Eqn. (4). Once receiving a batch of N uploaded samples in the cloud, the edge model would be adapted for one time via Eqn. (6). After adaptation, the edge model would update the parameters from the cloud. Thus the next coming sample would be inferred via the updated edge model. We have also taken into account scenarios of poor network connectivity (as detailed in Table 18). In this case, the edge model would be adapted for K time (K>1) 1) via Eqn. (6). Then the updated parameters would be distributed to the edge and the next coming sample would be inferred via the updated edge model.
B.5 ENTROPY-BASED CRITERION ON OVERCONFIDENCE MODELS
Recent studies have demonstrated that neural networks can exhibit overconfidence, which may have an influence on the measure of uncertainty based entropy- criteria. To evaluate the over- confidence in our edge model, we employ a commonly used metric Expected Calibration Error (ECE) (Naeini al., 2015). ECE measures the average differences between the model's predicted
19

---

Published as a conference paper at ICLR 2024
confidence and its actual accuracy across various confidence intervals. A lower ECE value indi- cates reduced overconfidence in the model's predictions. Compared with the best counterpart ETA (ECE=6.67%)) our CEMA achieves ECE=3.07% This substantial reduction in ECE suggests that the edge model in CEMA exhibits significantly less overconfidence. Consequently, the uncertainty estimation using entropy in CEMA could potentially be more accurate than that in ETA.
Moreover, is important to acknowledge that certain degree of overconfidence is an inherent aspect of neural network models. Despite this, our CEMA demonstrates robust capability effectively filter out unreliable and low-informative samples. This effectiveness indicates that the impact of overeonfidence on our CEMA is limited. Therefore, even the presence of inherent overconfidence in neural networks, the approach adopted by CEMA to assess uncertainty and selectively upload test samples validated.
20

---

Published as a conference paper at ICLR 2024
C MORE IMPLEMENTATION DETAILS
C.I MORE DETAILS ON DATASETS
ImageNet-C", We evaluate our method and the counterparts on ImageNet-C (Hendrycks & Diet- terich, 2019), which is a widely used benchmark dataset for out-of-distribution generalization. is built based on the validation set of the original ImageNet by corrupting the images. Coneretely, as shown in Figure 7. ImageNet-C includes 15 different corruption types, i.e., Gaussian noise, shot noise. impulse noise, defocus blur, glass blue, motion blur, zoom blur, snow. frost, fog, brightness, contrast, elastic transformation, pixelation, and JPEG compression. Each corruption has five differ- ent severity levels (i.e., from level 1 to level 5). Note that the larger severity level indicates a more severe distribution shift.
ImageNet-R². We also evaluate our CEMA and compared methods ImageNet-R (Hendrycks et al., 2021), which contains 30,000 images with various artistic renditions 200 ImageNet classes, which are primarily collected from Flickr and filtered by Amazon MTurk annotators.
Gaussian Noise
Shot Noise
Impulse Noise
Defocus Blur Frosted Glass Blu
Motion B
Zoom Blur
Snow
Frost
Fog
Brightness
Contrast
Elastic
Pixelate
JPEG
Figure 7: Visualizations of corrupted images with 15 corruption types in ImageNet- C benchmark, which are taken from the original paper of ImageNet-C (Hendrycks & Dietterich, 2019). Each corruption type has 5 severity levels, resulting 75 distinct corruptions.
C.2 MORE EXPERIMENTAL PROTOCOLS
CEMA (Ours). In sample filtration strategy of edge devices, we set the entropy threshold Em4 In C' as the initialized value following (Niu et al., 2022), where C' denotes the number of classes. Then the threshold Emax decreases based on Eqn. (2) with 1=1.0. We set another entropy threshold Emin=0.02 x In C' in the experiments. In addition to the removal of high/low-entropp test samples, use the criteria in (Niu et al., 2022) to remove those samples with similar gradients (called Non-redundant Sample Identification test-time adaptation of both the foundation model fec.) and the edge model Iw (+), we optimizer with a learning rate of 0.00025 and a momentum of 0.9. We set the batch size to 32 while optimizing the foundation model.
the original paper) for all the experiments. For the SGD
For the adaptation of the edge model, we set the batch size to 128, in which 32 samples are newly uploaded and the remaining 96 samples are randomly sampled from the replay buffer. We set the hyper-parameteer and 3 is set to 3 and 3, respectively. We set the replay buffer size to 10,000. Note
http://ttht....oooo/eee/////ette Phttps: :/gi thub. http:///ttttb..ooo//eeoo/////gge
21

---

Published as a conference paper at ICLR 2024
that all the models used in experiments pretrained on ImageNet training and available publicly. Specifically, ResNet18, ResNet101, MobileNetV2, MobileNetV3 and ShuffleNetV2 are from torchvision. DeiT-tiny and DeiT-base are from facebookzesearch/ deit." CLIP- ViT-B/32 from opena / CCIIP.' Since the edge model gar(+) may infer with a small batch size (e.g., the batch size is 1), introduce Batch Renormalization (loffe, 2017) to replace the vanilla Batch Normalization in the edge model following (Zhao et al., 2023). In this case, Iw(-) is able infer over a very small batch of test samples with moving average statistics instead of batch re-computing statistics.
Compared methods. We compare our methods with the following ofate-of-the-art TTA methods. BN Adaptation (Schneider et al., 2020) uses the weighted sum of training moving average BN statistics and batch re-computing statistics, which both the batch size and prior strength are set to 256. ONDA (Mancini et al., 2018) adapts batch normalization statistics over a batch of test samples with an exponential moving average, in which the momentum is set to 0.9, Pseudo Label (PL.) (Lee et al., 2013) adapts the model with the hard label generated by the model self. We train PL using an SGD optimizer with a learning rate of 0.001. Tent (Wang et al., 2021) updates the BN affine parameters via entropy minimization. The learning rate is set 0.00025 and the batch size is set to 64. Based on Tent, ETA (Niu et al., 2022) removes test samples with high entropy and similar gradients. The entropy constant En is set to 0.4 In C, where C/ is the number of task classes. The is set to 0.05. COTTA (Wang et al., 2022) reduces the error accumulation by using weight-averaged and augmentation-averet
predictions. We use 32 augmentations and the augmentation threshold is set to 0.1. The learning rate is set to 0.01. The restoration probability is set to 0.01. LAME (Boudiaf' al., 2022) modifies the output probability of the classifier instead of the parameters of the model itself. We use the KNN kernel with 5 nearest neighbors.
22

---

Published as a conference paper at ICLR 2024
Table 6: Comparisons with state-of-the-art methods ImageNet-C (severity levels 1, 2 and 4) regarding Accuracy ( (%). We adopt Resnet101 as the foundation model and ResNet18 as the edge model. denotes the TTA method that does not require any backpropagation and can be locally executed in edge devices. The bold number indicates the best result and the underlined number indicates the second- bccond-best result.
Noise
Weather
Digital
Severity Level=1
Gauss.
Shot
Impul.
Defic:
Glass
Motion
Zoom
Snow
Frost
Fog
Bril.
Contr.
Elastic
Pixel
JPEG
Avg.
ResNetL8 (baseline)
49.2
46.8
35.1
$1.7
48.7
57.1
44.0
46.4
52.6
52.8
67.6
58.1
60.3
59.8
59.3
52.6
BN Aduptation!
59.3
57.9
51.9
$7.2
57.4
62.4
54.9
54.1
51.7
61.3
67.9
64.9
62.2
64.8
632
59.8
ONDA!
58.7
58.0
51.3
55.2
56.1
62.3
55.0
53.6
57.2
61.9
68.3
64.6
62.3
65.1
63.0
59.5
LAME!
48.8
46.3
34.1
$1.4
48.2
56.8
43.6
45.9
52.3
52.5
67.3
57.8
60.1
59.5
58.9
52.2
60.5
60.3
55.9
58.8
60.2
63.6
58.0
57.5
59.0
63.1
67.7
65.4
62.6
65.3
63.5
61.4
Tent
60.6
60.2
55.4
58.4
50.0
63.6
57.8
56.9
58.9
63.0
67.8
65.0
62.6
65.2
63.6
61.3
CoTTA
59.1
58.7
52.6
56.7
57.4
62.8
56.0
54.5
57.6
62.2
67.8
64.6
62.5
65.1
63.1
60.0
57.1
63.9
59.5
67.7
62.9
65.6
CEMA (Ours)
61.7
61.5
58.2
59.8
60.8
61.5
58.8
59.2
59.6
63.6
67.1
65.2
62.5
65.3
63.4
62.0
Severity Level=2
Gauss.
Shot
Impul.
Defic:
Glass
Motion
7oom
Snow
Frost
Bri.
Contr.
Flastic
Pixel
JPFG
Avg.
ResNet18 (baselime)
37.9
25.8
45.2
59.7
BN Adaptation!
52.5
49.6
45.5
50.4
48.7
55.7
48.6
39.4
45.1
58.3
66.8
62.7
46
63.8
60.2
52.9
ONDA!
51.3
49.1
43.6
45.5
45.7
55.2
48.4
39.9
44.1
59.0
67.2
61.9
46.5
63.4
60.1
52.1
LAME
24.7
38.8
PL
55.6
54.6
50.6
52.6
53.7
58.9
53.5
47.9
48.6
61.4
66.5
63.0
48.5
64.7
61.3
56.1
Tent
55.1
54.0
49.7
52.0
52.7
58.7
52.9
46.3
48.3
61.0
66.6
63.0
48.4
64.4
61.2
55.6
COTTA
52.6
50.5
45.5
48.5
48.0
56.4
50.0
41.9
45.4
59.9
66.8
62.6
46.9
63.6
60.3
53.2
ETA
$7.0
56.1
52.6
54.4
$5.2
59.8
54.8
50.0
50.2
62.2
66.6
64.0
49.1
64.9
61.4
57.2
CEMA (Ours)
53.5
66.4
49.2
Severity Level=4
Gauss.
Shot
Impul.
Defoe:
Glass
Motion
Zoom
Snow
Frost
Fog
Brii.
Contr.
Elastic
Pixel
JPEG
Avg.
ResNett8 (baselinc)
7.8
6.2
6.5
18.6
12.1
16.2
22.1
17.5
21.9
28.4
57.9
14.2
39.3
26.4
43.6
22.6
BN Aduptation!
29.8
24.9
28.2
26.8
25.7
33.0
39.2
31.4
35.0
50.5
62.2
44.6
54.4
49.0
49.0
38.9
ONDA!
26.9
23.6
24.8
19.0
21.4
30.7
38.7
31.2
33.8
51.4
62.3
37.4
54.0
49.5
48.7
LAME!
6.9
5.2
5.1
18.3
11.4
15.8
21.4
16.8
21.3
27.3
57.6
13.4
38.6
25.6
43.1
21.8
36.1
36.6
33.7
40.7
40.3
38.2
62.5
59.0
54.9
53.9
Tent
36.7
34.5
34.8
29.6
31.6
39.8
45.4
38.9
38.9
55.7
62.3
45.5
58.6
54.2
53.4
44.0
CoTTA
29.2
25.6
27.1
21.2
23.3
34.7
41.4
32.8
35.4
53.2
62.5
43.8
54.9
$1.6
50.1
39.1
ETA
41.0
39.3
33.5
44.0
43.5
42.5
62.9
59.7
56.4
54.6
CEMA (Ours)
38.5
40.4
41.2
35.4
38.4
45.1
48.6
44.9
43.1
58.0
62.9
52.6
60.0
56.9
55.7
48.0
Avv. Uplowded Sumples
TenTPL.COTTA
ETA
CETTA (surs)
50k-
40%
50.4k
28.0k
22.0k
22.9%
20k-
16.8%
Level
Level
Level
Figure 8: Comparisons with Tent, PL, CoTTA and ETA CNN-based models in terms of the
average number of uploaded test samples on ImageNet-C with the severity levels 1, 2 and 4.
D MORE EXPERIMENTAL RESULTS ON IMAGENET-C
D. MORE COMPARISONS WITH CNN-BASED MODELS ON IMAGENET-C
In Table 6, we provide more results to compare our CEMA with state-of-the-an methods on ImageNet-C with the severity levels 1, 2 and 4. We adopt Resnet101 the foundation (teacher) model and ResNet18 as the edge (student) model. Our CEMA achieves the highest average accuracy with severity levels 2 and 4. As for the corrupted images with level I, our CEMA yiclds competitive performance with the best counterpart ETA. The possible reason is that the corruption the images in level is very slight. In this case, the foundation model may share close performance with the edge model and be hard to transfer knowledge to it.
We also compare the communication overhead of our CEMA with Tent, PL, CoTTA and ETA in Figure 8. From the results, our CEMA needs to upload smaller number of test samples to the cloud. For instance, CEMA only requires 22.0k uploaded samples, which is fewer than Tent
23

---

Published as a conference paper at ICLR 2024
30k
Lovele1
Lovd=2
Lovel=3
Lovdle
Lovd--5
# Uploades Samples
20k-
10kk
Gauss.
Stot
Impuise
Defocus
Glass
Motion
Zoom
Snow
Frost
Fog
Bright
Contrant
Elastic
Pisel
JPEG
Figure 9: The number of uploaded test samples on ImageNet-C with different corruption types and severity levels.
(50.0k) and ETA (22.0k). The reason is that we exclude both high-entropy and low-entropy samples in the adaptation process. This improves efficiency in bandwidth-limited cloud-edge systems. The results demonstrate the superiority of our CEMA over the data transmission. In addition, we report the number of uploaded samples of our CEMA on ImageNet-C with different corruption types and severity levels in Figure 9.
D.2 MORE COMPARISONS WITH TRANSFORMER -AAEEE MODELS ON IMAGENET-C
To verify the effectiveness of our CEMA on transformer- based models, we conduct more experi- ments ImageNet-C using DeiT-base as the foundation model and Deit-tiny as the edge model. Since the baseline methods BN Adaptation and ONDA depends on batchnorm layers, we do not compare these two baselines layers. Note that we only update the parameters in layernorm layers instead of batchnorm layers in transformer-based models.
cansformer-based models since these models have no batchnorm
From the results in Table 7, our CEMA outperforms the baselines Tent (44.5% vs. v8.21.8%), COTTA (44.5% vs. 26.5% and ETA (44.5% vs.44.5% greatly in severity level 5 on ImageNet-C. Since our CEMA 1) removes test samples that are harmful for the adaptation and 2) introduces a foundation model to transfer adapted knowledge to the edge model. Moreover, from Figure 10, our CEMA reduces more communication burden than the baseline methods, such as Tent, PL and ETA. For instance, our CEMA only requires uploading 13.9k test samples to the cloud, much lower than ETA (25.3k) and Tent (50.0k). The reason is that our CEMA devises entropy-based sample filtration strategy, which excludes high/low entropy samples without helpful information. Note that Tent and PL achieve poor performance when the severity level is 5 since they adopt high-entropy samples with harmful information for adaptation. In sum, the above results on transformer-bai based models further verify the effecttienees of our CEMA.
D.3 AVAILABILITY WITH CLIP FOUNDATION MODELS
We would like to highlight that it is common and practical that the foundation model possesses knowledge that covers the test samples inferred by the edge model. This can be achieved through several means: 1) simultaneously training a stronger foundation model and an edge model on the same training data; 2) using CLIP the foundation model to circumvent the need train one. Given the impressive ability for zero-shot classification of CLIP, it is likely to possess knowledge that covers test samples across extensive scenarios. For those cases that even CLIP cannot handle, the complexity exceeds the scope of our current research, and we would defer it to future studies.
In this section, we conduct more experiments on ImageNet-R to demonstrate that our CEMA works well with CLIP as the foundation model. Note that the CLIP model is pretrained based its private data and does not access the training data of the edge model (ResNet 8). Table 8, we adopt
as the foundation model and ResNet18 the edge model to perform cloud- edge adaptation. From the results, our CEMA consistently outperforms Tent and ETA. The results demonstrate that our CEMA integrates effectively with the CLIP foundation model in practice. We
24

---

Published as a conference paper at ICLR 2024
Table Comparisons with state-of-the-un methods on ImageNet-C regarding Accuracy (%). We adopt DeiT-base as the foundation model and DeiT--iny as the edge model. denotes the TTA method that does not require any backpropagation and can be locally executed in edge devices. The bold number indicates the best result and the underlined number indicates the second-best result.
Noise
Blur
Weather
Digital
Severity Level=1
Gauss.
Shot
Impul
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Bril
Contr.
Elastic
Pixel
JPEG
Avg.
DerT-tiny (buseline)
63,0
62.8
60.7
56.5
55.8
63.4
46.1
57.4
62.4
59.3
69.6
65.0
64.0
61.2
62.4
60.6
LAME
62.8
62.5
60.5
56.0
55.2
63.0
45.4
57.0
62.0
59.0
69.2
64.7
63.6
60.8
62.1
60.3
64.3
62.5
65.8
60.3
63.7
70,4
66.0
65.3
Tent
64.5
64.3
62.5
62.1
63.0
66.0
56.9
61.0
63.9
64.7
70.4
67.7
66.2
66.1
65.0
64.3
COTTA
63.2
61.3
57.2
56.7
63.9
46.9
58.3
63.0
60.4
69.8
65.9
64.4
62.1
63.1
61.3
ETA
64.8
64.6
62.9
62.4
63.9
66.2
59.2
61.8
64.2
65.5
70.3
68.0
66.3
67.4
65.8
64.9
CEMA (Ours)
65.3
64.9
63.2
63.0
64.4
66.5
60.1
62.5
64.1
65.5
70.2
67.8
66.2
68.0
66.2
65.2
Severity Level=2
Gauss.
Shot
Impul.
Defoc.
Glass
Mation
Zoom
Snow
Frost
Fog
Brit.
Contr,
Elastic
Pixel
JPEG
Avg.
DeiT-tiny (haseline)
58.4
57.0
54.5
50.5
45.0
56.0
37.4
41.3
52.5
54.1
68.4
63.2
43.6
56.2
59.0
53.1
LAME!
58.2
56.8
54.2
50.0
44.2
55.6
36.6
40.8
52.1
53.4
68.1
63.0
43.2
55.8
58.7
52.7
60.2
59.6
57.5
56.6
54.8
61.1
47.4
48.0
54.5
69.5
49.0
63.6
61.4
58.1
Tent
60.5
59.7
57.6
57.4
56.7
61.8
50.7
50.0
55.7
63.3
69.5
66.3
50,9
64.6
62.4
59.1
CoTTA
59.0
57.7
55.2
51.1
45.8
57.0
38.
42.3
53.4
55.6
68.7
64.2
44.3
57.3
59.7
54.0
60.2
58.2
57,9
62.3
52.3
69.7
53.0
66.4
CEMA (Ours)
61.4
611.6
58.9
58.6
39.2
62.7
55.8
54,1
$7.4
63.9
69.4
66.6
53.6
67.2
64.2
60.9
Severity Level=3
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frust
Bril.
Contr.
Elastic
Pixel
JPEG
DeiTitiny (baseline)
43.8
55.5
47.6
56.8
LAME!
48.9
47.7
48.3
37.5
19.2
43.5
30.8
44.3
43.8
46.2
66.4
60.3
55.1
47.0
56.4
46.3
52.7
52.8
53.1
46.1
35.6
53.3
42.4
49.8
46.9
58.4
67.9
63.7
62.3
58.4
59.6
53.5
Tent
53.1
53.1
53.4
47.9
41.0
54.7
46.3
51.5
48.2
60.0
68.1
64.1
63.8
60.1
60.7
55.1
COTTA
49.8
48.8
49.4
39.0
20.9
45.1
32.1
46.0
45.4
49.0
67.0
61.6
56.5
49.0
57.5
47.8
54.2
54.2
56.1
53.7
68.1
64.7
62.4
62.0
CEMA (Ours)
55,0
55.1
55,1
50.5
48.5
57.1
$2.9
55.4
51.8
60.2
68.4
64.3
65,5
63.4
63.0
57.7
Severity Levele4
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Brit.
Contr.
Elastic
Pixel
JPEG
DeiT-tiny (baseline)
30.4
33.4
16.1
30.2
26.0
36.5
43.2
63.8
44.7
25.8
49.2
LAME!
35.6
30.1
33.0
27.0
14.7
29.7
24.9
35.9
42.8
39.0
63.6
48.6
44.0
25.2
48.8
36.2
PL.
42.2
39.7
42.0
34,3
4.5
43.5
10.7
42.1
42.3
2.7
65.8
55.9
55.3
49.2
53.5
38.9
66.0
58.3
52.4
CoTTA
36.6
31.4
34,1
28.2
16.3
31.5
26.4
37.5
44.3
44.0
64.5
50.8
45.6
26.7
50.0
37.9
ETA
44.9
43.0
44.8
41.4
42.2
48.3
47.7
48.2
50.0
59.4
66.3
57.6
61.2
55.8
57.2
51.2
CEMA (Ours)
46.7
45.1
46.2
42.6
43.6
50.1
48.4
50.4
50.8
56.4
66.5
56.3
62.0
57.7
58.6
52.1
Severity Levele6
Gauss.
Tmpul.
Define
Mation
7anm
Snow
Fog
Rrit.
Contr.
Flastic
JPRG
Avg.
DeiT-riny (haseline)
17.0
18.2
17.4
19.2
12.6
22.9
20.9
32.6
37.6
32.9
59.6
23.9
23.5
10.3
38.5
25.8
LAME!
17.9
22.4
21.3
10.1
Pl.
1.0
2.3
1.1
17.8
2.4
35.9
3.6
9.4
15.2
0.8
62.8
38.6
3.9
35.9
46.2
18.5
Tent
13.3
13.6
27.1
38.7
11.7
14.6
63.2
41.3
2,4
47.8
21.8
CoTTA
17.6
18.8
18.1
19.7
12.7
23.9
21.0
33.7
38.7
34.8
60.4
24.4
24.1
10.6
39.3
26.5
ETA
32.1
33.7
33.4
33.8
35.0
42.9
43.4
45.9
46.0
53.2
63.9
33.9
50.2
50.6
51.0
43.1
CEMA (Ours)
36.7
36.2
44.8
64.0
49.9
50.1
52.8
Avg Uppoaded I Samples
Tea/PLCSSTA
ETA
CBTTA
50k
50.0k
50.0k
50.0k
50.0k
50.0%
40k
30k
32.9%
30.0k
25.3k
20k
16.8k
14.9
13.9k
10k
Level
Level
Level
Level
Level
Figure 10: Comparisons with Tent, PL, CoTTA and ETA on Transformer based models in terms of
the average number of uploaded test samples on ImageNet-C with the severity levels 1 5.
also verify that the edge model is able to be guided by different simultancously training foundation models in Table 13.
25

---

Published as a conference paper at ICLR 2024
Table 8: Comparisons with Tent and ETA ImageNet-R. We adopt CLIP-ViT-B/32 the founda- tion model and ResNet18 as the edge model.
Model
Acc. (%)
#Uploaded samples
ResNet18 (baseline)
20.4
Tent (Wang et al., 2021)
23.6
30,000 (100%)
ETA (Niu et al., 2022)
26.2
7,457 (25%)
CEMA (Ours)
26.9
6,521 (22%)
26

---

Published as a conference paper at ICLR 2024
E MORE ABLATION RESULTS
E. EFFECT OF EMAX IN EQN. (1)
Our CEMA employs hyperparameters Emax and Emin and in Eqns. (1) and (3), respectively. These two hyperparameters denote the thresholds while excluding high/low entropy test samples. In prac- tice, we determine these hyperparameters through systematic approach guided by the following principles: 1) We set these hyperparameters to be linked the number of classes instead of grounded in absolute terms. Specifically, we prescribe its value as x inC, where 0 < Y < 1 and C' denotes the number of classes. Consequently this hyperparameter exhibits a robust insensitivity tions in the number of classes. 2) Next we select the appropriate values of these hyperparameters by examining the impact of hyperparameters on the ImageNet-C (Gaussian noise, severity level 3) dataset. Based on the results, we select relatively well-performing hyperparameters and keep them as constants by default across different datasets, including ImageNet-R and various corruption types and severity levels within ImageNet-C.
fluctua-
In our entropy-based sample filtration, we adopt an entropy threshold Emax to remove high-entropy samples. Since the high-entropy samples may have negative impact adaptation performance. In Table 9, we report the adaptation performance when we upload less/more high entropy samples by adjusting Fmaxs When Finax is small, our CEMA removes too many samples during adaptation and thus it is hard to learn helpful knowledge from the remaining samples. When Emux is too large, some high-entropy samples would participate in the adaptation and contribute harmful gradients, resulting in performance degradation. From the results, our CEMA achieves the highest adaptation performance when 00.4 Thus set Emax to 0.4 in our experiments.
Effect of the entropy threshold Emax of our CEMA on ImageNet-C (Gaussian noise, severity level=3).
Table 9:
Emax
0.2
0.3
0.4
0.5
0.6
Accuracy (%)
50.2
50.8
51.1
50.5
46.4
#Uploading samples
10,560
14,276
17.479
20,067
21,931
E.2 EFFECT OF a AND 3 IN EQN. (6)
We conduct ablation studies to investigate the effect of different hyper-paramette a and in Eqn. (6) in Table 10 and 11, respectively. Here, and 33 both are selected from {1, 2, 3, 4, 5, 6}. From the results, when a=3 our CEMA achieves the best performance (51.1%). As for the hyper-parameter 8, we obtain the highest accuracy (51.1%) when 3=3. Thus, we set a=3 and 3=3 in the experiments. Gaussian noise, including ImageNet-R and ImageNet-C (encompassing 15 types of corruption and 5 severity levels, 75 different scenarios in total). Extensive experiments show that our CEMA works well with these hyperparameters.
other scenarios on various datasets, we keep them the same as those on
Effect of hyper-parameter a in Eqn. (6) on ImageNet-C (Gaussian noise, sever- ity level 3),
Table 10:
Table 11:
Effect of hyper-parameter 3 in Eqn. (6) on ImageNet-C (Gaussian noise, sever- ity level 3).
3
4
5
8
2
3
4
5
Accuracy (%)
50.5
$1.0
51.1
51.0
50.6
50.3
Accuracy (%)
50.7
51.0
51.1
50.7
50.4
50.0
E.3 EFFECT OF KNOWLEDGE DISTILLATION LOSS
While updating the edge model via Eqn. (6), entropy minimization is not equal to optimizing with the pseudo label from its predictions. Since the entropy minimization and pseudo label approaches serve different functions to improve the performance of the edge model. The pseudo labels provide a direct label for the corresponding sample to align the decision boundaries. On the other hand, entropy
27

---

Published as a conference paper at ICLR 2024
minimization encourages the model to make more confident predictions by penalizing uncertainty in its output distribution. Furthermore, it is important to clarify that the entropy minimization approach operates on the average entropy acToss the entire batch of samples, rather than focusing the confidence of individual samples. This methodology aims to optimize the overall confidence of the model, guiding it towards each sample to converge towards its highest predictive probability. This distinction underscores the complementary nature of our CEMA, utilizing both cross-entropy loss for decision boundary alignment to the foundation model and entropy loss for enhancing the model's 's collective confidence.
assured predictions
collective basis, rather than encouraging
In Table 12, we report the performance of our CEMA with/without LKL as well as Lee in Eqn. (6). The baseline without CKL andCcE employs only CENT- From the results, with the KL divergence LKL. our CEMA achieves better accuracy than the baseline (CENT) (50.5% vs. 50.0%). Since the foundation model transfers its knowledge on out-of-distribution samples to the edge model via fKL- In addition, our CEMA further improves performance with LCE- The results show that the combination of both losses outperforms the adoption of either one in isolation (51.1% 50.5%). It highlights the necessity and significance the CE loss.
Table 12:
Effect of components in the loss for updating the edge model on ImageNet-C (Gaussian noise) with & the edge model.
LENT
LCE
LKI.
Accuracy (%)
#Uploads
X
50.2
18,497
X
50.0
17,263
V
X
V
50.5
17,379
V
50.7
17,315
V
51.5
17,479
E.4 POTENTIAL OF STRONG FOUNDATION MODELS.
In the cloud, we can exploit stronger foundation models for adaptation when we have more computa- tional budgets. In this sense, our CEMA has great potential in real-world applications. In Table 13, we report the results on ImageNet-C (Gaussian noise, severity level 3) with different foundation models, namely, ResNet101, ResNet152 and ConvNeXt-T (Liu et al., 2022). Note that ResNet152 and ConvNeXI-T are stronger foundation models, which outperform ResNet101 on ImageNet. From the results, as the foundation model becomes stronger, the edge model yields higher accuracy. The results show the potential of CEMA with stronger foundation models in real-world applications.
Table 13: Effect different foundation models.
Level 3
Level
Foundation Model
Acc. (%)
#Upload
Acc. (%)
#Upload
ResNet101
51.1
17,479
29.8
9,889
ResNet152
51.5
17,461
30.3
10,484
Con--Nett-T
52.0
17,936
31.2
10,236
E.5 APPLICABILITY TO DIFFERENT EDGE MODELS.
In Table 14, we report the results on ImageNet-C (Gaussian noise, severity level 5) with different light- models that can deployed on the resources-limited edge, including MobileNetV2. MobileNetV3 and ShuffleNetV2. On all these edge models, our CEMA outperforms ETA in terms of both the adaptation accuracy and the communication overhead. For example, our CEMA achieves higher accuracy (37.1% 31.1%) than ETA with ShuffleNetV2 the edge model with much fewer number of uploaded samples (13,975 vs. 24,558). The results show that our CEMA is appli- cable various lightweight edge models.
28

---

Published as a conference paper at ICLR 2024
Comparisons of different edge models on ImageNet-C (Gaussian noise, severity level 5) with ResNet101 as the foundation model. We adopt two popular lightweight edge models, i.e., MobilcNetV2, MobileNetV3 and ShuffleNetV2, verify our proposed method CEMA.
Table 14:
Model
Acc. (%)
#Upload
MobileNetV2 (baseline)
19.4
ETA (Niu et al., 2022)
45.6
25,133 (50%)
CEMA (Ours)
47.3
16,117 (32%)
MobileNetV3 (baseline)
25.3
ETA (Niu et al., 2022)
40.1
19,094 (38%)
CEMA (Ours)
41.2
15,414 (31 %)
ShuffleNetV2 (baseline)
11.2
ETA (Niu et al., 2022)
31.3
24,668 (49%)
CEMA (Ours)
37.1
13,975 (28 ))
E.6 EFFECTIVENESS OF THE ENTROPY-BASER CRITERIA
To verify the effectiveness of the proposed entropy- based sample filtration strategy, we compare our CEMA with two variants on ImageNet-C (Gaussian noise, severity level 5), namely uploading all test samples and randomly uplouding an equal number of test samples. The variant randomly uploading an equal number of lest sampley denotes randomly uploading only partial test samples to the cloud, in which the number of uploading samples the same as our CEMA. From Table 15, our equal number of test samples (29.8% vs. 28.0%). In
CEMA outperforms randomly uploading addition, though the variant uploading all test samples achieves higher performance since it provides sufficient test samples for distillation. It requires many more test samples to be uploaded from the edge to the cloud (50,000 vs. 9,889). The results demonstrate the effectiveness of our devised entropy- based filtering criteria.
Table 15: Effectiveness of the proposed entropy-based criteria on ImageNet-C (Gaussian noise, severity level 5) with ResNet18 as the edge model.
Strategy
Accuracy (%)
#Uploading Samples
Uploading all test samples
30.6
50,000
Randomly uploading an equal numher of test samples
28.0
10,153
Uploading with entropy-based based criteria (our CEMA)
29.8
9,889
E.7 EFFECT OF THE REPLAY BUFFER SIZE
To investigate the effect of the size of the replay buffer, we conduct an ablation with the different sizes selected from {0. 1000, 2000, 3000, 5000, 10000, oo). Note that the size oc denotes the buffer is unlimited. From Table 16, our CEMA achieves better accuracy on ImageNet-C when we increase the replay buffer size. We obtain the best performance at 51.1% when the buffer size 10,000 (10 samples for each class on average). Employing a replay buffer of unlimited size does not yield any improvement in adaptation accuracy, which remains at 51.1% %. However, this leads to a significant increase in storage usage, escalating from 5.6 GB to 9.8 GB. The reason is that we can sample more diverse samples for the larger replay buffer. This may provide more distribution information for adaptation. Note that replay buffer with size of 10,000 with image resolution x 224 requires around 5.6 GB memory. This can be affordable for the typical cloud GPU servers. Thus we set the buffer size to 10 10,000 in all the experiments.
29

---

Published as a conference paper at ICLR 2024
Table 16: Effect of the size of the replay buffer B ImageNer-C (Gaussian noise, severity level 3)
with ResNet18 as the edge model.
Size
1000
2000
3000
5000
10000
Accuracy (%)
47.7
50.0
50.3
50.5
50.9
51.1
51.1
Storage (GB)
0.6
1.1
1.7
2.8
5.6
9.8
E.8 EFFECT DIFFERENT UPDATING WAYS
To investigate the effect of different ways to update parameters, we conduct more experiments on ImageNet-C (Gaussian noise, severity level 3). We consider two ways to update models: 1) updating all the parameters and 2) only updating parameters in batchnorm (BN) layers. From Table 17, only updating BN on the foundation model and the edge model achieves the best adaptation perfor- mance and the lowest communication overhead (especially distributed parameter size). Updating all the layers may disrupt the previously learned knowledge and lead to inferior adaptation perfor- mance. Furthermore, it would require the distribution of more parameters and result in significant communication overhead. Thus, we choose to only update BN layers for both the foundation model and the edge model.
Table 17: Ablations the effects of different ways to update parameters ImageNet-C (Gaussian noise, severity level 3). fe and fw denote the foundation model and the edge model, respectively. "Distributed Param. Size" denotes how many model parameters should be transferred between the cloud and the edge.
Only updating BN for fo
Only updating BN for fw #Upload
Distributed Param. Size (MB)
Acc. (%)
14,221
11.68
31.9
16,871
0.0096
46.4
17,608
11.68
43.6
17,479
0.0096
51.1
E.9 EFFECT OF UPDATING INTERVAL IN EDGE
In scenarios where communication and computation resources are limited, the edge devices may only download and update the edge model once while cloud performs every K times adaptation (K>1). To investigate the effect of K, perform more experiments noise, severity level 3) with different from 1 to From Table 18, the adaptation performance slightly drops when K grows. For example, when K increases from 1 to 3, the adaptation accuracy only drops from 51.1% to 50.8%. Even when becomes 5, the adaptation accuracy is still 50.5%. These demonstrate the effectiveness of our CEMA in scenarios with limited bandwidth.
ImageNet-C (Gaussian
Table 18: Performance comparisons on ImageNet-C (Gaussian noise, severity level 3) with different updating intervals K on the edge side.
K Accuracy #Uploading samples
2
3
4
5
51.1
$1.0
50.8
50.7
50.5
17,479
17,492
17,333
17,396
17,402
E.10 COMPARISONS WITH MORE SAMPLE IDENTIFICATION STRATEGIES
To further demonstrate the effectiveness of our sample identification strategy, add more experi- ments on ImageNet-C to compare our CEMA with more strategies, namely SENTRY (Prabhu al., 2021) and BALD (Houlsby al., 2011). SENTRY measures the sample information via the predic- tion consistency regarding different data augmentations. Besides, BALD achieves this by calculating the entropy differences between the current sample and previous samples.
30

---

Published as a conference paper at ICLR 2024
From Table 19, SENTRY achieves much worse accuracy than CEMA (45.6% vs. 51.1%). The results show that the prediction consistency regarding different data augmentations is hard identify the helpful samples in entropy minimization. Besides, BALD yields adaptation accuracy of 50.7%, which is still worse than our CEMA. The results demonstrate the effectiveness of our CEMA in filtering out low-informative samples.
Table 19: Comparisons different strategies to identify unreliable and low-informative samples on ImageNet-C (Gaussian noise) with ResNet18 as the edge model.
Strategy
Accuracy (%)
#Uploads
SENTRY (Prabhu et al., 2021)
17.299
BALD (Houlsby al., 2011)
50.7
17,319
CEMA (Ours)
51.1
17,479
E.11 MORE COMPARISONS ON OBJECT DETECTION
We conduct more experiments by applying our CEMA on corrupted COCO 2017 (Lin et al., 2014) with YOLOv5 (Redmon al., 2016) model. We generate this corrupted version (Gaussian noise, severity level 3) of COCO 2017 dataset following ImageNet-C (Hendrycks & Dietterich, 2019). We use YOLOv5-nano as the edge model and YOLOv5-large as the foundation model. We set the learning rate to 0.0001 and the other hyperparameters are the same as those in the image classi- fication task. From Table 20, our CEMA outperforms the baseline (18.3 vs. 11.6 mAP) and ETA (18.3 15.0 mAP). Moreover, this improved performance was achieved with smaller number of samples uploaded for adaptation. The results demonstrate the applicability and effectiveness of our CEMA on the object detection task.
Table 20: Comparisons on corrupted COCO 2017 (Gaussian noise, severity level 3).
Model
mAP
#Upload
YOLOv5-nes (baseline)
11.6
ETA (Niu et al., 2022)
15.0
3,875 (78%)
CEMA (Ours)
18.3
3,261 (65%)
31


Reviews:
[{"review_id": "HwltmqWiA9", "reviewer": "Reviewer_xxok", "summary": "This paper introduces a novel learning paradigm aimed at enhancing the adaptability of cloud-edge models to address challenges posed by out-of-distribution test samples in real-world scenarios. The proposed approach is both practical and holds substantial significance. Specifically, to reduce communication overhead, the authors have incorporated a dynamic sample filtering strategy, allowing for the identification and exclusion of unreliable and low-informative samples. Furthermore, to further augment the edge model's capabilities and fully capitalize on the abundant cloud resources, the authors have integrated a substantial foundational model to serve as a guiding teacher for the edge model. Extensive experimental results on ImageNet-C and ImageNet-R datasets serves to underscore the efficacy of the presented method.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tThe proposed collaborative cloud-edge model adaptation (CEMA) paradigm addresses a highly practical problem in the realm of cloud-edge model deployment, emphasizing the challenges of distribution shifts and the limited resources of the edge devices.\n2.\tThe proposed CEMA paradigm is a pioneering achievement in the field. It effectively divides adaptation tasks and distributes them between the cloud and edge devices, resulting in optimized resource utilization and the assurance of robust performance. \n3.\tThe experimental results demonstrate that the proposed method not only achieves the highest level of out-of-distribution performance but also reduces communication costs by an impressive 60% when compared to SOTAs on the ImageNet-C and ImageNet-R benchmarks.\n4.\tThe paper is well-written and easy to follow. Furthermore, it is accompanied by illustrative figures that enhance its overall readability.", "weaknesses": "1.\tIn Section Identification on low-informative samples, the author claims that ‘We emphasize that uploading samples does not block the edge from inferring on next incoming samples. In other words, the processes of inference and uploading can be executed simultaneously.’. How do the authors decide which test samples use which updated model to make a prediction? More explanations are required. \n2.\tIn Equation (6), the foundation model assigns pseudo labels to the uploaded samples. Simultaneously, the authors employ entropy to update the model, introducing another pseudo label (the maximum value). Is there a potential conflict between these approaches? What consequences might arise if the entropy loss is eliminated?\n3.\tIn Algorithm 1, line 3, ‘Calculate S(X) via Eqn. (x)’, It is confused what is Eqn.(x).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BYSYGO3wmE", "reviewer": "Reviewer_iQ7Q", "summary": "The paper introduces a Cloud-Edge Model Adaptation (CEMA) paradigm for dynamic model adaptation. This approach delegates adaptation workloads to the cloud, thereby reducing the burden on edge devices. To minimize communication overhead, CEMA excludes unreliable high-entropy and low-informative low-entropy samples from uploading to the cloud. The model leverages knowledge distillation from the foundation model to guide the edge model, and a replay buffer is employed to enhance data utilization efficiency. Experimental results demonstrate a 60% reduction in communication costs compared to state-of-the-art methods on ImageNet-C.", "questions": "Q1. The authors employ the entropy of the logits to assess uncertainty and selectively upload test samples, excluding both unreliable and low-informative ones. However, recent research has pointed out that neural networks can exhibit overconfidence. In such cases, can the uncertainty of a sample still be accurately evaluated based on the logits?\n\nQ2. The sample selection process involves dynamically adjusting the threshold and incorporating more samples into training. This idea seems to be similar to self-paced learning. Can the authors elaborate on the relationship between self-paced learning and their sample-selection scheme?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* This paper introduces the Cloud-Edge Model Adaptation (CEMA) paradigm, which addresses the dynamic model adaptation problem in a novel way.\n\n* The paper is of high quality, the language is clear, the structure is clean, the related work review in the appendix is adequate (including a valuable comparative analysis with various methods), and the figures are clear and easy to follow.\n\n* The manuscript is very clear in the explanations and the methodology. \n\n* As a practical method for model adaptation, I think this paper is of great chance to benefit the community, especially in real-world scenarios.", "weaknesses": "The overall framework appears to be somewhat straightforward as it contains multiple steps. The selection scheme used in the paper is relatively simple, as mentioned in Q1.  Additionally, the selection scheme is designed to exclude data that is either entirely out-of-distribution or absolutely in-distribution. There could be alternative methods to identify these two types of data beyond logits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o3UzfJXHXk", "reviewer": "Reviewer_5VG6", "summary": "The authors present a Cloud-Edge Model Adaptation (CEMA) paradigm that executes dynamic model adaptation, which puts all adaptation workloads to the cloud and only requires vanilla inference in edges. A replay-based entropy distillation method is also proposed to improve the adaptation performance of the edge model. Extensive experiments show that CEMA achieve SOTA performance with lower communication cost.", "questions": "1. From table 15, it seems that the performance improves as the replay buffer increase. Why not use all the uploaded samples for adaptation?\n\n2. What's the performance if no teacher model is used?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed cloud-edge model adaptation (CEMA) framework seems novel to me.\n\n2. The CEMA paradigm only requires the edges to perform forward computation, which is important considering that backpropagation on edge is difficult.\n\n3. The proposed dynamic unreliable and low-informative sample exclusion are simple but effective.\n\n4. Extensive experiments and ablation studies are provided. There are large performance improvements over previous methods.", "weaknesses": "1. The proposed method is only evaluated on classification tasks. Could the proposed method be extended to other tasks such as object detection?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fdNTImeHja", "reviewer": "Reviewer_Q1Z2", "summary": "This paper focuses on adapting deep learning models for edge devices with limited resources in dynamic environments. Traditional approaches involve deploying fixed models, which can result in reduced performance as scenarios change. This paper devises the Cloud-Edge Model Adaptation (CEMA) paradigm, in which the edge models only need to perform forward propagation and the edge models can be adapted online, by performing a data filtering strategy to allow high-quality data to be uploaded to the cloud and a replay-based entropy distillation.", "questions": "1. In equation 6, is fθ(x) and the pseudo labels yˆ  the same thing or different?\n2. To use KL divergence loss and CE loss together is interesting. I wonder how the hyperparameter alpha and beta change in different dynamic scenarios.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written, easy to follow.\n2. Manage what data to train is a sound approach to reduce training efficiency.\n3. Experiments are comprehensive.", "weaknesses": "From algorithmic perspective (low-informative data identification), the novelty is limited. There should be existing papers studied how to filter out low-informative data. These can be added into the paper related works and experiments to compare.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)