Source-Target Unified Knowledge Distillation for Memory-Efficient Federated Domain Adaptation on Edge Devices
Authors: Xiaochen Zhou
Yuchuan Tian
Xudong Wang
Venue: ICLR 2022 Submitted (2022)

Abstract: To support local inference on an edge device, it is necessary to deploy a compact machine learning model on such a device.
When such a compact model is applied to a new environment, its inference accuracy can be degraded if the target data from the new environment have a different distribution from the source data that are used for model training.
To ensure high inference accuracy in the new environment, it is indispensable to adapt the compact model to the target data.
However, to protect users' privacy, the target data cannot be sent to a centralized server for joint training with the source data. Furthermore, utilizing the target data to directly train the compact model cannot achieve sufficient inference accuracy due to its low model capacity.
To this end, a scheme called source-target unified knowledge distillation (STU-KD) is developed in this paper. It starts with a large pretrained model loaded onto the edge device, and then the knowledge of the large model is transferred to the compact model via knowledge distillation.
Since training the large model leads to large memory consumption, a domain adaptation method called lite-residual hypothesis transfer is designed to achieve memory-efficient adaptation to the target data on the edge device. Moreover, to prevent the compact model from forgetting the knowledge of the source data during knowledge distillation, a collaborative knowledge distillation (Co-KD) method is developed to unify the source data on the server and the target data on the edge device to train the compact model. STU-KD can be easily integrated with secure aggregation so that the server cannot obtain the true model parameters of the compact model. Extensive experiments conducted upon several objective recognition tasks show that STU-KD can improve the inference accuracy by up to $14.7\%$, as compared to the state-of-the-art schemes. Results also reveal that the inference accuracy of the compact model is not impacted by incorporating secure aggregation into STU-KD.

Under review as a conference paper at ICLR 2022
SOURCE-TARGET UNIFIED KNOWLEDGE DISTILLA- TION FOR MEMORY-EFHICIENT FEDERATED DOMAIN A DA PTATION ON EDGE DEVICES
A nonymous authors
Paper under double- houble-blind review
ABSTRACT
To conduct local inference on edge devices, it necessary to deploy compact ma- chine learning models on such devices. W hen such a compact model is applied to new environment, its inference accuracy can be degraded if the target data from the new environment have a different distribution from the source data used for model training. To ensure high inference accuracy in the new environment, it is indispensable to adapt the compact model to the target data. However, to pro- tect users' privacy, the target data cannot be sent to a centralized server for joint training with the source data. Furthermore, utilizing the target data to directly train the compact model cannot achieve sufficient inference accuracy due to its low model capacity. To this end, scheme called source- target unified knowledge distillation (STU-KD) is developed in this paper. It first adapts a large source model to the target data on the edge device, and a large target model is obtained. The knowledge of the large larget model is then transferred to the compact model via knowledge distillation. Since training the large model leads to large memory consumption, a domain adaptation method called lite residual hypothesis transfer is designed to achieve memory-efficies adaptation to the target data on the edge device. Moreover, to prevent the compact model from forgetting the knowledge of the source data during knowledge distillation, collaborative knowledge dis- tillation (Co-K) KD) method is developed to unify the source data on the server and the target data on the edge device to train the compact model. STU-KD can be easily integrated with secure aggregation so that the server cannot obtain the true model parameters of the compact model. Extensive experiments conducted upon several tasks of object recogmition show that STU-KD can improve the inference accuracy by up to 14.7%, as compared to the state- of- the art schemes. The results also reveal that the inference accuracy of the compact model is not impacted by incorporating secure aggregation into STU-KD.
INTRODUCTION
Many computer vision (CV) applications, such as mobile robots, require local inference on edge devices because of the requirements on data privacy and low latency. To enable local inference on edge devices, it necessary to deploy compact machine learning models on such devices. For exam- Jerriceeettonn.. ple, considering an edge computing device_ Jetson Nano with 472 GLLOPS GPU and 4 GB memory space which is commonly used for image recognition, a ResNet- 18 model HHe et all 2016) is pre- ferred over a ResNet- 50 model, the inference time of the former case (26 ms) is much smaller than the latter one (64 ms) (Yang et al.. 2020). Such compact machine learning models can be obtained by manual design based experts' experience (Sandler et al.) 2018 & Howard et al. 2019 Lin et al], 2020) or by some automated machine learning (A utL techniques, e.g., network com pressionon Ning et al. 2020 Li et al., 2020b and neural architecture search (He et al. 2018 Liu et al., 2019).
When an edge device with a compact model works in a new environment, the unlabeled target da- ta collected from the new environment can have a different distribution from the labeled source data that are used to train the compact model, i.e., domain shift (Gretton et al. 2009) occurs. Con-

---

Under review as a conference paper at ICLR 2022
Step 1: dommin adupiation for the large source model
Step 2: souree-target unifled training for the compact model
Lite residual hypothesis transfor
Knowledge distillation
Supervised leamning
Callaborativee knowlodge distillation
Target data
Targel cataa
Source data
Edge device
Fdgge device
Centralized server
Large source model
Large larget model
Compact model
Data
Model parameter inpur
input
Figure 1: Illustration of source- target unified knowledge distillation.
sequently, the compact model suffers low inference accuracy on the target data. To ensure high inference accuracy in the new environment, is necessary to adapt the compact model to the target
data. A typical approach is to send the target data to cloud server where the model is trained via
unsupervised domain adaptation (UDA) ) (e.g., the methods in Kang et all 2019 Tanq & Tia (022); Xu et al] (2020) and then deployed back to the device. However, it leads to loss of data privacy.
To avoid this issue, another type of approach is to train the compact model locally over the target data via UDA methods. It doable on an edge device. For example, training ResNet- 18 with batch size 48 consumes nearly 172 GFLOPs and 1.1 GB memory space per batch, which is affordable for Jetson Nano. However, this type of approach cannot achieve high inference accuracy on the target data due to limited model capacity of the compact model.
To this end, a scheme called source- target unified knowledge distillation (STU-KD) is developed in this paper: The key idea is to utilize a large model with sufficient model capacity to learn fine- theigra representations of the target data, and then transfer its knowledge to the compact model. As shown in step 1 in Figure [1 large source model is loaded on the edge device and is then adapted to the target data to obtain a large target model. The challenges of this step are -wo-fold. First, the target data are unlabeled, so the aine-graind representations cannot be leamed via supervised leaming, As a result, unsupervised domain adaptation (UDA) is needed in step 1. Second, the edge device does not have the source data, since the volume of the source data can be too large to be stored the edge device, or the source data cannot be ex posed to the edge device due to confidentiality of these data. Without the source data, many UDA methods (K ang et all, 2019 Tang & Jia 2020; Xu et all 2020) are not applicable for the adaptation of the large source model. Thus, source- souree-free UDA methods (Li et all, 2020aa Liang et all, 2020 Liu et all (021) must be employed. However, existing source- free UDA methods e.g., source hypothesis transfer (SHOT) (Liang et all, (020), require retraining of the large source model, which leads to large memory consumption on the edge device. For ex- ample, if ResNet- 50 is retrained with batch size 48 on Jetson Nano, the computational cost is nearly 364 GFLOPs, which is affordable for Jetson Nano. However, the memory consumption per batch is nearly 5 GB, exceeding the 4 GB memory space limit of Jetson Nano. To tackle the challenges in step 1 of STU-KD, a eeeoory-eficcien UDA method called lite residual hypothesis transfer (L- RHT) is designed by enhancing the architecture of a source-free UDA method such as SHOT. More specifically, the same loss function as that of SHOT is adopted, so unsupervised leaming can be conducted with unlabeled target data. However, the feature extractor in SHOT must be replaced with a new architecture that can be trained in a -effore-effiiientt manner: Thus, the new architecture is designed by adding lite residual (LR) modules (Cai et all 2020) to the feature extractor such that its outputs can be feature extractor fixed. As a result, in LRHT the training process of the large source model involves neither the feature extractor nor the classifier. Since training the LR modules generates a much s- maller volume of activations and demands much lower memory footprint, the training process of the large target model is highly memory-effiche efficient. In other words, by using LRHT the large source model can be adapted to the target data to obtain a large target model in a memory- -effcireeefiient manner.
by training the LR modules only while keeping the parameters of the
In step 2 of STU-KD (in Figure I, the large target model is utilized to generate soft labels for the target data. The knowledge of the target data is then transferred to the compact model by train- ing it over the taryet data and the soft labels via knowledge distillation (KD) (Hinton et al.) 2015). However, no source data are involved in the KD process, leading to the compact model gradually forgetting the the knowledge of the source data, i.e., catastrophic forgetting (McCloskey & Cohen
2

---

Under review as a conference paper at ICLR 2022
occurs. As a result, when the edge device encounters data from the source domain after the KD process, the compact model can suffer low inference accuracy. For example, consider a mobile robot whose compact model is trained via the above KD process. When it moves to an environment similar to that of the source domain, its compact model cannot recognize the objects in that envi- romment with high accuracy. To avoid catastrophic forgetting, it is necessary to train the compact model considering both the target data and the source data. Thus, a collaborative knowledge dis- tillation (Co- (D) method is designed as follows. On the edge device, a compact model is trained over the target data and the soft labels via KD. On the server, another compact model is trained over the source data via supervised learning. The challenge here is how to consolidate these two compact models into a global compact model. The setting of this challenging problem is similar to that of federated learning (FL) (McMahan et all 2017). However, there exists one major difference. FL requires that different nodes have the same type of loss functions, while Co-KD has to use the cross- cesss-enropy loss on the server and the loss function of KD (i.e., Kullback- Leibler (KL) divergence in the state- of- the- art methods) on the edge device. Due to such a difference, existing FL algorithms [MMM (c.g,,MMMM ahan et all (007)) Karimireddy et al. (2020)) are not effective for consolidating the com- pact models in Co- D. To this end, an alternating direction method of multipliers (A AMM based leaming algorithm is designed for Co- CD-KD to consolidate the compact model on the edge device and that on the server, as shown in step 2 in Figure 11 More specifically, the training process of the global compact model is formulated into a consensus problem over the edge device and the server. The consensus problem is then divided into two subproblems by A D DMM The edge device trains its compact model by iteratively solving the subproblem related to the target data, and the server trains its compact model by iteratively solving the subproblem related the source data.
1989)
In the second step of STU-KD, the edge device needs to upload the parameters of the compact model to the server. It is possible for the server to recover some information of the target data from these parameters (Zhu et al., 2019). To ensure privacy of the target data, secure aggregation (Bonawitz et all, (017) is adopted in STU-KD so that the server cannot obtain the true parameters of the compact model.
Extensive experiments are conducted to evaluate the performance of STU-KD. The results show that STU-KD increases the compact model's inference accuracy on the target data by up to 14.7% while maintaining high inference accuracy on the source data, as compared to the state-of-art methods. Moreover after employing secure aggregation, the performance of STU-KD is not affected
2 RELATED WORK
Cross Domain Model C Compression. Many schemes (Yu et a [2019; 9. Chen et all, 2009 Feng et all 2020; Dillard et al., 2020; Nquven- et all, 2020) combine network compression with unsu- pervised domain adaptation (UDA) ) to obtain compact model that performs well on the target data. These schemes adopt traditional UDA methods that assume the target data and the source data are stored in one place, which are not applicable when the target data and the source data are isolated. Moreover, these schemes need to train a large source model via UDA. The large memory footprint of training the large source model hinders the application of these schemes on edge devices.
On-device A daptation. Recently, some schemes are developed to adapt an on- on-device model to the target data. TinyTL (Cai et all 2020) assumes that the on- on-device model has sufficient model capacity, and adapts this model to the labeled target data with low memory footprint. However, TinyTL is not applicable to the scenario with unlabeled target data. MobileDA (Yang et al. 2020 directly trains a compact model via UDA method called deep CORAL (Sun & Saenko, 2016). It employs a teacher model to boost the inference accuracy of the compact model. However, due to its low model capacity, directly training the compact model cannot achieves sufficient inference accuracy on the target data. A Althughh& a teacher model is employed, is only trained over the source data, so the teacher model can only achieve limited improvement of the inference accuracy.
Domain daptation Under Data Isolation. Some schemes are designed to conduct UDA in a scenario where the source data and the target data are held by different data owners. In federated domain adaptation (FDA), ), e.g., federated adversarial domain adaptation (Peng et all 2020), the data owners can collaboratively train globally shared model. In the training process, each data owner only shares its local updates rather than the target or source data. Schemes like source- free domain adaptation (SFDA) can accomplish UDA in the absence of the source data. In Li et all (2020a), the
3

---

Under review as a conference paper at ICLR 2022
LR
LR
Trainable
parameters
Building Block
Buikding Block i+1
Loss Js
Frozen
parameters
Target data
Feature extractor with LR modules
Classi fier
Figure 2: The pipeline of lite residual hypothesis transfer.
source model is trained over some generated Sarget-style samples. In Liang et all (2020), the source model is trained through information maximization loss over the target data and self- supervised aseudo-labeling. The SFDA schemes in (Kumi et al. 2021; Liu et al. 2021 leverage generative models to generate fake source samples for UDA. However, to obtain a model with sufficient ac- curacy, both FDA and SFDA schemes require the source model to have sufficient model capacity. However, such a requirement cannot be satisfied by a compact model with low model capacity.
METHODOLOGY
One centralized server and M edge devices (M 2 1) are considered in this paper. The labeled source data in the server are denoted as (m., where w, represents the feature vector of source sample and ys the ground-trun truth label of the source sample. The source data are sampled from the souice domain denoted as Ps. The unlabeled target data on the edge devices are denoted {æ,} where It represents the feature vector of a target sample. The target data are sampled from the target domain denoted as Pt. Each edge device needs to accomplish a classification task with K classes using a compact model. The compact model on the edge device E {1, .... M } is defined as function fm. and its model parameters are denoted as wm- Given the above setup, source- target unified knowledge distillation (STU-KD) is developed to adapt the compact model to the target data.
3.1 LITE RESIDUAL HY POTHESIS TRANSFER
To boost the inference accuracy of the compact model, a large source model with sufficient model capacity, defined as function fr, is adopted in STU-KD, It can leam more tneeggrined represen- tations of the target data compared with the compact model. Before being deployed to the M edge devices, the large source model fr is pretrained over the source data by minimizing an objective function Jr (wr)
((.....)) [fr wx) v.)),, where wr denotes the parameters of the large source model, and L(-,)) -) is the standard cross- entropy loss. On edge device large source model is then adapted to the target data. The challenges of the adaptation are two- fold. First, the target data are unlabeled so that fine- grained representations of the target data cannot be learned via supervised learning. As a result, unsupervised domain adaptation (ÚDA) is need- ed. Second, edge device m does not have the source data, many UDA methods (Kang et all 2019; Tang & lia, 2020 Xu et al. 2020) are not feasible for the adaptation. Thus, source- free UDA methods (Li et aLl 2020a Liang et all 2020 Liu et all (021h must be emploved. However, exist- ing source- free UDA methods, e.g., source hypothesis transfer (SHOT) (Liang al. (020), require retraining of the large source model, w which leads to large memory consumption on the edge device.
{,.... M) }. the
To tackle the above challenges of the adaptation, a memory- cefforr-efficient UDA method called lite residual hypothesis transfer (LRHT) is designed by enhancing the architecture of source- free UDA method such as SHOT. LRHT adopts the same loss function as that of SHOT (denoted as Js in Figure 2 so unsupervised leaming can be conducted over the unlabeled target data. The details of the loss function of SHOT are provided in A ppendix A However, the feature extractor in SHOT must be replaced with a new architecture that can be trained in memory- effficien manner. As shown in Figure 2 the new architecture is designed by adding lite residual (LR) modules (Cai et all 2020) to the feature extractor such that its outputs can be fine-tuned by the LR modules. The LR modules tine-tunetin the outputs of the feature ex extractor by changing the intermediate activations of the feature extractor. Define the i-th building block of the feature extractor as a function di and the LR module added to the it as function r,, Given the activations a;-1 of the 1) 1)-th building block, LR

---

Under review as a conference paper at ICLR 2022
module r; changes the activations of the i-th building block as a, 9e (a;-))) + r((aa-1), where gi denotes the original activations of the i-th building block, and r; (a,-1) denotes the residual activations of LR module rt. During the adaptation, the parameters of the feature ex extractor and that of the classifier are fixed, and only the LR modules are trained to fine- tune the outputs of the feature extractor. Compared to the case of directly training the feature extractor (like SHOT), training the LR modules generates a much smaller volume of activations and demands much lower memory footprint. Thus, by using LRHT the large source model can be adapted to the target data to obtain large target model in a eeemory-effiiieee manner.
3.2
COLLABORATIVE K KNOWLEDGE DISTILLATION
The next step is to transfer the knowledge of the large target model to the compact model.
3.2.1
CASE I: SINGLE EDGE DEVICE
Consider the case of a single device that contain compact model fi with the model parameters wi-- On the edge device, the large target model is leveraged to generate the soft labels of the target data. The knowledge of the target data is then transferred to the compact model via knowledge distillation (KD) (Hinton et all 2015). More specifically, the compact model is trained over the target data and the soft labels via minimizing the loss function of KD. However, no source data involved in the KD process, which leads to the compact model gradually foryetting the knowledge of the source data. That is, catastrophic forgetting (McCloskey & Cohen, 1989) occurs.
To prevent catastrophic forgetting, the target data and the source data need to be unified to train the compact model in above process of knowledge transfer. Thus, a collaborative knowledge distillation method (Co-K (O-KDD is designed as follows. On the edge device, compact model is trained over the target data and the soft labels via KD. The loss function of KD, denoted as JR, is written as
Jx (w)) = Ez, (p(2 7) [1=,) +))],
where z. (Iy; wi) ) is the logit of the compact model, zT = fr(;;; w'y) is the logit of the large target model with parameters w/y (i.e,, the soft label), and p(z, r) denotes the soften probability vector with a temperature of a logit z. On the server, another compact model is trained over the source data via cross-entropy loss denoted as Jc. The expression of Jc is
Jc(w1)= = Eis.... v..) 1 (x, w1), v.)]
The challenge here is how to consolidate the two compact models into a global compact model. The setting of this challenging problem is similar to that of federated learning (FL) (McMahan et all, 2017). However, there exists one major difference. FL requires that different nodes have the same type of loss functions, while Co-KD has to use the cross- entropy loss on the server and the loss func- tion of KD (i.e., Kullback- Leibler (KL) divergence in the state- of- the art methods) on the edge de- vice. Due to the difference, existing FL algorithms (e.g., McMMaan et all (201)); 7); Karimireddy et al. (2020): He et al. (2020a)) are not effective for consolidating the compact models in Co-KD.
To this end, an alternating direction method of multipliers (A (ADMM) (Boyd et all 2011) based leam- ing algorithm is developed for Co-KD to consolidate the compact models. The training process of the global compact model formulated into a consensus problem Ω over the edge device and the server:
min (w)) + (1 (w)) w0,
(1)
s.t. wT wo.
where a is balancing hyperparameter, and un denotes the parameters of the compact model trained on the server. For problem Ω that has a consensus constraint, A DDM is utilized to divide problem Ω into two subproblems Qo and Q i.e.,
Ro min Lo wi, A1) = (1 - a)Jo(w)) + (Ai, wi wo) - Gl(w - woll and wep
w
Ω min L, (Wo, wT A1) = awjx(w)) + (Ai, w wp) + fll woll2. wi
where A denotes Lagrange vector, and p is a penalty hyperparameter. Subproblem 24 and sub- problem 2 are then iteratively solved on the server and on the edge device, respectively. More
5

---

Under review as a conference paper at ICLR 2022
specifically, in the t-th iteration of Co-KD, the t- iterate of w1, denoted as wn (t), is determined on the edge device by w1 (1(t) := arg min.., (w (t- - 1), w1, A (t - 1)), where wa 1) denotes the (t - 1)-th iteration of wo. and \ 1) denotes the (t 1)-th iteration of X A Afterwards, the edge device uploads its Lagrange vector (1 - 1) and w) (t) to the server, and then the serv- er determines (t)()) = arg min Lo( wg; w/ (t), A (t. 1))- A Afterwartss, the server sends uv to the edge device, and the edge device determines A (t) := - 1) + p( w/ (t) - wo (t)). S- ince both (t) and w (t) do not have closed-fomm solutions, stochastic gradient descent (SGD) is utilized to numerically determine (t(() and wi (t). As convergence criterion is reached, e.g., (wwo() wo 1) S ca where cy is predefined threshold, Co-KD stops. (t) is regarded as the parameters of the global compact model that is then deployed to the edge device.
Since a solution obtained by ADMM satisfies Karush-Kuh--iiee converge to local optimum of Problem Ω. Moreover, the compact model is pretrained over the source data. can avoid bad local optimum of poor generalization during training of wo, because pretraining over the source data can be regarded as regularization on the parameter wo, as studied in (Erhan et all 2010). As result, the global model obtained by Co-KD can always converge to a local optimum with sufficient generalization performance, i.e., it can achieve high inference accuracy over both the target data and the source data.
(KKT) conditions, (t) can
3.2.2 CASE II: MULTIPLE EDGE DEVICES
When a single edge device only has limited number of target data, it is hard to learn the fine- grained representations of the target data even by the large model. As a result, the compact model trained by Co-KD still suffers low inference accuracy on the target data. To tackle this challenging, Co-KD can be extended to unify the target data of the multiple edge devices for training global compact model. Due to page limit, the details of Co-KD for multiple edge devices are provided in A Appendix B and only the main idea is provided below.
The training process of the global compact model is first. formulated into a consensus problem S2' over the M edge device and the server. Problem S2' is then divided into (M + 1) subproblems by following A DMM, and each edge device as well as the server iteratively solves one subproblem. In the t-th iteration, edge device m determines Wm.. (t) and then sends (t) and (m 1) to the server. A Afterwards, the server determines wp(() and then broadcasts (t) to all the edge devices.
Edge device m then computes X, (t). As the convergence criterion is reached, the compact model with parameters (t) is broadcast to all the edge devices as the new compact model.
3.3 DISCUSSION ON PRIVACY PRESERVATION
In STU-KD, the edge device does not upload the target data to the centralized server, providing a basic level of privacy preservation. However, recent studies (Melis et all, 2019 Zhu et all 2019 Zhao et al., (020) show that is possible for the server to recover some information of the target data from the uploaded parameters. This indicates that STU-KD is vulnerable to such recovery attacks.
One advantage of STU-KD is that it can be easily integrated with secure aggregation (Bonawitz et all, 2017) to resist the recovery attacks. In the case of multiple devices, the server only requires > w',,() to update wo(t - 1) (see equation 5 in A ppendix B,, where w', (t) denotes the parameters uploaded by edge device m. To prevent the server from obtaining w/n (t) without affecting the update process of wy(t 1), the MM edge devices can add noise to their pa- rameters before uploading, based on the following rule. For any two edge devices i and j (i < j) from the M edge devices, edge device i adds a random vector nu.; to w/ (t), and edge device j sub- tracts ni.j from w (t) (how to generate ni.j can refer to A Apeedix C.) By adding and subtracting these random vectors, the disturbed parameters im. (t) = w'n, (t) + Em mjjj Mmm.j Σiem ni, are obtained on edge device m. vn (t) is then uploaded to the server. The server cannot recover w/,, (t)
from w. (t), but it can compute I.M= w'n, (t) = EM= (t), and then updates wy(t 1).
In the case of a single edge device, the edge device can be regarded as two virtual devices by dividing the target data into two parts. Each virtual device uses one part to train a compact model. A Aterwards, the above secure aggregation mechanism for multiple devices can be applied directly.

---

Under review as a conference paper at ICLR 2022
Table 1: Inference accuracy (%) on Office-31 dataset (target data and source data)
Method
A
A-W
D-*A
D-W
W A
W W>
Avg.
ResNet- 18 (He et al, 2016)
70.7 = 0.0
66.4 0.0
40.4 0.0
84.0 + 0.0
46.4 0.0
91.6 0.0
66.6
SHOT (Liang et al, 2020)
81.9 + 2.5
82.1 0.5
64.5 1 0.8
92.7 + 1.0
63.9 0.8
95.5 + 0.5
80.1
TO-KD
942 - 0.3
88.9 0.0
73.6
97.2
74.1
0.1
87.8
STU-KD
94.0 1: 0.2
88.8
0.1
0.3 72.3
97.5 0.1
74.6 0.1
19.1
87.7
ResNet- 18 (He et al, 2016)
85.1 0.0
85.1 0.0
0.0
9.00
9..
97.5 0.0
92.2
SHOT (Liang et al, 2020)
74.1 +.11::0.9 0.9
73.4
2.0
74.7 1.1
94.7+
70.4
90.4 0.7
79.6
TO-KD
6.6
76.7
98.0
72.9 2.4
93.3 1.3
78.4
STU-KD
81.1 0.8
83.5 0.8
97.3 +
100.0 0.0
94.2 1.8
98.8 + 92.5
Table 2: Inference accuracy (%) on ImageCLEF-DA dataset (target data and source data)
Method
I-+C
I>P
C->1
C-+P
P P=11
P CC
Avg.
ResNet 18 (He at. 2016)
88.8 + 0.0
73.5 0.0
75.7 0.0
6.0
0.0
81.7 0.0
77.1
SHOT (Liang et al., 2020)
95.6 0.6
74.5
0.7
73.3
0.9
0.8
94.7 0.5
85.7
TO-KD
96.3 0.1
0.2
0.2
7..6
92.8
95.3 0.1
88.9
STU-KD
96.1 0.1
+.14
92.4+0.1
79.1 0.1
92.8 + 0.1
95.0 + 0.0
89.0
ResNet: 18 (He et al,, 2016)
96.7
96.7
9.0
76.7 9.00
76.7
89.4
SHOT (Liang et al, 2020)
95.0 0 2.7
0.9
9.07
68.9 96.1
70,0 + 1.5
86.6
TO-KD
89.2 + 1.0
95.4 1.4
9.9
96.7+1.2
69.6 + 0.8
68.5 + 1.5
86.3
STU-KD
95.0 1.0
0.4
72.1 98.1=1111
75.8 = 0.8
89.7
4 EXPERIMENT
4.1
EXPERIMENTAL SETUP
Three public datasets for domain adaptation are used to evaluate STU-KD. Office-31 (Saenko et all
2010) has three domains: A mazon (A), DSLR (D), and Webcam (W). ImageC DF has three domains: ImageNet ILSV 2012 (I), Caltech- 256 (C), and Pascal VOC 2012 (P). Efiic-Homee
(Venkatewwara et all 2017) has four domains: Artistic images (Ar), Clip Art (CI), Product images
(Pr), and Real- World images (Rw).
The following schemes are compared with STU-KD: the compact model without domain adapta- tion, source hypothesis transfer (SHOT) (Liang et al., 2020), and target- only knowledge distillation (TO-KD). Here TO-KD is constructed by replacing collaborative knowledge distillation (Co- (D) in STU-KD with knowledge distillation (Hinton et al., 2015). In addition, four variants of STU- KD are also evaluated. The first three variants are constructed by replacing the ADMM based leaming algorithm in STU-KD with federated average (MMMahan et al. 2017) (denoted as STU- KD-AVG), distributed SGD (Zinkevich et all 2010) (denoted as STU-KD DSDD), and SCA FFOLD The forth variant is formed by in- tegrating STU-KD with secure aggregation (B Bonawwtz et all 2017 (denoted as STU-KD-SA). For SHOT, its compact model is directly trained over the target data. For STU-KD and its variants, large target model is obtained by adapting a large source model to the target data, and the large target model is used to generate soft labels for the target data. The difference is that the compact model of TO-KD is only trained over the taryet data and the soft labels, while the compact models of the rest schemes are trained over the target data, the soft labels, and the source data.
Karimireddy et al. 2020) (denoted as
For each transfer task, 90% of the source data are used for training; the rest 10% of the source data used the test data. The inference accuracy on the source data is determined by testing each scheme on the test data. All the target data are used for training and test, as is done in (Long et all, 2017; 2018 & Liang all 2020). The inference accuracy on the target data is determined by testing each scheme on all the target data. In all the experiments, ResNet- 18 (He et all 2016) is selected as the the compact model. ResNet-50 is selected as the large source model for STU-KD. The compact model and the large source model are trained through backpropagation and mini- batch SGD with momentum 0.9 and weight decay 1e 3 is adopted as the optimizer. The learning rate is set to No 0.001. The batch size on the edge device is set to 8, and the batch size in the server is set to 32. Moreover, for all the experiments, the balancing hyperparameter a is set to 0.8; the penalty heprpaarameters is set to 0.3. STU-KD is implemented based on library called FedML (He et al.
1

---

Under review as a conference paper at ICLR 2022
Table 3: Inference accuracy (%) on Office- dataset (target data and source dataa)
Mathod
Ar-+CI
Ar-pPr Ar-Re Re CI
Cl-Ar CI
Cl-Pr
Cl-Re
-Re Pr-+Ar
Pr-+CI Pr-RRe
Re-> Ar Re-CCI
Re-Pr Avg.
ResNet- 18 (He st.ll, 2016)
31.4
44.7
55.2
39.3
49.3
52.7
38.9
92.7
62.5
56.7
39.0
70.8
47.8
SHOT (Liang al, 2020)
36.0
58.6
64.5
49.2
64.8
63.8
49.0
37.5
70.3
58.6
45.5
76.0
56.1
TO-KD
54.1
78.0
80.1
66.8
77.3
78.6
66.2
52.0
81.4
71.7
58.0
84.1
70.7
STU-KD
53.4
78.3
80.3
67.0
774
78.6
66.4
52.3
82.0
72.1
58.1
84.3
70.8
ResNet-18 18 etaeltll el. 2016)
56.8
56.8
56.8
72.1
72.1
72.1
90.3
90.3
90.3
80.3
80.3
80.3
74.9
SHOT (Llanc 2020)
42.0
46.8
60.5
47.2
49.7
52.1
69.5
61.0
76.4
68.0
56.7
71.0
58.4
TO-KD
15.5
19.1
60.1
17.6
15.5
50.3
60.1
60.2
78.3
67.2
57.0
72.8
57.8
STU-KD
64.5
66.0
69.0
71.8
76.0
74.4
87.7
90.4
91.0
69.6
69.4
72.0
75.2
2020b in PyTorch (Paszke et. al., 2019).
Each experiment is executed for three times, and the mean of the inference accuracy is reported. All the experiments are executed on server with one i9. 10900k CPU, one GeForce RTX 3090 GPU, and 64 GB RAM. More details of the experimental setup are given in Appendix
4.2 EXPERIMENTAL RESULT
The main results of a single edge device are shown in this subsection. Due to page limit, the results of different hyperaarameter settiggs, different network architectures, and the visualization results are provided in A ppendix EE.1 A ppendix E.21 and A ppendix E.3 respectively. The results of multiple edge devices are provided in A ppendix E.44 The results on Office 31 dataset, ImageCLEF-DA dataset, and Office- Home dataset are reported in Table[I] Table and Table respectively. In each table, the upper half records the inference accuracy of each scheme on the all the target data; the bottom half records the inference accuracy of each scheme on test data. The method ResNet- 18 represents the compact model without domain adaptation. Firstly, the inference accuracy on the target data is evaluated. TO-KD and KTU-KD (our schemes) outperform all the existing schemes on all the transfer tasks. Compared with the state-of of- the art scheme SHOT, our schemes improve the inference accuracy on the target data by 7.7%, 3.3%, and 14.7% on Office- 31 dataset, ImageCLEF dataset, and Office- Home dataset, respectively. The gain of our schemes can be explained as follows. The large target model used in STU-KD can leam more fine- ane-grained representations of the target data than the compact model. Thus, the compact model that inherits the knowledge from the large target model can achieve higher inference accuracy on the target data, compared with the compact model that are directly trained using the target data. Since both STU-KD and TO-KD utilize the large target model, their performance is comparable. When considering the inference accuracy on the source data, the performance of SHOT and TO-KD degrade due to catastrophic forgetting, compared with the compact model without adaptation. For STU-KD, its performance is much closer to that of the compact model without adaptation, indicating that STU-KD indeed alleviates or even prevents catastrophic forgetting catastrophic foryetting. Note that in some cases (e.g., D D-A, C-+L, and Ar-CCl,, STU-KD achieves higher inference accuracy on the source data than the compact model without adaptation. The reason can be that by training the compact model over both the target data and the source data, the compact model leams more general representations of the source data, which boosts its inference accuracy on the source data.
The peak memory footprint of each scheme is shown in Figure 3 where SHOT (ResNet-50) repre- sents training the ResNet-50 model on the edge device via SHOT. The memory footprint is deter- mined when the batch size is set to 8. For both TO-KD and STU-KD, their peak memory footprints are caused by training the large model via lite residual hypothesis transfer (LRHT). Compared to the case of directly training the large source model on the edge device, STU-KD reduces the mem- ory footprint by nearly 67%, indicating that LRHT indeed saves the memory space and enables the adaptation of the large model on the edge device.
Three variants of STU-KD (STU-KD-AVG, STU-KD-SGD, and STU-KD-SCAFFOLD) are evalu- ated on Office- 31 dataset, and the results are shown in Table Ithough STU-KD-AVG is compa- rable to STU-KD in terms of inference accuracy on the target data, it suffers poor inference accuracy on the source data. As shown in Figure 4 STU-KD-AVG only converges to a point with low infer- employs an advanced FL algorithm (i.e.,
ence accuracy on the source data. STU--DD-CCAFIRR SCAFFOLD (K Karimireddy et all, 2020),, so it achieves higherinferent inference accuracy on the source data compared to STU-KD- AVG. However, such inference accuracy is still lower than that of STU SDU-D
8

---

Under review as a conference paper at ICLR 2022
Table 4: Inference accuracy (%) on Office-31 dataset for the variants of STU-KD (target data and source data)
Method
A-+D
A-W
D->
D-#W
A
W-+
Avg.
STU-KD-AVG
93.6 + 0.5
89.1 0.2
97.3 0.1
74.2 = 0.3
98.7 0.3
87.7
STU-KD-SOD SDD
9..
88.9 0.1
67.6
9.2
73.8 0.3
0.1
86.8
HUU-KD-SCAFFOLE
89.1 9.6
0.1
9.3
74.3 0.1
98.5
87.8
STU-KD
9.2
0.1
0.3
97.5
74.6=0.1
9.11
87.7
STU-KD-SA
94.2
88.8 0.4
0.5
9..4
74.6
0.2
87.7
STU-KD-AVE
4..
50.8 0.9
6.1
100.0
56.7 0.7
8..7
67.1
STU-KD-SOO KD-
68.4
83.1 0.9
92.7+
1.0
92.5 2.0
97.5
89.0
STU-KD-SCA FCOLD
62.9 1.9+8.. 8.3
81.8
8..0+4.0
1.0
82.5 8.0
92.1+7.7
84.6
STU-KD
81.1 0.8
83.5
0.8
97.3
1.0
94.2 1.8
98.8 2
92.5
STU-KD-SA
84.3 = 0.2
83.7 + 0.3
9.1
100.0 0.0
92.1 0.7
98.8 98.8=0.0 0.0
92.7
Peak meonry footprint (MB)
800
Z2SHOT (ResNet-18)
Communication cost (MB)
Z STU-KD-SSCI
700
TO-KD
Inference Accuracy
STU-KD-SCAFFOLD
600
STU-KD
10
STU--DD-AVG
500
(RexNet-50)
0.6-
STU-TU
STU-KD-SA
400-
0.4
AYG
300-
STU--D-SCAFFOL
10
200
STL- STU-KDa (ems)
STL-KD
100
STL-KT-SCAFFFOLDD
0.0
10
20
30
X 200s
Methods
Epochs
Methods
Figure 3: The peak memory footprint of STU-KD and the comparison methods. companiso
Figure 4: The inference accu- racy curves of STU-KD and its variants on A task.
Figure 5: The communication cost of STU-KD and its vari- ants on W A task.
and the gap is nearly 8. 8.1% in average. Moreover, the communication cost of STU-KD -CCA FFOLD twice that of STÚ-KD, since SCA FFOLD requires uploading both local model updates and a set of control parameters. The comparison on these FL based variants verify that FL is not effective in STU-KD to consolidate the compact models from the edge device and the server. As for STU-KD- SGD, its performance is the closest to that of STU-KD among these three variants. However, its communication cost is too large. As shown in Figure the communication cost of STU-KD-SGD nearly 100 times that of STU-KD. The reason is that the edge device needs to frequently sends its local gradients to the server to compute the gradients of the objective function, which leads to the large communication cost. The results of STU-KD-SA is also reported in Table Compared with STU-KD, STU-KD-SA achieves the comparable inference accuracy on both the source data and the target data, verifying that secure aggregation does not affect the performance of STU-KD, Note that the slightly better performance of TTU-KD-SA than STU-KD is due to the random seeds used in the experiments, and it can disappear when more experiments are conducted.
CONCLUSION
In this paper, source- target unified knowledge distillation (STU-KD) scheme was developed to adapt the compact model on the edge device to the target data while protecting users' privacy. In the scheme, large source model first loaded and adapted the target data the edge device, and a large target model was obtained. Such an adaptation was enabled by a memory efficient lite residual hypothesis transfer algorithm. A collaborative knowledge distillation method that unifies the target data and the source data for training the compact model was developed to transfer the knowledge of the large target model to the compact model while preventing catastrophic forgetting. Secure aggregation was also employed in STU-KD to enhance privacy preservation. Extensive experiments showed that STU-KD can achieve the highest inference accuracy over the target data while maintaining the original high inference accuracy over the source data, compared with state- of of-the-and art schemes. STU-KD is now evaluated on public datasets. How incorporate STU-K STU-KD into real- wal-world application can be an interesting topic for future work.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, A Atonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aron Segal, and Karn Seth. Practical secure aggregation for privacy- preserving machine learning. In ACM SIG SIGSAC Conference on Computer and Communications Security (CCS), pp. 1175-1191, 2017.
Stephen Boyd, Neal Parikh, and Eric Chu. Distributed optimization and statistical learning via the alternating direction method of multipliers. Now Publishers Inc, 2011.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. International Conference on Learning Representations (ICLR), 2019.
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. TinyTL: Reduce memory, not parameters for efficient on-device learning. In Advances in Neural Information Processing Systems (Neurips), volume 33, 2020.
Mathilde Caron, Piotr Bojanowski, A Anman Joulin, and Matthijs Douze. Deep clustering for un- supervised learning of visual features. In European Conference on Computer Vision (ECCV), September 2018.
Shangyu Chen, Wenya Wang, and Sinno Jialin Pan. Cooperative pruning in cross-domain deep neural network compression. In International Joint Conference on Artificial Intelligence (IJ CAT), pp. 222220222222 2108, 2019.
Laurent Dillard, Yosuke Shinya, and Taiji Suzuki. Domain adaptation regularization for spectral pruning. 2020.
Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre- training help deep learning? International Conference on Artificial Intelligence and Statistics, pp. 201-208, 2010.
Xiaoyu Feng, Zhuqing Yuan, Guijin Wang, and Yongpan Liu. Admp: An adversarial double masks based pruning framework for unsupervised cross- dossin compression. arXiv preprint arXiv:2006.041277 2020.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning (ICML), pp. 1180-1189, 2015.
A. Gretton, AJ. Smola, J. Huang, M. Schmittfull, KMM Borgwardt, and B. Schölkopf. Covariate shift and local learning by distribution matching, pp. 131-160. 2009.
Chaoyang He, Murali A Anavaarm, and Salman Avestimehr: Group knowledge transfer: Federat- ed learning of large cnns at the edge. In Advances in Neural Information Processing Systems (Neurips), pp. 14068- 14080, 2020a.
Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, X Xiaoyang Wang, Pra- neeth Vepakomma, A bhishek Singh, Hang Qiu, al. Fedml: A research library and benchmark for federated machine leaming. arXiv preprint 20XI::2077.13111 2020b.
Kaiming He, X Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 778,778, 2016.
Yihui He, Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model compression and acceleration on mobile devices. In European Conference on Computer Vision (ECCV), pp. 784-800, 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 1503.02531.11 2015.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In IEEE International Conference on Computer Vision (ICCV), pp. 1314-1324, 2019.
10

---

Under review as a conference paper at ICLR 2022
Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self aaff-uegeenedd training. In International Conference on Machine Learning (ICML), pp. 1558- 1567, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), pp. 448-456, 2015.
Guoliang Kang, Lu Jiang, Yi Yang, and A Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 4893-4902, 2019.
Sai Praneeth K Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and A nan- da Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning, In Interna- tional Conference on Machine Learning (ICML), pp. 1132-5143, 2020.
A Andreas raasse, Pietro Perona, and Ryan Gomes. Discriminative clustering by regularized informa- tion maximization. Advances in Neural Information Processing Systems (Neurips), pp. 775- 783, 2010.
Vinod K Kurmi, Venkatesh Subramanian, and Vinay Namboodiri. Domain impression: A source data free domain adaptation method. In IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 615-625, 2021.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9641-9650, 2020a.
Yawei Li, Shuhang Gu, Kai Zhang, Luc Van Gool, and Radu Timofte. DHP: Differentiable meta pruning via hypemetworks. In European Conference on Computer Vision (ECCV)), pp. 608-624, 2020b.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning (ICML), pp. 6028- 6028-6039, July 18-18 2020.
Ji Lin, Wei-Ming Chen, Yujun Lin, Chuang Gan, Song Han, al. Mcunet: Tiny deep learning on iot devices. Advances in Neural Information Processing Systems (Neurips), pp. 11711-11722, 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference on Learning Representa tions (ICLR), 2019.
Yuang Liu, Wei Zhang, and Jun Wang. Source-free domain adaptation for semantic segmentation. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1215- 1224, 2021.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. In International Conference on Machine Learning (ICML), pp. 2017.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information P Processing Systems (Neurips), pp. 1647- 1657, 2018.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential leaming problem. In Psychology of Learning and Motivation, volume 24, pp. 169-165. 1989.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Araas. Communication effficieen learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 1273- 1282, 2017.
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage collaborative leamingg In IEEE Symposium on Security and Privacy (SP), pp. 691-706, 2019.
11

---

Under review as a conference paper at ICLR 2022
L. T. Nguyen-Meidine, Meidine, E. Granger, M. Kiran, J. Dolz, and L. A. Blais-Morin Joint progressive kmowledge distillation and unsupervised domain adaptation. In International Joint Conference on Neural Networks (IJ CNN), pp. 1-8, 2020.
X uefei Ning, Tianchen Zhao, Wenshuo Li, Peng Lei, Yu Wang, and Huazhong Y Yang. DSA: More efficient budgeted pruning via differentiable sparsity allocation. In European Conference on Com- puter Vision (ECCV), pp. 592-607, 607, 2020.
dam Paszke, Sam Gross, Francisco Massa, A Aam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, henhhppeffrmance deep learning library. Advances in Neural Information Processing Systems (Neurips), 32:8026- 8037, 2019.
X ingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko. Federated adversarial domain adapta- tion. In International Conference on Learning Representations (ICLR), 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, A ditya Khosla, Michael Bernstein, et al. Imagenet large scale visu- al recognition challenge. International Journal of Computer Vision, 115(3):211-252 2015.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. dapting visual category models to new domains. In European Conference on Computer Vision (ECCV), pp. 213-226, 2010.
Tim Salimans and Durk P Kingma. Weight nomalization: A simple reparameterization to accelerate training of deep neural networks. Advances in Neural Information Processing Systems (Neurips), pp. 901-909, 2016.
Mark Sandler, A ndrew Howard, Menglong Zhu, A ndrey Zhmoginov, and Liang- Chigh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In IEEE Conference on Computer Vision and Pattern Recognition R), pp. 4520- 2018
Baochen Sun and Kate Saenko. Deep CORAL: Correlation aligmment for deep domain adaptation. In European Conference on Computer Vision (ECCV), pp. 443-450, 2016.
Hui Tang and Kui Jia. Discriminative adversarial domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5940- 5947, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine
Learning Research, 9(1),, 2008
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5028-5027, 2017.
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang- A dversarial domain adaptation with domain mixup. In AAAI Conference on Artificial Intelligence (AAAI), volume 34, pp. 1502-6509, 2020.
Jianfei Yang, Han Zou, Shuxin Cao, Zhenghua Chen, and Lihua Xie. MobileDA: Toward edge- domain adaptation. IEEE Internet of Things Journal, 7(8):6909- 6918, 2020.
Chaohui Yu, Jindong Wang, Yiqiang Chen, and Zijing Wu. A ccelerating deep unsupervised domain adaptation with transfer channel pruning. In International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2019.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. iDLG: Improved deep leakage from gradients. arXiv preprint arXiv: 2001. 2001.02610.00 2020.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in Neural Infor- mation P Processing Systems (Neurips), pp. 14774- 14784, 2019.
Martin A Zinkevich, Markus Weimer, Alex Smola, and Lihong Li. Parallelized stochastic gradient descent. In Advances in Neural Information Processing Systems (Neurips), pp. 2595-2600 2603, 2010.
12

---

Under review as a conference paper at ICLR 2022
A LOss FUNCTION OF SHOT
Define the parameters of the feature extractor as w. and the loss function SHOT as Js. Js consists
of two terms: information maximization (IM) loss rauus et all 2010 Hu et all 2017 denoted as Jy and a cross-entropy (CE) loss of self- sepersupeee learning denoted as Jp. The expression of Js can be written as
Js(wa) (.)) = (ww)) + (.p(
(2)
where 3 a balancing hyperparameter. Define the target output of a target sample æ; as fr (xx). IM loss Jn captures the uncertainty of the target outputs of the target data and the diversity of those larget outputs by
K
K
Jr(we)= = Exx
de (x(((())) log du (fr(x:))] +
Pu. log Pk.
where K is the number of classes in the classification task, (a) = in the softmax output of vector and Pu is the k-th element in the mean output embedding p = Ea, Ex, (o(f(()))) of the target data. CE loss Jp is written as
Σ. exp(a;] is the k-th element
exp(az)
(ww)) Ezz, [L(f((x)) 9c)l,
where di is a pseudo-label generated by clustering based Seeudo-labeling mechanism (Caron et al. 2018 Liang et all (2020) for Zt..
B Co-KD FOR MULTIPLE EDGE DEVICES
W When single edge device only has limited number of target data, is hard to leam the fine- grained representations of the target data even for the large model. As a result, the compact model trained by Co-KD still suffers low inference accuracy on the target data. To tackle this challenging, Co-KD can be extended to unify the target data of the multiple edge devices for training a global compact model. The M edge devices collaborate with the cloud server to train global compact model by solving
min
1
Σ
(m)) (w) + (1 a)Jcww)
(3)
M
where w denotes the parameters of the global compact model. The training process of the global compact model is formulated into consensus problem 22 over the M edge device and the server.
M
Σ
( (wm) + +1 (x) Jc
min W0,301 ..... 2P
M
(4)
s.t. Wm we. m
M.
where ((m is the local loss function of KD on edge device m, w.. is the compact model on edge device m, and wo is the compact model updated in the centralized server. Co-KD is then extended to iteratively solve problem wm (t) by
follows. In the t-the iteration of Co-KD, edge device m determines
w. (t) = arg min
a
((m) (wm) + (Am 1), wm - wolt - 1)) + L f1 wolt - 1)1
M K
and sends its Lagrange vector Am (t 1) and wm (t) to the cloud server. Next, the server updates its compact model by
M Σ
wu(() arg min (1 a)Jo( Mno
(m
1), wm wo) + (l (t) woll
The updated compact model is then sent to the M edge devices. A Afterwards, edge device updates 1) via
Am (t
Am (1) Am 1) plum (1) (t))
13

---

Under review as a conference paper at ICLR 2022
3
X 4
X
x
3
Maxpoal
64
256
22
1024
312
512
20-45
AvgPeo
strile
TC
Figure 6: Source
As the convergence criterion is reached. Co-KD stops. The compact model with (t) then broadcast to the M edge devices their new compact models.
Similarly the case single edge device, SGD is utilized determine wm (t) on edge device m and to determine wo) on the server. More specifically, the gradient gm (wm 1)) used to update wm (t - 1) is computed by
Im (mm m (t. 1))
V Vw..
fmmm) (w (t. 1)) Am 1) + pm (t wo 1))-
Let w'm (t) = Am (t- 1) + (t) denote the parameters uploaded by edge device m. The gradient go(wml (t- 1)) used to update wo(t can then be represented as
golwo(t 1)) V w. Jo(wo
(Am 1) + pl (m wolt 1)))
(5)
M
Vw, 1)) + M pwwllt - 1) Σ m=1
w (t).
C ADDITIONAL INFORMATION ABOUT SECURE AGGREGATION
To generate the same random vector edge device and edge device the same random seed used in the random number generator on each edge device. Such a random seed can shared among the two users using Hellemn key exchange while keep the server unknown to the random seed.
D MORE DETAILS OF EXPERIMENTAL SETUP
Three public datasets for domain adaptation are used to evaluate STU-KD, Office-31 (Saenko et all, 2010) is widely used dataset for domain adaptation. It has 4, 652 images and 31 classes collect- ed from three domains: A mazon (A), DSLR (D), and Webcam (W). ImageC L EF-DA has three domains: ImageNet ILSVRC 2012 (I), Caltech- 256 (C), and Pascal VOC 2012 (P). For each do- main, there are 12 classes and 50 images in each class. O Office-Home (Venkateswara et al 2017) has 15, 500 images and 65 classes from four distinct domains: A rtistic images (Ar), Clip Art (CI),
Product images (Pr), and Real-Worl Woold images (Rw).
For STU-KD, variant of ResNet-50 (He et all, 2016) is selected the large model. Concretely, a standard ResNet-50 model pretrained over ImageNet (Russakovsky et all 2015)) is employed as the base module. Its original fully connected (FC) layer is then replaced with a bottleneck lay- er and a task-specific FC layer. Moreover, batch normalization (BN) layer (loffe & Szegedy. 2015) is put after the FC layer the bottleneck layer, and a weight normalization (WN) layer (Salimans & Kinama 2016) is put after the task- specific FC layer (Liang et al., 2020). Following the same procedure, a variant of ResNet-18
constructed and is selected as the compact model
for all the schemes. Moreover, four lite residual modules are added to the ResNet-50 50 model for lite residual hypothesis transfer. The architecture of the ResNet-S 50 model with four lite residual modules is shown in Figure
14

---

Under review as a conference paper at ICLR 2022
Table 5: Inference accuracy (%) data and source data)
Office-31 dataset with different hyperparameter settings (target
Setting
A-D
A-W
D-
D-W
W W-A
W-D
Avg.
0.5. 0.3
94.2 - 0.2
88.7
714 0.3
97.9 0.1
74.6 0.3
19.1:0.11
87.7
p 0.3
94.2 0.3
88.7+
0.2
97.9
74.7+
99.2++0.0
87.7
a p = 0.3
0.3
0.1
7.1
97.7
0.2
9.0
87.7
a p. - 0.3
9..0=
8.88
0.3
9.1
74.6
9.11
87.7
a 0 = 0.5
9..0=
8..
7.41
9.1
0.3
9.0
87.7
0.8. p 0.7
913
8.1
0.2
97.7
71.5
99.2
87.7
a 0.5.p
86.1
8.8
9.0
1.0
9.7
9.7
94.4
a 0.6.p = 0.3
83.9
8..1
9.0
1.0
0.7
9.0
93.6
a 0. = 0.3
1.0
8.
9..
100.0
9.0
9..
93.4
a 0. 0.3
0.
8.8
9.1
1.0
94
9..
92.5
0.8, p 0.5
1.2
83.9
98.0 0.0
1.0
96.2
99.2 0.7
92.9
0.8, p 0.7 81.2
1.2
84.2
98.0 0.0
100.0
95.8 0.7
98.8 0.0
93.0
The compact model and the large source model are trained through backpropagation, and mini- batch SGD with momentum 0.9 and weight, decay 1e-3 is adopted as the optimizer. The learning rate is set to 7/ = 0.001 for the lite residual modules in the large model and the feature ex extractor in the compact model. A widely used learning rate scheduler (Ganin & L Lempitsky, 2015: Long et all 2017: 2018 8) is also adopted, i.e., 1) n1 + 10p) 10p)-0.75, -0.75, where is the training progress changing from 0 to
E MORE EXPERIMENTAL RESULTS
E.1 PAPEPARRA ANALYSE SIS
To analyze the effects of hyperparameters on STU-KD, the experiment of STU-KD on Office-31 (Table D data is repeated with different settings of the hyperparameters a and p. The results are shown in Table 5 In terms of the inference accuracy on the target data, the overall performance of STU-KD with different settings are comparable, indicating that the inference accuracy on the target data is not much sensitive to the settings of the hyperparameters. Now consider the inference accuracy on the source data. It is obvious that smaller a leads to higher inference accuracy on the source data. The reason is as follows. The smaller is the value of a, the larger is the weight of the CE loss in equation (. As a result, the global compact model needs to achieve a smaller error over the source data after the adaptation, which can lead to higher inference accuracy on the source data. As for the results show that the larger is the value of the large is the inference accuracy on the source data, though the effect of p is not strong as that of a. The above results indicate that the inference accuracy on the source data is more sensitive to the setting of the hyperparameters than the inference accuracy on the target data. Properly reducing the value of a or increasing the value of p can achieve better inference accuracy on the source data.
E.2 DIFFERENT NETWORK ARCHITECTURES
To show that STU-KD is applicable to different network architectures, two additional experiments conducted. In the first experiment, Proxyless-Mobee Mobile (Cai et all, 2019) is selected as the compact model, and ResNet-50 (He et all, 2016) is selected as the large source model. The results are shown in Table[G] In the second group, ResNet- 34 selected as the compact model, and the ResNet- 101 is selected as the large source model. The results are shown in T
ccording to Table and Table TO-KD and STU-KD (our schemes) outperforms other schemes in terms of the inference accuracy on the target data, and these two schemes achieve comparable inference accuracy on the target data. Moreover, STU-KD outperforms TO-KD in terms of the inference accuracy on the source data. Such results are consistent with that of Table [ Table 2 and Table ] indicating that STU-KD is applicable to different network architectures for both the compact model and the large source model.
15

---

Under review as a conference paper at ICLR 2022
Table 6: Inference accuracy (%) Office-31 dataset with Proxyless- Moxyless--obile the compact model and ResNet-50 as the large source model (target data and source data)
Method
A-+
A-W
D-A
D-W
W-+A
W-+
Avg.
Proxyless Mobile (Cal et 2019)
73.7 0.0
61.9
0.0
95.6 0.0
36.2 0.0
99.2 = 0.0
73.8
SHOT (Liann al. 2020
87.4
85.5 0.4
66.8 0.8
97.9 0.1
66.9
98.6 0.6
83.9
TO-KD
0.44
88.9 0.1
0.1
97.3
0.1
98.8
87.8
STU-KD
94.1 0.3
89.2 + 0.2
73.6 0.1
97.6 + 0.2
73.8 0.1
99.2 + 0.0
87.9
Proxyless- Mobile (Cal al., 2019)
89.7
89.7 0.0
100.0 +:
100.0 + 0.0
100.0
100.0 0.0
96.6
SHOT (Liang at. 2020)
0.5
78.6 0.2
3.7
100.0 1: 0.0
85.
98.8
89.4
TO-KD
63.0 2:00::2.5 2.5
0.5 84.2
68.7
78.0
100.0 1: 0.0
76.2 3.1
98.8 0.0
80.8
STU-KD
84.5
100.0
100.0 0.0
98.8
100.0
94.6
Table 7: Inference accuracy (%) on Office-31 dataset with ResNet- 34 as the compact model and ResNet- 101 as the large source model (target data and source data)
Method
A >D
A :W
D >
D >
W
>
W
>D
Avg.
ResNet:34 34 (He et al., 2016)
73.1 0.0
70.8 + 0.0
90.3 +
53.1
+ 0.0
97.4
72.2
SHOT (Liang al, 2020
84.3...3 0.8
86.0
0.6
60.5 0.6
94.5
66.6
1 0.3
98.0 0.6
81.6
TO-KD
96.3 0.1
90.8 0.3
77.0 0.1
96.5 0.2
0.1
99.6
89.6
STU-KD
96.3 + 0.1
93.0 + 0.1
0.1 76.5
97.9 + 0.1
77.7 0.2
99.6 +
90.2
ResNet-34 (He et al., 2016)
88.7
88.7
96.0 0.0
9.0
98.8
98.8
94.5
SHOT (Liang et al, 2020)
17.4144
77.5
68.7= 2.8
97.3
2..
0.0
81.5
TO-KD
61. - 0.6
40.3 - 8.3
78.0 1.8
1.8
99.3
16.7
94.2
75.6
STU-KD
83.3 1.2
85.9
1 0.2
97.3
1.1
100.0 0.0
97.9 0.7
98.8 0.0
93.9
E.3 VISUALIZATION
In the case of a single device, the representations leam by ResNet- 18, SHOT, TO-KD, and STU- KD are visualized by t-SNE (Van der Maaten & Hinton, 2008), shown in Figure 7 As expected, the representations leamed by STU-KD are the most discriminative for both target domain and the source domain. A lthough TO-KD achieves comparable inference accuracy on the target data that achieved by STU-KD, the target representations learned by KD is less discriminative than that of STU-KD. In terms the source representations, SHOT outperforms TO-KD, which verifies the results in Table [ Such result also indicates that directly using KD can lead severe catastrophic forgetting.
E.4 RESULTS OF MULTIPLE EDGE DEVICES
Two group of experiments are conducted in the case of multiple edge devices. First, the target data on the multiple edge devices come from the same target domain. Second, the target data on the multiple edge devices come from different target domains. More specifically, in the second experiments, the
Table 8: Inference accuracies (%) on the data from A
W task (target data and source data)
Method
Device 1
Device 2
Device 3
Device 4
Device 5
Avg.
SHOT (Liang et al,, 2020)
76.35
77.99
75.85
73.84
76.35
76.08
TO-KD
79.54
81.84
83.02
81.01
83.86
81.85
STU-KD
88.43
88.43
88.43
88.43
88.43
88.43
STU-KD-SA
88.43
88.43
88.43
88.43
88.43
88.43
SHOT (Liang et al, 2020)
77.66
76.24
77.30
78.37
75.53
77.02
TO-KD
77.78
77.66
77.78
77.54
76.95
77.34
STU-KD
87.12
87.12
87.12
87.12
87.12
87.12
STU-KD-SA
87.70
87.70
87.70
87.70
87.70
87.70
16

---

Under review as a conference paper at ICLR 2022
Seave Tinget
Seave
(a) ResNet- 18
(b) SHOT
Sourc Dirget
Soutc Target
(c) TO-KD
(d) STU-KD
Figure 7: Visualization results of (a) ResNet- 18, (b) SHOT, (c) TO-KD, and (d) STU-KD A D task.
Table 9: Inference accuracies (%) on the data from A
D task (target data and source data)
Method
Device 1
Device
Device
Device 4
Device
Avg.
SHOT (Liang et al,, 2020)
74.50
71.29
75.50
77.71
75.30
74.86
TO-KD
80.32
76.64
78.11
79.99
81.86
79.38
STU-KD
85.81
85.81
85.81
85.81
85.81
85.81
STU-KD-SA
85.68
85.68
85.68
85.68
85.68
85.68
SHOT (Liang et al, 2020)
78.72
78.37
74.82
75.53
78.37
77.16
TO-KD
76.71
76.24
73.29
75.41
76.60
75.65
STU-KD
86.76
86.76
86.76
86.76
86.76
86.76
STU-KD-SA
86.17
86.17
86.17
86.17
86.17
86.17
17

---

Under review as a conference paper at ICLR 2022
Table 10: Inference accuracies (%) on the data from A + DW task (target data and source data)
Method
Device 1
Device
Device
Device
Device 5
Device 6
STU-KD (separated)
90.36
90.36
90.36
88.59
88.59
88.59
STU-KD (joint)
89.62
89.62
89.62
90.19
90.19
90.19
STU-KD (separated)
86.99
86.99 87.11
86.99
87.58
87.58
87.58
STU-KD (joint)
87.11
87.11
87.11
87.11
87.11
Table 11: Inference accuracies (%) on the data from I - CP task (target data and source data)
Method
Device 1
Device
Device
Device
Device 5
Device 6
STU-KD (separated)
95.22
95.22
95.22
77.38
77.38
77.38
STU-KD (joint)
95.30
95.30
95.30
77.55
77.55
77.55
STU-KD
(separated)
96. 11 96.11
96.11 1
96.11
98.33
98.33
98.33
STU-KD (joint)
96.11 1
96.11
96.11
96.11
96.11
edge devices are divided into two groups. The devices from the same group have the target data from the same domain, while the target domains of the two groups are different.
In the first group of experiments, M is set 5, and the target data are randomly partitioned into parts each of which is held by one edge device. Three schemes are evaluated on Office-31 31 dataset: SHOT (Liang et all, (020), TO-KD, and STU-KD. For SHOT and TO-KD, each edge device trains its local compact model independently, without interactions with the centralized server. For STU- KD, five edge devices and the server collaboratively train global compact model. The experimental results on A W task and that A D are reported in Table and Table respectively.
As shown in Table B and Table 9 TO-KD outperforms SHOT in terms of the inference accuracy on the target data. The gain comes from the large model capacity of large model. By unifying the target data from all the edge devices, STU-KD further improves the average inference accuracy on the target data by 6.58% on A + W task, and by 6.43% on A + task. Besides, STU-KD leverages the source data to train the global compact model, and thus achieves the highest inference accuracy on the source data, A After employing secure aggregation, STU-KD-SA achieves com parable performance to that of STU-KD, indicating that secure aggregation does not affect the performance of STU-KD.
In the second group of experiments, two transfer tasks are conducted: 1) A from Office-31 31 dataset selected as the source domain, and D and W from Office-31 are selected as the target domains, denoted as A DW; 2) I from ImageCL DF-ageLLFFDDD DA dataset is selected as the source domain, and C and from ImageCLEF-DA are selected as the target domains, denoted as - CP. task A DW, the two target domains are similar; while in task I - CP, the two target domain are much different from each other. The number of edge device M is set 6. For each transfer task, the first three edge devices come from the one target domain, and the rest three edge devices come from another target domain. Two schemes are used to trained the compact models on the edge devices. First, STU-KD is only applied to the edge devices from the same target domain, denoted STU-KD (separated). Second, STU-KD is applied to all thee six edge devices to leam a global compact model, denoted as STU-KD (joint). The results of task A DW and task I + CP shown in [able|[[]and Table respectively.
Comparing the performance of these two schemes, it indicated that applying STU-KD to the case of different target domains does not highly degrade the performance of the obtained compact model. That is, STU-KD applicable to the case of different target domains, though STU-KD does not provide extra performance gain for the compact model by involving more data into the training process.
18


Reviews:
[{"review_id": "sv8mIZ5aYTP", "reviewer": "Reviewer_wCzy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper solve the domain shift problem of the edge devices from mainly 3 aspects:\n1. The cost of fine-tuning/training a large source model loading on the edge device using the target data. To solve this problem, the author uses several lite-residual modules to imply hypothesis transfer, which can save a lot of memory.\n2. Distilling the knowledge from the large model to a compact model. To maintain source knowledge during the distilling process, a source-target unified distilling method is designed.\n3. Train the compacted model in a distributed manner considering privacy problem.\nWhat are the advantages of using lite-residual modules compared with adding another full-connected features layers?\n", "summary_of_the_paper": "This paper solves a domain adaptation problem on the edge device. Using the designed hypothesis transfer and knowledge distillation method, the training cost can be reduced and the inference performance on the target data can be enhanced.", "summary_of_the_review": "This paper aims to solve a domain adaptation problem on the edge device which is a common problem today. The proposed method can help to improve the performance of a various of edge devices which I think is feasible and useful.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hwtsDbA9pDC", "reviewer": "Reviewer_uoT4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Summary\nFor me, the considered problem is new and complex, but not realistic. The proposed source-target knowledge distillation scheme is complicated and impractical. The experimental comparisons and the relevant analyses are rough. \n\n## Pros\n1. The considered problem setting is new and challenging.\n2. The proposed method is technically sound. \n3. This paper is well-written and easy to follow.\n\n## Cons\n1. The problem setting is not clear, which makes it not realistic. If imposing resource constraints on target clients, only considering memory footprint is not enough. Other factors like the FLOPs of the model are missing. Since the authors consider the knowledge forgetting of the source central server, do source data should share the same label space as target data? The federated learning setting is realistic because it can extend to scenarios with many clients. However, this paper mainly considers one client. The multiple clients in the appendix all come from one target domain, but the realistic setting is that different clients come from different domains.\n\n2. The proposed source-target knowledge distillation scheme combines lots of existing techniques. Besides, the proposed scheme is complex and impractical, involving many hyperparameters. In the scheme, the authors directly use SHOT (ICML'20) for adaptation from a large source model to a large target model. It is confusing to introduce the complex lite residual hypothesis transfer just to avoid the memory adaptation during adaptation. Have you tried other memory-efficient adaptation methods with the large source model? I think an elegant and promising solution is to use larger source models to provide only penultimate-layer features or the predictions of the target data, like [1, 2]. The ADMM solution to collaborative knowledge distillation seems reasonable, but the two introduced hyperparameters, i.e., $\\alpha$ and $\\rho$, make the method impractical.\n\n3. Experimental comparisons are unfair and analyses are insufficient. In Table 1&2&3, other baselines only use ResNet-18 for all steps, while the proposed methods use pre-trained ResNet-50 as the source model. Under this situation, it is meaningless to say how the proposed methods outperform other baselines. Besides, I think as a federated setting paper, only considering one client for most experiments is unconvincing. I encourage the authors to add experiments with many clients and maybe clients from different domains. Analysis of the hyperparameters is missing. Advanced baselines of federated learning should be considered, FedAVG is not up-to-date. Details on how to split target data to multiple devices in the appendix should be provided. Does the data split consider the balance of classes? In knowledge distillation or model compression, analysis on different architectures is required. However, in this submission, the large network is fixed as ResNet-50 and the compact network is fixed as ResNet-18.\n\n## References\n[1] Unsupervised Domain Adaptation of Black-Box Source Models, arxiv preprint\n\n[2] Distill and Fine-tune: Effective Adaptation from a Black-box Source Model, arxiv preprint\n", "summary_of_the_paper": "This paper considers a new and complex setting involving domain adaptation, federated learning, and knowledge distillation: Under the premise of protecting privacy, one needs to deploy a compact model from a source central server to target client devices and requires the model to learn new knowledge with target unlabeled client data while remembering knowledge of source data on the central server. Generally, the authors propose a source-target unified knowledge distillation scheme. Within this scheme, the authors propose solutions to tackle corresponding difficulties with this setting. Specifically, to avoid the low inference accuracy due to low model capacity, a large source central model is adapted to target clients with SHOT (ICML'20). A lite-residual hypothesis transfer method is proposed to keep memory-efficient adaptation on target clients. A collaborative knowledge distillation method is proposed to defy catastrophic forgetting of source knowledge. To protect the privacy of target clients, a secure aggregation method is used. The authors validate the effectiveness of the proposed scheme on three domain adaptation datasets.\n", "summary_of_the_review": "Generally, in the current stage, I recommend rejecting this submission. Justifications are in the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HTqdbYW5hxd", "reviewer": "Reviewer_99KT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-- This submission targets to a practical problem: train a model with high generalization ability on a decentralized setting\n\nWeakness:\n-- Both steps in the proposed method are from previous works, which include: the lite-residual module,  the combination of a supervised loss and a KL loss. And therefore, the technical contribution of this works seems weak, which concerns me.\n\n-- The experimental results are not convincing enough. IN Table 1 and Table 3, STU-KD is outperformed by some prior works. Based on the reported results, it is hard to demonstrate the efficacy of the proposed method. Or do I miss anything here? If yes, please correct me.\n\n-- The draft needs to be carefully checked, for example, in Eq. 1, there are two J_{p} (w_{r})s in the equation. ", "summary_of_the_paper": "This submission proposes a unified knowledge distillation to train a compact model, which is expected to perform well on target data in a different statistical distribution. Generally, the proposed method consists of two major steps: the first step is to introduce lite-residual connection to conduct lite-residual hypothesis transfer; the second step is to combine supervised loss (based on source data) and a KL loss (based on source data and target data) to make the trained model not forget source data knowledge.", "summary_of_the_review": "This submission targets to a practical research problem: how to train a model with high generalization ability via decentralized data. However, the technical novelty is limited and the experimental results are not convincing enough. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "18GnKg6pODh", "reviewer": "Reviewer_uaQn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n- Overall a well-explained methodology that is well-adapted to the particular system set-up. \n- Good problem explanation and literature review. \n- Nice to see a discussion on privacy. \n\nWeaknesses: \n- In terms of performance on the target data, TO-KD method performs better in certain settings and in the cases STU-KD outperforms it is only by narrow margins. So then what is the benefit of the additional effort of training on the source data? Especially considering that it requires additional computational resources. Clearly, the performance on the source data is worse for TO-KD, but how important is that for the device?\n\nSmaller comments: \n- For me it remains unclear why we need to both adapt large model to target data as well as train the compact model on source data + maintain a close distance to large model output over target. What would the performance be if I were to just train the compact model on a loss fn with source + KL-divergence (without having adapated the global model to target data first)? Or has this been done in previous work?\n\n", "summary_of_the_paper": "A compact model deployed to a device may not work well if this device has a different data distribution. This work proposes to load a large pretrained model onto a device and then adapt it to the target data on the device. As directly training the full large model is too memory-heavy, this work proposes to adapt the large model's knowledge by training only part of its parameters on the local device data. To then transfer global model knowledge to the compact model this work proposes a collaborative knowledge distillation. ", "summary_of_the_review": "Paper solving a problem of interest with a well-explained methodology but questions on results remain", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers a domain adaptation setting where a source domain model trained on a server is adapted on a client using target domain dataset. The paper considers the setting where the client only has a modest memory footprint (e.g., an edge device) and uses a recently proposed technique "TinyTL" (NeurIPS 2020) which is based on freezing the network weights but only updating the biases and adding a lightweight residual module. The basic idea of the paper is also based on SHOT (ICML 2020).

While the reviewers appreciate the problem setting and the basic idea, there were several concerns, some of which included:

- Limited novelty: The paper's key ideas are largely based on SHOT and TinyTL and a simple combination of these with not such significant challenges or insights offered.
- Federated setting not considered adequately: Although the paper title and the abstract/introduction talk about the federated setting, the paper largely focuses on a single source and single client setting.
- Inadequate baselines and experiments: The federated learning baselines used in the paper are fairly basic ones (e.g., FedAvg). Some of the experimental results are not convincing enough.

The paper received mixed scores and the reviewers engaged in discussions with the authors. However, the concerns still linger. Based on the reviews, discussion, and my own reading and assessment of the paper, I think the paper falls short of the acceptance threshold. The authors are advised to consider the reviewers' concerns to improve the manuscript for a future submission.
Decision: Reject