Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers a new and complex setting involving domain adaptation, federated learning, and knowledge distillation: Under the premise of protecting privacy, one needs to deploy a compact model from a source central server to target client devices and requires the model to learn new knowledge with target unlabeled client data while remembering knowledge of source data on the central server. Generally, the authors propose a source-target unified knowledge distillation scheme. Within this scheme, the authors propose solutions to tackle corresponding difficulties with this setting. Specifically, to avoid the low inference accuracy due to low model capacity, a large source central model is adapted to target clients with SHOT (ICML'20). A lite-residual hypothesis transfer method is proposed to keep memory-efficient adaptation on target clients. A collaborative knowledge distillation method is proposed to defy catastrophic forgetting of source knowledge. To protect the privacy of target clients, a secure aggregation method is used. The authors validate the effectiveness of the proposed scheme on three domain adaptation datasets.

main_review: ## Summary
For me, the considered problem is new and complex, but not realistic. The proposed source-target knowledge distillation scheme is complicated and impractical. The experimental comparisons and the relevant analyses are rough. 

## Pros
1. The considered problem setting is new and challenging.
2. The proposed method is technically sound. 
3. This paper is well-written and easy to follow.

## Cons
1. The problem setting is not clear, which makes it not realistic. If imposing resource constraints on target clients, only considering memory footprint is not enough. Other factors like the FLOPs of the model are missing. Since the authors consider the knowledge forgetting of the source central server, do source data should share the same label space as target data? The federated learning setting is realistic because it can extend to scenarios with many clients. However, this paper mainly considers one client. The multiple clients in the appendix all come from one target domain, but the realistic setting is that different clients come from different domains.

2. The proposed source-target knowledge distillation scheme combines lots of existing techniques. Besides, the proposed scheme is complex and impractical, involving many hyperparameters. In the scheme, the authors directly use SHOT (ICML'20) for adaptation from a large source model to a large target model. It is confusing to introduce the complex lite residual hypothesis transfer just to avoid the memory adaptation during adaptation. Have you tried other memory-efficient adaptation methods with the large source model? I think an elegant and promising solution is to use larger source models to provide only penultimate-layer features or the predictions of the target data, like [1, 2]. The ADMM solution to collaborative knowledge distillation seems reasonable, but the two introduced hyperparameters, i.e., $\alpha$ and $\rho$, make the method impractical.

3. Experimental comparisons are unfair and analyses are insufficient. In Table 1&2&3, other baselines only use ResNet-18 for all steps, while the proposed methods use pre-trained ResNet-50 as the source model. Under this situation, it is meaningless to say how the proposed methods outperform other baselines. Besides, I think as a federated setting paper, only considering one client for most experiments is unconvincing. I encourage the authors to add experiments with many clients and maybe clients from different domains. Analysis of the hyperparameters is missing. Advanced baselines of federated learning should be considered, FedAVG is not up-to-date. Details on how to split target data to multiple devices in the appendix should be provided. Does the data split consider the balance of classes? In knowledge distillation or model compression, analysis on different architectures is required. However, in this submission, the large network is fixed as ResNet-50 and the compact network is fixed as ResNet-18.

## References
[1] Unsupervised Domain Adaptation of Black-Box Source Models, arxiv preprint

[2] Distill and Fine-tune: Effective Adaptation from a Black-box Source Model, arxiv preprint

summary_of_the_review: Generally, in the current stage, I recommend rejecting this submission. Justifications are in the main review.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper solves a domain adaptation problem on the edge device. Using the designed hypothesis transfer and knowledge distillation method, the training cost can be reduced and the inference performance on the target data can be enhanced.
main_review: This paper solve the domain shift problem of the edge devices from mainly 3 aspects:
1. The cost of fine-tuning/training a large source model loading on the edge device using the target data. To solve this problem, the author uses several lite-residual modules to imply hypothesis transfer, which can save a lot of memory.
2. Distilling the knowledge from the large model to a compact model. To maintain source knowledge during the distilling process, a source-target unified distilling method is designed.
3. Train the compacted model in a distributed manner considering privacy problem.
What are the advantages of using lite-residual modules compared with adding another full-connected features layers?

summary_of_the_review: This paper aims to solve a domain adaptation problem on the edge device which is a common problem today. The proposed method can help to improve the performance of a various of edge devices which I think is feasible and useful.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This submission proposes a unified knowledge distillation to train a compact model, which is expected to perform well on target data in a different statistical distribution. Generally, the proposed method consists of two major steps: the first step is to introduce lite-residual connection to conduct lite-residual hypothesis transfer; the second step is to combine supervised loss (based on source data) and a KL loss (based on source data and target data) to make the trained model not forget source data knowledge.
main_review: Strengths:
-- This submission targets to a practical problem: train a model with high generalization ability on a decentralized setting

Weakness:
-- Both steps in the proposed method are from previous works, which include: the lite-residual module,  the combination of a supervised loss and a KL loss. And therefore, the technical contribution of this works seems weak, which concerns me.

-- The experimental results are not convincing enough. IN Table 1 and Table 3, STU-KD is outperformed by some prior works. Based on the reported results, it is hard to demonstrate the efficacy of the proposed method. Or do I miss anything here? If yes, please correct me.

-- The draft needs to be carefully checked, for example, in Eq. 1, there are two J_{p} (w_{r})s in the equation. 
summary_of_the_review: This submission targets to a practical research problem: how to train a model with high generalization ability via decentralized data. However, the technical novelty is limited and the experimental results are not convincing enough.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A compact model deployed to a device may not work well if this device has a different data distribution. This work proposes to load a large pretrained model onto a device and then adapt it to the target data on the device. As directly training the full large model is too memory-heavy, this work proposes to adapt the large model's knowledge by training only part of its parameters on the local device data. To then transfer global model knowledge to the compact model this work proposes a collaborative knowledge distillation. 
main_review: Strengths: 
- Overall a well-explained methodology that is well-adapted to the particular system set-up. 
- Good problem explanation and literature review. 
- Nice to see a discussion on privacy. 

Weaknesses: 
- In terms of performance on the target data, TO-KD method performs better in certain settings and in the cases STU-KD outperforms it is only by narrow margins. So then what is the benefit of the additional effort of training on the source data? Especially considering that it requires additional computational resources. Clearly, the performance on the source data is worse for TO-KD, but how important is that for the device?

Smaller comments: 
- For me it remains unclear why we need to both adapt large model to target data as well as train the compact model on source data + maintain a close distance to large model output over target. What would the performance be if I were to just train the compact model on a loss fn with source + KL-divergence (without having adapated the global model to target data first)? Or has this been done in previous work?


summary_of_the_review: Paper solving a problem of interest with a well-explained methodology but questions on results remain
review_title: null

Meta-review: This paper considers a domain adaptation setting where a source domain model trained on a server is adapted on a client using target domain dataset. The paper considers the setting where the client only has a modest memory footprint (e.g., an edge device) and uses a recently proposed technique "TinyTL" (NeurIPS 2020) which is based on freezing the network weights but only updating the biases and adding a lightweight residual module. The basic idea of the paper is also based on SHOT (ICML 2020).

While the reviewers appreciate the problem setting and the basic idea, there were several concerns, some of which included:

- Limited novelty: The paper's key ideas are largely based on SHOT and TinyTL and a simple combination of these with not such significant challenges or insights offered.
- Federated setting not considered adequately: Although the paper title and the abstract/introduction talk about the federated setting, the paper largely focuses on a single source and single client setting.
- Inadequate baselines and experiments: The federated learning baselines used in the paper are fairly basic ones (e.g., FedAvg). Some of the experimental results are not convincing enough.

The paper received mixed scores and the reviewers engaged in discussions with the authors. However, the concerns still linger. Based on the reviews, discussion, and my own reading and assessment of the paper, I think the paper falls short of the acceptance threshold. The authors are advised to consider the reviewers' concerns to improve the manuscript for a future submission.
Decision: Reject