Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: I read the authors response. I am satisfied with the explanations on the privacy party. However, decentralized training part is still unsatisfactory since the empirical evaluation is not really decentralized. Back of the envelope calculations are at best correlated to the actual times spent by each node.  Hence, the numbers in Table 3 are not physical numbers, rather result of an idealized network. Moreover, the x-axis of Figure 2 being training step is still not acceptable. Decentralized and centralized methods should be compared in terms of time which is the only fair metric. I stick to my original decision.

-----

The manuscript is proposing a method for domain adaptation in a private and distributed setting where there are multiple target domains and they are added in a sequential manner. The proposed method considers only the domain adaptation methods in which the source model training and the target model training are done separately. In this setting, existing adapted models can be used as a source domain since a trained model suffices for adaptation. One major contribution of the paper is proposing a straightforward but successful method to choose which domain to adapt from. The main algorithmic tool is estimating Wasserstein distance and choosing the closest domain. The second contribution is distributed training setting for privacy and decentralization.

Choosing which model to adapt from is an interesting contribution. The proposed setting is definitely sensible and the proposed method is theoretically sound. Hence, I consider this as a valuable contribution to the domain adaptation literature. Moreover, results suggest that it also results in significant performance improvement.

Privacy and decentralized learning part has major issues. First of all, the privacy and learning private models is a sub-field of machine learning with a large literature. Authors do not discuss any of these existing work. Second of all, authors do not specify the definition of privacy they are using. Only guarantee the  algorithm provides is not passing data around. However, this is clearly not enough. Passing gradients might result in sharing sensitive data. The actual data can be reconstructed (upto some accuracy) using the gradients passed between nodes. Therefore, either a justification or a privacy guarantee result is needed. Both of these are major issues which need to be fixed.

Decentralized learning is also an important problem which have been studied significantly. Related work section is missing majority of recent and existing work on distributed learning and federated learning. Moreover, empirical study is very counter intuitive. Results are given in terms of accuracy vs number of training steps. The important metrics are amount of massages passed and the total time of the distributed training. Many distributed algorithms trade off having less accurate gradients (consequently having higher number of gradient updates) with less message passing. Hence, I am not sure how to understand the distributed domain adaptation experiments. I am not even sure what the time in Table 3 actually means since it is clearly not even experimented in a distributed setting.

In summary, the submission is addressing an important problem. Moreover, the contribution on collaborator selection is interesting and seems to be working well. However, the private and decentralized learning parts are rather incomplete from related work and experiment sense. Moreover, I am also not sure can we call this method private or not.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper focuses on the problem of unsupervised domain adaptation in practical machine learning systems. To address the problems in current unsupervised domain adaptation methods, the authors propose to a novel multi-step framework which works in a decentralized and distributed manner. Experimental results show the effectiveness of the proposed approach. 

This paper is well-motivated and the proposed method is novel for unsupervised domain adaptation. The paper is well-supported by theoretical analysis, however, the improvements are not that significant on some experimental results. For the above reasons, I tend to accept this paper but wouldn't mind rejecting it. 

Questions: 
1. The experiments do not really show the superiority of the proposed method compared to the common centralized approaches as they have similar performances on both collaborator selection and distributed domain adaptation. Can you convince the readers more with some other experiments?
2. What is the weakness of such decentralization models?
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: 

###Summary###

This paper tackles unsupervised domain adaptation in a decentralized setting.  The high-level observation is that the conventional unsupervised domain adaptation has two bottlenecks, namely excessive centralization and poor support for distributed domain datasets. The paper proposes Multi-Step Decentralized Domain Adaptation (MDDA) to transfer the knowledge learned from the source domain to the target domain without sharing the data.  

The paper also explores explore a proposition: in addition to adapting from the labeled source, can uDA leverage the knowledge from other target domains, which themselves may have undergone domain adaptation in the past. 

The proposed MMDA method contains a feature extractor (E), a domain discriminator (D) and task classifier (C) for each domain. The target domain components are initialized with the respective source components. The source domain discriminator D_s target domain discriminator D_t are synchronized by exchanging and averaging the gradients. The paper also proposes Lazy Synchronization to reduce the communication cost of the algorithm.

The paper also proposes Wasserstein distance guided collaborator selection schema to perform the domain adaptation task. 

The paper performs experiments on five image and audio datasets: Rotated MNIST, Digits, and Office-Caltech, DomainNet and Mic2Mic. 

The baselines used in this paper include "Labeled Source", "Random Collaborator", and "Multi-Collaborator". The experimental results demonstrate that the proposed method can outperform the baselines on some of the experimental settings. The paper also provides a detailed analysis of the model and experimental results. 

### Novelty ###

This paper does not propose a new domain adaptation algorithm. However, the paper introduces some interesting tricks to solve the MMDA task such as the lazy synchronization between the source domain discriminator and the target domain discriminator. 

###Clarity###

Several critical explanations are missing from the paper:
1) When training the source domain discriminator D_s and target domain discriminator D_t, if the features between the source domain and target domain cannot be shared with each other, how to train the D_s and D_t. For example, the D_s cannot get access to the features from the target domain, how to train D_s? 
2) How is the target classifier C_t updated when there are no labels for the target domain?
3) As far as I understand, the domain discriminator is this paper is trained adversarially. The detailed adversarial training step is unclear.  

###Pros###

1) The paper proposes an interesting transfer learning schema where the data between the source and target domain can not be shared with each other to protect the data-privacy.

2) The paper provides extensive experiments on multiple standard domain adaptation benchmarks, especially the most recent dataset such as the DomainNet. 

3) The paper provides detail empirical analysis to demonstrate the effectiveness of the proposed methods. 

###Cons###

1) The most critical issue of this paper is that some explanations are missing, e.g. how are D_s, D_t, C_t trained? Refer to the #Clarity.

2) The presentation and writing of this paper need polish. The author should do more relative surveys to motivate the authors. One critical relevant reference of this paper is:
"Secure Federated Transfer Learning", Yang Liu et al 

https://arxiv.org/pdf/1812.03337.pdf

3) The baselines used in this paper is also trivial. It is desirable to compare the proposed method with state-of-the-art domain adaptation methods.

Based on the summary, cons, and pros, the current rating I am giving now is "reject". I would like to discuss the final rating with other reviewers, ACs.
To improve the rating, the author should explain the questions I proposed in the #Clarity
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper focuses on the problem of domain adaptation among multiple domains when some domains are not available on the same machine. The paper builds a decentralized algorithm based on previous domain adaptation methods.

Pros: 
1. The problem is novel and practical. Previous domain adaptation assumes that source and target domains are available but it can happen when the source and target domains have connection issues.
2. The method exploits asynchronizing accumulating and synchronizing update, which reduces the cost of communication between domains.
3. The paper proposes to use Wasserstein distance to select the optimal domain as the source domain for the target domain. 
4. The experimental results show that the proposed method outperforms baselines.

Cons:
1. The asynchronizing accumulating and synchronizing update is not novel. It has been used in other communities such as reinforcement learning.

Overall, the paper is good and it is technically sound. The contribution is not significant to the community but providing a new perspective for domain adaptation. I vote for weak accept.

Thank Reviewer1 for reminding. I think the paper still has some novelty and the comments address my concerns. I do  not change my score. Also, I'm not unhappy if the paper is rejected. It is more like a borderline paper.
review_title: Official Blind Review #3

Meta-review: This paper proposes a solution to the decentralized privacy preserving domain adaptation problem. In other words, how to adapt to a target domain without explicit data access to other existing domains. In this scenario the authors propose MDDA which consists of both a collaborator selection algorithm based on minimal Wasserstein distance as well as a technique for adapting through sharing discriminator gradients across domains. 

The reviewers has split scores for this work with two recommending weak accept and two recommending weak reject. However, both reviewers who recommended weak accept explicitly mentioned that their recommendation was borderline (an option not available for ICLR 2020). The main issues raised by the reviewers was lack of algorithmic novelty and lack of comparison to prior privacy preserving work. The authors agreed that their goal was not to introduce a new domain adaptation algorithm, but rather to propose a generic solution to extend existing algorithms to the case of privacy preserving and decentralized DA.  The authors also provided extensive revisions in response to the reviewers comments. Though the reviewers were convinced on some points (like privacy preserving arguments), there still remained key outstanding issues that were significant enough to cause the reviewers not to update their recommendations. 

Therefore, this paper is not recommended for acceptance in its current form. We encourage the authors to build off the revisions completed during the rebuttal phase and any outstanding comments from the reviewers. 
Decision: Reject