Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper describes dynamic residual adapters designed to adaptively account for latent domains, and weighted domain transfer. This framework injects adaptivity into networks, preventing them from overfitting to the largest domains in distributions, a failure mode of traditional models that are exposed in latent domain learning. The approach closes a large amount of the performance gap to domain-supervised solutions. 

This paper is well written. The motivation of this paper is clear and the proposed framework does break the limitation of the existing deep learning methods. Below I present some suggestions, which hopefully can help the authors improve their study:

The authors do not describe any processing they have done of the data. This should be clearly included in the methods section. More experimental details and insightful discussions should be provided. I suggest the authors repeat the benchmarking using a selection of datasets more similar to those used in current studies.
The benchmarking results are insufficiently described in the text and can only really be seen in the tables/figures. The authors should explain the results in more detail in the text and include their interpretation of the results.
Section 3. In this part, the authors introduce the algorithm. Since there are many formulas in this section, additional explanations on the learning procedure would help understand the proposed method.
review_title: This paper is well written. The motivation of this paper is clear and the proposed framework does break the limitation of the existing deep learning methods.

final_score: rating: 5: Marginally below acceptance threshold
review_content: ------ Update after discussion with authors ---------

I would like to thanks the author for their efforts by adding additional experiments, which surely enhances the significance of the proposed approach. Based on these, I increased my score to 5.

I have re-checked the final revised version, I think the current version still *requires proper organizations and justifications*. For example, the added experiments still talked about the accuracy, the in-depth analysis seems lacking. I think a substantial revision of the paper in terms of structure, idea presentation, and analysis is still needed. Based on this, my final score is 5.

-----------------------------------
Summary: 

This paper studied how to learn a neural network with multiple domains without knowing the exact domain label (by merging all the domains as a large domain). Then they proposed dynamic residual adapters and weighted domain transfer to address this issue. The empirical results showed its practical benefits.

------------------------------------------------------

Overall review 

Pros:

[1] This paper is well-motivated. I like the analyzed scenario and I think it can have a strong practical utility. 

[2] The high level of proposed ideas is technically sound. 

Cons:

[1] The submitted version seems to be a preliminary version with many missing and unclear elements.

[2] As an **empirical** paper, the experimental results are not sufficient for ICLR.

[3] Some technical details need better justifications and discussions.

Based on these, I recommend a rejection at this time but encourage a major revision for resubmission.

----------------------------------------------------

Detailed explanations

[1] Missing elements

[a] I am rather confused and unclear about the whole learning procedure. It seems the author used DRA in the residual module. However, the role of WDT is unclear. What is the global training loss in the proposed approach? WDT is a part of the loss or used for analyzing the problem?  I would like to see a pseudocode/protocol for the whole algorithm or a clear network structure for illustrating the idea in Sec 3.3-3.4.

[b] The mathematical notations defined in this paper are presented oddly (particularly in sec 3) for example:

[b1] Equation (1), $\alpha$ and $\theta$ are scalars or vectors? what is meaning for $|\alpha|\ll |\theta|$? I guess it is $\text{dim}(\alpha)$ but it makes me rather confused.

[b2] The same problem for eq(2) and $\epsilon$
 
[b3] In WDT, the same problem for $\delta$, $\delta_i$ and $\delta_j$

These confusions make it more difficult to understand the approach.

[2] The empirical results

The current empirical results only compare MLFN, which is not sufficient. 

I noticed the author claimed, “Note the goal here is not to compare to the performance of existing multi-domain approaches that Visual Decathlon was designed for, but to show that deep networks struggle with learning small latent domains when no domain annotations are provided.”

I agree with this opinion if the paper aims to only analyze this scenario (generally from a theoretical perspective).  These kinds of experiments are sufficient. 

By contrast, the current version aims at **proposing a new empirical approach for the real-world practice**, which is not sufficient. I would like to see a **strong practical result** either outperforming the recent baselines or applying in many real-world problems.

[3] Technical details

[a] I suggest not using the term “domain labels” since it can be confusing to label $y$ information in the unsupervised domain adaptation. I think “domain index” or “task index” are better choices.

[b] The visualization of $\delta$ sounds interesting but I can not understand the meaning. A better explanation is expected. 

[c] Fig (1),(2) why PCA visualization? Why not Tsne?

[d] The benefits of self-attention are unclear. More analysis (not numerical accuracy) is expected.

--------------------------------------------

Suggestions 

I suggest a major revision on the proposed approach, empirical results, and more analysis (not accuracy) on the benefits of the idea.

review_title: Review #1

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
The authors propose a method for latent domain learning, where input data come from different domains and the domain labels are unknown. The proposed method consists of two parts: dynamic residual adapter and weighted domain transfer. The dynamic residual adapter acts as a mixture of expert layer. And the weighted domain transfer which augments the dataset by interpolating between different input pairs. Empirical results show that when combined together, the proposed method perform better than training a regular model.

Pros:
1. Latent domain discovery is a very interesting topic. 
2. Empirical results show that the method brings improvement to minority domains.

Cons:
1. Maybe I missed something, but I don't there are new insights in the paper. The proposed dynamic residual adapter is just an instance of MoE [1] with adapters, which I think should be a baseline in the experiments.
2. "Section 3.4 Weighted Domain Transfer" is not well-motivated and very confusing. Here you want to interpolate between x_i and x_j. But why do you compute the difference between the input x_i and the feature \mu_i in equation 3? Are they comparable with each other? What is the goal that you want to achieve here?

Other comments:
1. I think the introduction describes the problem too much, leaving it no space to expand your idea and intuition. For example, you start describing your idea at the very last paragraph.
2. When creating the augmented examples, can you leverage the gate information that you produced from the MoE?
3. It will be more helpful to understand the DRA component if you can provide PCA over the original activations.


[1] OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER
review_title: interesting problem, but lack of new insights

final_score: rating: 6: Marginally above acceptance threshold
review_content: 1. Summarize what the paper claims to contribute. Be positive and generous.
The paper claims to contribute a new method *dynamic residual adapters (DRA)* coupled with *weighted domain transfer (WDT)* to tackle *latent domain learning.* The proposed method improves the model performance against small domain datasets without hurting the model performance against large domain datasets in two different settings:
  (1) multi-domain setting (10 different tasks with 10 different domain per task)
  (2) multi-style setting (1 task with 4 different styles)
Impressive empirical results! I especially enjoyed reading PCA graphs and its qualitative analyses.

2. List strong and weak points of the paper. Be as comprehensive as possible.
  (1) Strengths
    a. Exhaustive empirical analyses. (ablation tests and tests with varying K value)
    b. Qualitative analyses backed by graphs like PCA and example images. 
    c. Impressive accuracy improvements.
    d. Intuitive theoretical explanation. Cool idea to use a gate to make RA dynamic.
  (2) Weaknesses
    a. Skipped the math that yields the equation (3) in the section 3. Would be nice if the steps are attached as an Appendix.
    b. In Table 1, there is a data domain size metric: $\pi_{d}$ . Can you add how this is computed? Also, how small is the smallest data in sheer number of examples?
    c. Table 1 has RestNet56, but Table 2 doesn't. Why did you make this choice of experiment design?
    d. With PACS dataset, you have experimented with the model (k-means+RA). I didn't quite understand the model setup and the motivation. In my understanding, the model learned the latent domain labels via k-means. And, then, based on this pseudo domain labels, the model is fine-tuned with Residual Adapter applied. Did I understand the model correctly? What is the motivation of doing this? I am not sure if this is a fair comparison between the RA and the proposed DRA+WDT methods. It seems rather a comparison between DRA+WDT and K-Means.
    e. In Figure 3, the paper says "Middle: WDT exchange or different domains, sketch is particularly inactive." I had a bit of difficulty parsing what "inactive" means here because the "Middle" figure is about "WDT Exchange Strength" and because "sketch" has the largest strength. Can you explain what "inactive" here means?
    f. Based on Table 3, the positive effect of WDT is not strong. I wonder if WDT is necessary.
    g. It seems to me that the strength of the proposed method is much more evident in the second problem type (PACS) where the task is the same across different domains. In the first problem type (Visual Decathlon), the DRA+WDT's performance boost is not consistent across different domains. I see that DRA+WDT hurts the performance compared to the baseline ResNet26/56 on a few different domains, such as Daim., Gtsrb, Omn., and Svhn. $\pi_{d}$ values of the domains of PACS are greater than those of Visual Decathlon, excluding svhn. Why do you think that is? When should one use or not use DRA+WDT in order to avoid hurting the model performance?
    h. Minor formatting issues. See 6 below.

3. Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.
Accept because of the impressive performance of the proposed method against PACS (single task, multi domain) with clear visual analyses. Meantime, 2.(2).g requires more explanation to make the paper's claim stronger.

4. Provide supporting arguments for your recommendation.
See 2.(1) Strengths, 3, and 2.(2).g above.

5. Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment. 
See 2.(2) Weaknesses above.

6. Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.
  (1) The indentation of Table 5 seems to be inconsistent with the rest of the tables in the paper.
  (2) In Figure 3, the paper mentions $P_{k}$ without the denotation explained explicitly in any of the main body of the text. I had to re-read the paper to find a footnote 5 to finally understand what this denotation meant. It would be good to briefly explain this denotation in the same description of Figure 3.

review_title: Impressive results on PACS (single task, multi-domain) for latent domain learning with a gate

Meta-review: While all reviewers agree that the topic is interesting and the work has merit, several issues have been pointed out, especially by R1 and R3, that indicate that the work is not  ready for acceptance at this stage. the authors are strongly encouraged to continue to work on this topic, taking into account the feedback received.
Decision: Reject