Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present the latent domain learning aproach to multi-domain learning without domain annotations by introducing sparse latent adaptation (SLA) for learning sparse domain gates without domain labels.
main_review: The paper is well written, easy to follow and the motivations and contributions are clear.  Multiple datasets are used for evaluation against multiple SOTA approaches.

SLA is referenced before defining it in 3.1

I am confused on how g_k is learned in Eq 4.  How does it learn domain related gating without domain labels?  Figure 5 shows that in PACS some domains seem to be learned (eg sketch) while others may not have been learned.  Could this suggest that sketch images required additional parameterization to increase performance due to task difficulty? It is mentioned that the goal is not to recover domain labels, but simply adding additional parameterization may be similarly beneficial in single domain data.  Have the authors tested whether the performance improvement extends to single domain settings?  The results are interesting and I feel the paper could benefit from a deeper analysis on what is being learned by the "domain" gates.  If they are not domain specific is this really multi-domain learning?. Also, how are the number of gates (K) chosen?

I would prefer a bit more examination on the results and what is being learned as well as comparisons to only learning the gating parameters and training the whole model together with the gates.
summary_of_the_review: The paper is well motivated, clear and well evaluated with strong results.  It could benefit from a deeper analysis on what is being learned, how that contributes to different data settings and if novel aspects of datasets can be learned from this approach (eg does the model identify previously missed domains).  Overall a good paper that could be better with more detail.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed latent domain learning for adaptation. Experiments on multiple benchmark datasets show improved result. Several visualizations also illustrate the effectiveness of the proposed approach.
main_review: Strength:
1. The proposed latent domain learning makes sense and technically sound to me. It also has great potential in real world applications.
2. Extensive experiments and sufficient analysis validated the approach empirically.
3. Writing is good and easy to follow.

Weakness:
1. It seems that the optimal value of hyperparameter K is different for different datasets. Is there any thorough methodology to pick a good value instead of using K from 2 to 5 or just 2 as experimented in the current draft?
2. For the results, it was averaged over 5 random initialization, what is the variance for each experiment? It is not sufficient enough to use the mean itself for comparison as there might be a very large variance that indicates the model is not robust.

summary_of_the_review: Based on the above analysis, I am leaning towards acceptance for now.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the problem of learning a classification model on data from multiple domains, when explicit domain assignment for each data point is not provided. 

To solve this problem, the paper proposes the data to be coming from 1 of K unknown (or latent) domains. A single 1x1 convolutional layer per latent domain is applied to feature maps in each residual layer of ResNet. A gating function (based on sparsemax) is proposed to decide which of the K convolutional layers will be applied. The entire network is learned jointly by minimizing the classification objective function.
main_review: **Strengths**

1. The paper is well-written, easy to follow and understand.
2. The paper claims to propose a new setting for learning from multiple domains, where domain labels are unknown. This is an important problem since labeling domains can be challenging and/or expensive in real-world applications. While I am not extremely familiar with the field, according to the paper, prior works assume the knowledge of domain associated with a data point.
3. The idea of using sparse gating mechanism to allow for multiple transformations makes sense since it is possible that transformations learned across various domains are useful for predicting image category.
4. Qualitative and quantitative experiments seem to corroborate that authors' proposed method is learning a sensible domain representation.

**Weaknesses**

1. The experimental setup used in the paper, as well as the baselines are not properly justified. I specifically don't understand
    - Why is the ResNet backbone kept frozen, and not fine-tuned along with rest of the parameters?
    - Why do the authors use same parameters across different methods/models? A fair way to compare across methods would be to perform cross-validation across range of hyper-parameters and report the performance on the test set.
2. It is a bit surprising to me that all the proposed multi-domain approaches match perform worse when compared to latent-domain? I'd imagine that a model having a domain information for each datapoint would set an upper bound on the performance when compared to methods with no domain information. Is the presence of domain information hurting for the model performance? 
3. Although the paper states that the reported numbers are averaged over five random initializations, the standard errors are not reported
4. I would like to hear authors' take on connection between self-supervised learning approaches, and given latent domain learning. The SSL methods learn image representations using self-similarity. In an extreme case where each image is its own domain, SSL might be suited to perform representation learning on such multi-domain data.


**Minor points**

- In Table 1, $\tilde{U}_{D+1}$ is undefined. I presume it is validation or test data from $D+1$ domain.
- In Section 3.1, using a linear shift in feature map of ResNets for mapping domains (to a "canonical domain"?) is not properly motivated. I assume this strategy has been used in prior works in language but it would be useful to motivate it in the context of domain adaptation problem.
- In Section 3.2, some of the symbols (eg. $\tau$, $q$) are introduced without proper definition. While I was able to understand them by going back and forth between current paper and referred paper, it would be helpful if the authors clarify the notations.
- Section 2.2 is a relatively minor detail given that the datasets used in the paper are not highly imbalanced and both observed accuracy and uniform accuracy are highly correlated to each other. Authors can consider moving this detail to the experiments section.


summary_of_the_review: Overall my opinion is that the paper explores an interesting problem of learning latent domains present in data, when no explicit domain information is provided. I feel the experiments performed in the paper can be improved a bit. I've scored the paper accordingly.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a new task called latent domain learning and proposes a baseline that extends an existing multi-domain learning method. The latent domain learning assumes that training data are sampled from different domains yet their domain labels are latent, and aims at learning models that generalize well to the domains. The proposed baseline deploys multiple parallel feature transform layers that are chosen on the fly through gating variables, with the hope that the gating variables learn to predict the latent domain label of input and choose feature transforms of the domain accordingly. This model demonstrates superior performance to existing multi-domain learning methods in the latent domain learning setting. 

I recognize that latent domain learning is an interesting problem with great potential, and has many applications such as learning using web-crawled images. However, the proposed method looks limited in terms of novelty and the quality of writing is below the standard.

main_review: [Strengths]

1. A new and interesting problem setting with practical values:
The new task introduced in this paper, i.e., latent domain learning, looks like a tweak of existing problems such as multi-domain learning at first glance, but is interesting and has great potential as it could be practically useful when learning and testing models with data from heterogeneous domains (e.g., web images crawled by predefined search keywords). I also agree that the definition of visual domains is often vague and contrived, thus believe the motivations of latent domain learning make sense. 

2. Strong performance:
The proposed method outperforms multi-domain learning methods in the latent domain learning setting (although such a result is to be expected, the gap seems not large enough, and some scores of previous work look abnormal).


[Weaknesses]

1. Weak quality of writing:
The manuscript is overall readable but sometimes hard to follow, probably due to its weird terminology and expressions. Also,
- U^tilde in Table 1 is not defined.
- It is unclear why Eq. (2) is called “uniform” accuracy.
- The description for Eq. (3) is overly complicated. 
- The operation denoted by stars is not defined appropriately, one may guess it stands for correlation though.

2. Limited novelty:
The proposed model LSA looks like an extension of the residual adapter (Rebuffi et al., 2018), dubbed RA in the paper. Both RA and LSA adopt and modify the well-known residual connection by adding domain-specific feature transforms. The main difference between them is that to choose appropriate feature transforms RA utilizes domain labels explicitly while LSA estimates the latent domain label of input through gating variables. The use of gating variables could be counted as a contribution, but they are not a new idea as they have been widely used in other fields of machine learning.
Also, I would note that some objections against RA in the paper look invalid and accordingly the contribution of this paper should be undervalued. As far as I understand the residual adaptation method (Rebuffi et al., 2018) aims at designing models that maximize parameter sharing across different domains; only a small number of parameters of their residual adaptation modules are learned in a domain-specific manner while the majority of their parameters are shared across domains. 

3. Experiments:
Compared to RA, the improvement by the proposed method does not look sufficiently large. Also, the records of RA are weird: I wonder why its score is degraded when using domain labels, and how it is trained without domain labels in the latent domain learning setting.

summary_of_the_review: The new problem setting, latent domain learning, looks interesting and practically useful. However, the baseline model is limited in terms of novelty, its performance is not impressive (decent though), and the paper misleads readers about the important previous work (i.e., RA). 
The paper has both pros and cons, but I would value novelty and correctness more, thus leaning towards rejection.
review_title: null

Meta-review: The problem setting studied in this paper is an extension of the problem setting of multi-domain learning, where domain information is missing in training. This is an interesting and practical problem setting. However, regarding technical novelty, the contributions are relatively limited. Specifically, first, the overall idea is an extension of an existing multi-domain learning method [Rebuffi et al. 2017] by replacing the domain indicators with learnable gates. Second, the idea of introducing learnable gates is borrowed from some previous works. Third, the sparse activation is also based on an existing work of sparsemax. Though the authors claim sparsemax or sparse activation was only used in the NLP domain, this does not increase the technical novelty by applying sparsemax to the CV domain. 

To be fair, the combination of the aforementioned techniques to solve the so-called latent domain learning problem looks reasonable, while the technical contribution is not significant. Overall, I feel slightly positive about this work and recommend a weak acceptance (it could be considered for publication only if there is space).
Decision: Accept (Poster)