Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Short summary: 

The authors propose a zero-shot domain adaptation method with the assumption of each domain having the same set of labels. They obtain finite sample complexity bounds that contain the number of samples(n) per class in the training, the number of seen(training) and unseen(test) tasks under regularity assumptions on the loss and mappings. They provide accuracy results for MNIST and MNIST with rotated digits. 

Method: 

First, a common linear mapping is used to obtain a common representation for tasks. Then each domain-specific classifer(linear) combines at each index as a rank K tensor. A tensor completion is applied at each index that would give classifier parameters for unseen data as well, that allows them to predict without any data from the target domain. They assume same number of samples and same input dimensions for each domain (the latter can directly be extended to different dimensions). For the experiments, they use LeNet and directly predicted decomposed tensors rather than training and applying tensor completion.

Contributions: 

The authors extend Tripuraneni et al. (2020) method and proofs from transfer learning to zero-shot case. They show a uniform convergence bound that has polynomial complexity with respect to the # tasks, # dimensions and rank. They derive excess risk on the order of n^-1/4. 

main_review: Strengths:

- In my opinion, it is a good contribution for zero shot domain adaptation analysis, it can have good potential future work on the theoretical side.
- The authors support their analysis with small experiments and by showing the stability of their results. They obtain a bound on the polynomial scale with exponential input dimensions so it is satisfactory.

Weaknesses:

- The contributions are a little limited. The method is very similar with Tripuraneni et al. (2020) if I am not missing anything, but with a condition that not using any test tasks in the training. Also the authors define uniform absolute difference in order to keep distribution same with the true distribution. 
- The total number of source and target domains need to be known beforehand, so it might not be directly used for a new target domain.
- The paper is not the first to do finite sample analysis in zero-shot domain, see [1]. You could rephrase your claim.
- The paper was well written(there are a couple of typos especially in the equations - such as in theorem 4.2, 3rd line should be $=$, not $\leq$.), however it made me go back and forth a lot(needs a bit organization). Its relation with previous work could be stated better, the analysis and method parts could be separated. Some of the definitions in section 4 can be carried to preliminary since they are borrowed from the previous work. 

Questions:

- How does the correlations between the labels/domains affect your bounds? How do you quantify the similarity between the domains other than through classifier parameters? 
- Did you test your algorithm on larger real data? You could try lower p and with not-very-different source and target domains. 
- The method adds sum of linear functionals to a neural network with a couple of weight norm constraints, so I would expect results other than 30 degree rotated MNIST(although there are convexity and linearity assumptions in the theoretical part). 
- In table 3, what did you choose as the value of K? How did you choose your experimental details, such as why 11 classes?

[1] Blitzer, John, Dean P. Foster, and Sham M. Kakade. "Zero-shot domain adaptation: A multi-view approach." Tech. Rep. TTI-TR-2009-1 (2009).
summary_of_the_review: In general, I am a little bit concerned about the significancy of the paper's contributions. The method and theory part shows a lot of similarity with Tripuraneni et al. (2020). However, bringing it and the analysis to zero shot domain adaptation is valuable in my opinion.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers the problem of zero-shot domain adaptation. The contributions are two-fold. The authors propose a novel domain adaptation technique and provide bounds on the prediction error, which they provide proofs for. In addition, they provide empirical validation of their proposed approach, on two sets of datasets: one derived from MNIST, the second corresponding to sensors.
main_review: Positive aspects:
- The error bounds presented appear correct, to the best of my knowledge.
- The authors provide some empirical evidence of the benefits of their approach, on two different datasets. The results appear good, demonstrating the interest of their approach.
- The paper is well written, clear.

Concerns:
- Concerning the empirical evaluation, I would note that conducting additional experiments on datasets more complex than MNIST-derived ones (and the sensing dataset) would be beneficial to show that the method works well in a more realistic setting. Is there another dataset that the authors could scale to, such as adapting the CelebA dataset to a similar task as the MNIST experiments?
- Could the authors provide more details on how the different hyper-parameters were determined?
summary_of_the_review: I have not found flaws in the proofs or reasoning of the paper, and find the theoretical contributions particularly interesting. While I feel the experimental part is less impressive, I feel that this paper deserves acceptance. As noted in my confidence score, I am not overconfident about this, and would recommend weighing my opinion accordingly.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a specific domain adaptation framework where a subset of all possible domains is are observed during training. Given sufficient training samples for the observed source domains, a common latent representation as well as a domain specific linear classifier is learned. Theoretical conditions under which the learned representations will be effective for unseen domains are provided. Experiments using two datasets are provided. 
main_review: Strengths: While the theoretical results are interesting, the underlying assumptions are quite restrictive and will not hold for many computer vision tasks.
Weaknesses; Results on MNIST and the fiber sensing dataset are vastly inadequate. There are more interesting datasets for evaluating domain generalization methods, such as WILDS. The WILDS leaderboard lists a few SOTA results.
summary_of_the_review: Unless I see results on more challenging datasets such as WILDS, I will remain unconvinced of the usefulness of this work. It is not just the number of source domains T that matter.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the problem of domain generalization in the setting where each domain is indexed on a multidimensional array. It assumes that only a limited number of domains is observed during training and presents a technique for learning a classifier with provable good generalization performance on the unseen domains. The method consists of learning a common latent feature representation for all domains and domain-specific linear functionals, which are obtained by imposing a low-rank constraint on the tensor built by aggregating the functionals of all domains. The authors present a thorough theoretical analysis of the algorithm, including an upper bound for the excess risk on the target domain.
main_review: The paper is very well written, with very few typos, and technically sound. Nonetheless, I think that some aspects should be further clarified. Specifically:

(-) Despite the gains in computational efficiency, what are the disadvantages, if any, of replacing the optimization problem in eqs. (2)-(3) by the surrogate in eq. (5)? Is it just a constraint on the space of admissible solutions?

(-) In the experiments with the MNIST dataset, the authors create linear classifiers on two network layers, namely fc2 and fc3. How are the predictions of the two layers combined to produce the final decision? Moreover, what would be the effect of building the classifiers only on top of the last layer (fc3)?

(-) In the same experiment (MNIST), what is the value of $K$? More importantly, how was it chosen and what is the effect of varying it?

My only concerns about the paper are:

i) The applicability of the method seems a bit limited. I don't see a very large class of practical applications where the domains can be represented by a vector of attributes and hence fit into the proposed framework.

ii) The experimental results are not totally convincing. The method should be compared with at least one SOTA algorithm for domain generalization, e.g. Matsuura and Harada (2020).


Refs: 

Matsuura, Toshihiko, and Tatsuya Harada. "Domain generalization using a mixture of multiple latent domains." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 07. 2020.
summary_of_the_review: Although I have some doubts about the applicability of the method to practical applications, I don't see this as a reason for rejection. Moreover, the paper definitely has significant novelty and is technically sound. I would like to see an improved experimental section to increase my score.
review_title: null

Meta-review: Thanks for your submission to ICLR.

This paper explores zero-shot adaptation from a theoretical perspective.  Three of the four reviewers are quite positive about the paper, particularly after the discussion phase.  One reviewer was more negative, citing a lack of compelling experiments and some possibly restrictive assumptions.

The authors responded to these concerns, as well as the concerns of the other reviewers.  One of the more positive reviewers increased their score from 6 to 8.  I did not hear from the negative reviewer, but my feeling is that I tend to agree with the authors that the focus of the paper is more on the theoretical side.  Moreover, the authors did add some additional results to the main paper, so I am of the opinion that the paper should indeed be accepted to the conference.

Even though this is paper is on the theoretical side, please do include as strong a set of empirical results as possible in the final version.  Also keep in mind the other suggestions from the reviewers when preparing the final manuscript.
Decision: Accept (Poster)