Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies representation learning under covariate shift. Under the IDG setting and the proposed assumptions, the paper gives a so-called variational characterization of the optimal representation. This characterization shows that the optimal representation should remain discriminative while has the same support across domains. It is argued that without any target information, no representation can do uniformly well over constant representation, thus supporting the necessity of target knowledge. The paper provides practical objectives of the proposed variational characterization by self-supervised learning using domain-covering augmentations. The proposed representation learning scheme is tested on several datasets.
main_review: Novelty of the contribution: It seems to me that the main contribution of the paper is to provide informative formulations that characterize the optimal representation, which provide guidance on designing a representation learning scheme under covariate shifts. This is a potentially important result, though I am not sure how much this distinguishes from the literature. From the related work discussed in Section 5, it has been known that it is sufficient to match the support across domains. The authors emphasize that the condition they proposed is also necessary, but how non-trivial is the necessity? Intuitively, if the representation of the source and target could have different support, then it is hard to guarantee good performance in the target. Perhaps the main contribution here is to make this intuition concrete, or perhaps crafting the minimax optimality in IDG? The authors might need to clarify and provide a more convincing argument to support their novelty. 

Requiring access to lots of information: As discussed by the authors, one limitation of their framework is to assume access to an overly demanding amount of information that seems not practical. Would a more meaningful framework be to look at representation trained only on the source domain (as in the setting of Section 6.2 which is more practical) and some degree of information from other domains, and consider the characterization of optimality under this setting. In this latter setting, is there a way to generalize Theorem 1 and if so how much would the characterization change? (I understand a full answer may be difficult, but at least some indications to boost my confidence would be helpful; see also the next point that has a similar flavor)

Assumption on the domain-covering augmenter: Proposition 2 relies on the existence of a domain-covering augmenter. It is argued that image-text augmentations are nearly domain-covering, but it seems hard to expect that they are strictly domain-covering. So there appears a gap in the theory: The theory requires a strict domain-covering augmenter, while in practice this can only be nearly satisfied. Nevertheless, I think the proposition gives insights how the augmenter could be useful..

Numerical results: It seems that the benefit of domain bottlenecks is not very significant. In Table 1, sometimes the improvement of CLIP S/L + CAD over CLIP S/L is not significant enough, e.g., 94.7\pm 0.4 of CLIP L+CAD vs 93.7\pm 0.8 given in the PACS column. The only significant improvement is seen in the DomainNet column where the improvement is about 1%. 

A few minor issues: 
Page 1 the first summary point: it is said that the paper provides “all” objectives that have the correct optima. Why couldn’t there exist other objectives not given in this paper?
Page 3, the paragraph after Thm 1: it seems that the cross references to Thm 2 should be corrected to refer to Thm 1 (and similarly for other instances).
Page 3, some minor typos in the paragraph after Proposition 1: e.g., “to to”, “worst IDG”.
Table 1: it would be better to explain the level of the confidence interval (CI) in each entry, e.g., are they 95% CIs?
summary_of_the_review: The paper provides insights on the optimal representation under covariate shifts, which is a potentially important problem. I think the authors did concrete work to develop the theory and set up the numerical experiments, though there are also limitations discussed in the main review.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work focuses on learning representations for domain generalization. It proposes to minimize the idealized domain generalization risk (IDG risk; defined on page 2). Under several assumptions (importantly, domain covering augmentations), the IDG risk objective can be altered (Thm.1 -> Prop.2 -> Eq.(6)) to a more practical one. With additional variational techniques (Sec.4.2), the proposed algorithms approximately minimize conditional entropy and domain bottleneck in (6). Experiments on standard benchmarks show that the proposed method can out-perform SOTA alternatives. 
main_review: Strength

- Theoretical analysis on the conditions for idealized domain generalization

Weaknesses

- Writing is poor
- Empirical improvements come from external datasets rather than the method itself

The paper focuses on learning representations for domain generalization. The main idea is to learn a representation that matches the support across domains while maintaining discriminative information (Thm.1). Starting from idealized domain generalization risk (page 2), the paper goes through a long path of derivations and assumptions to derive practical objectives. The theoretical understanding is interesting, but the exposition is poor and empirical results may not be as promising as it seems.

1. Technicalities (and required clarifications for some parts)

- It is surprising to see that the current version does not discuss hypothesis class in question. For example, after (2), what's the candidate hypothesis set for h (argmin_h) when defining the set of source risk minimizers? The hypothesis class usually plays a central role in domain adaptation (e.g., Cortes et al. (2011; 2019)). Additionally, a motivating example would be great for introducing IDG risk and why we should minimize it.
- Page 3 mentioned different scenarios of Assumption 1 (GCS) for different losses like log-loss, 0-1 loss and MSE. Can you elaborate or provide some derivations?
- After Thm.1, it is claimed that "risk minimization using a single domain is as good as performing risk minimization on all domains". However, both R_{IDG} and R involve expectations over domains or pairs of domains. None is an optimization over a single domain.
- It is not clear how "the key requirement" can be satisfied by the given augmentation. Specifically, how to design such augmentation, other than CLIP?

Ref:

- Cortes, C. and Mohri, M. Domain adaptation in regression. In International Conference on Algorithmic Learning Theory, 2011.
- Cortes, C., Mohri, M., and Medina, A. M. Adaptation based on generalized discrepancy. The Journal of Machine Learning Research, 2019.

2. The writing is poor. Some parts of the paper are challenging to follow

- Notations are not consistent. Sometimes d_t (after Eq.(1)), sometimes D_t (Eq.(2), Theorem 1 etc)
- This paper fails to properly introduce the necessary background on CLIP.
- Exposition in Sec.4.1 is poor. It would be much better if the paper can provide a running example about A, X and Z.
    - A is said to be a RV. If I understand it correctly, an augmentation A can take value in, for example, {image rotation, cropping, mirroring} as shown in Fig.2a. Then p_{A|x} is the distribution of the above three augmentations conditioned on x. It is not clear why p_{A|x}=p_{A|x'} implies they have the same Bayes predictions. If two images are destined to be mirrored with probability one, then they must have the same prediction? If A is supposed to be the actual mirror image of x, then it is more than an arbitrary RV and should be specified more precisely. It is unclear where this "key requirement" comes from.
    - What is the task/classes in Fig.2? Is it classifying "floppy ear" versus "pointed-eared"? If so, the augmentations in Fig.2b look arbitrary and do not make sense.
    - "Enforcing the support constraint for augmented data (X, A)", but the constraint of Eq.(5) does not involve the augmentation A.
- Sec.4.2
    - What is the definition of the domain bottleneck B[Z, D]? And why it is related to the support constraint in Eq.(5)?
    - What is \mathcal{A} after Eq.(7)?

3. Experiments

- What is the "fixed domain" for SingleDom? And why it is DC?
- For approximate DC, what's the result if SingleDom is used instead of Supervised?
- The improvement in Table 1 seems to be mainly due to the fact that CLIP is pre-trained using a much larger dataset. Thus the proposed methods are only marginally better than vanilla CLIP if I understand it correctly. The results on TerraIncognita can justify this. To add some discussion, how can we ensure domain-covering in practice when even CLIP may not have sufficient coverage?
summary_of_the_review: In summary, the paper theoretically analyzes the conditions under which domain generalization would be possible. However, the writing is not clear at places and the empirical results are not very convincing.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: At a high level, this work considers the problem of designing machine learning systems that generalize well even when the "domain" (i.e., the data-generating process) under which the system is tested (the "target") does not match that under which it was trained (the "source"). More specifically, the authors look to provide an answer to the question of what makes a "good" representation or encoding (a stochastic transformation of the original features), from the perspective of achieving a small risk (expected loss) on the target domain, assuming the learner only has data (and thus representations) from the source domain.

To this question, they provide one answer by showing that in a generalized covariance shift scenario, a natural notion of representation optimality (their "IDG optimality") is only possible if a representation minimizes the (best) risk on the average target distribution, while ensuring that the support of the representations is constant across domains. This is obviously a strong requirement, and they show that one cannot expect to satisfy it without domain knowledge going beyond the source.

They also show that there is some hope for special cases of domains in which we have augmentations that preserve label information and "cover" the support of the input distribution across all domains. In such a case, the authors show that their strong optimality requirement can be satisfied by designing the encoder to maximize the mutual information with the augmented data. Relaxing this objective leads to objectives that are more practical, and the authors complement their theoretical analysis with a rather in-depth set of empirical tests that evaluate the efficacy of their methodology for representation learning.
main_review: *Strong point(s) of the paper*

- The paper contributes some lucid new theoretical insights to the problem of encoder design that is conducive to trained models which are domain robust, and does an effective job of linking this basic theory up with practical training procedures.

- The writing is solid. High-level theoretical points are quite clear in the main paper (details in the appendix), the relation to existing work is described, and the experimental results are distilled into new questions/insights which are easily parsed by the reader.

*Weak point(s) of the paper*

- The theoretical results are stimulating, but they are quite idealized and leave a substantial gap between what we can know and check in practice, and what we want to know formally. For example, while the notion of a domain-covering augmentation is very simple mathematically, but one expects that evaluating whether we have something close to such an augmentation in practice becomes quite challenging when tasks are difficult to describe in words or in simple visual terms.

*Questions*

- Most of the notation is immaculate, but I would like to confirm about  $\\mathrm{R}_{h}^{d}[Y \\,|\\, Z]$. Is the assumption that $Z \\sim p\_{Z \\,|\\, X}$ and the expectation is now with respect to $Y$, $X$, and the randomness in sampling $Z$? (instead of just $Y$ and $X$)
summary_of_the_review: The problem is fundamental to machine learning, the theory developed by the authors is stimulating and to the best of my knowledge new, the empirical analysis is informative, and the writing throughout is very strong. I vote to accept.
review_title: null

Meta-review: The paper aims at characterizing conditions for optimal representations required for the domain generalization problem under covariate shift. Under the Idealized Domain Generalization (IDG), the paper provides a variational characterization of the optimal representation and shows a number of intriguing results: (i) optimal representation should remain discriminative  across domains, (ii)  the representation’s marginal support needs to be the same across source and target. (ii) It is also shown that without any target information, no representation can do uniformly well over constant representation, thus supporting the necessity of target knowledge. Finally, the paper provides practical objectives of the proposed variational characterization by self-supervised learning using data-augmentation with experimental results. 

The reviewers had raised a number of concerns, many of which were alleviated through the responses provided by the authors. All in all, in the discussion period, the reviewers indicated the novelty of the results and their importance in learning good representations for the domain generalization problem. The reviewers still had reservations about the following points, and I strongly recommend to the authors to address these points in the revised version (please see the reviews for more details): 

(i) For the experiments, it is clear that there is a noticeable gap between the derived theory and the experimental results. It was argued that alt-text augmentation of CLIP is one of the practical choices for (approximate) domain-agnostic augmentation, but it is difficult to verify this.  

(ii) The empirical gain for the proposed method over CLIP is not very significant. As shown in Table 1 (or the complete results in Table 4), the performance of the proposed method is tightly related to the original CLIP method. In the case where the pre-trained representations of CLIP fail to cover the target set (see the TerraIncognita column in the table), the proposed method can be much worse than other alternatives. Thus I'm worried about the usefulness of the proposed method in practice.

(iii) One of the reviewers had asked about the importance of the necessity condition and its implications (e.g. in practice). Please, make sure to address this in the final version. 

Also, there has been several recent works on learning disentangled representations for the domain generalization problem, using e.g. weakly-supervised approaches (Matsuura et al, '20), or model-based approaches (Robey et al, '21). It would be interesting to see how the results/ideas in this paper would connect to/improve those settings.
Decision: Accept (Poster)