Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper aims at identifying and quantifying transferability in learning algorithms. The paper proposes a new upper bound on the target error. The proposed algorithm is an adversarial optimisation problem that empirically proved to be effective. 
main_review: The paper tackles an interesting and important problem. However, the main issue I have with the paper is that it is hard to read and barely provides intuitive explanations on the results. It is also a very notation-heavy paper, but they seem to be used without much introduction and discussion. The paper could significantly benefit from a section on problem setup and notations. For instance, before section 4, z is used but it is introduced in Section 4. Also, S_i is not introduced, I am assuming it is one of the source domains.  

In addition, the main contribution seems to be to draw connection between IPM measures and the transferability that is already known in the literature. The results section is also poorly presented and is not clear what the experimental setup is. It is also not better explained in the supplements.

I have some concerns about the problem set up and the overclaim of the paper. I understand though that this paper follows similar ones in the domain transfer literature, but I believe for more modern deep learning settings it is not sufficient. It would be interesting to see how these bounds be different if the i.i.d assumption is removed and a more realistic setting is used. In fact, I believe the main obstacle in using such bounds is that the main question in transferability is how to address these confounders. For instance, if we had a dataset of cats vs dogs where all the cats have a blue background and all the dogs a red one, we expect with the simplicity bias in neural networks, the model focuses on the background alone. Therefore the transferability to another problem with colour as the main discriminative signal is intuitively easier than to a cat vs dog dataset where images are taken in the wilds. 

With the rise of self-supervised and pre-training, I believe this paper could be more impactful if it included discussions on those lines of work. They aim at learning the invariances and that is how they achieve better generalisation.

limitations_and_societal_impact: All in all, I believe the technical quality of the paper may be good but poorly presented.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper defines a notion of "transferability" between features from different domains, which is different from common distribution discrepancy measures such as total variation and Wasserstein distance. This measure of transferability is measured with labeled samples from both domains, and provides a bound on the target error. The empirically test transferability of the features learned by current domain generalization algorithms, and some don't do that well in this transferability measure. They propose an algorithm based on optimizing the measure of transferability across training domains and test it on some domain generalization datasets, including a satellite dataset.
main_review: Quality

The general structure of the work seems to be to find a metric (transferability) that plugs into a generalization bound, then convert this into an algorithm by optimizing the metric. This paper gives some interesting results about source vs target accuracy when subject to parameter space pertubations, but these comparisons are somewhat better suited for their method since they optimize something related to this directly. They give a comprehensive theoretical analysis including finite sample results. The main concerns are in the formulation and the empirical results. For formulation, it's not immediately clear what this method leverages better than other domain generalization methods since it also only uses the given training domain information for enforcing invariance - is the bound tighter than others, etc.? More intuition on this could help. For empirical results, from Figure 3 it seems that the SD baseline is comparable to the proposed algorithm in target accuracy across all the datasets. Also, error bars aren't given in the plots and the results could be within noise.

- As the authors mention in line 258, the measure of transferability depends on labeled samples from the target domain. They circumvent this in the algorithm by optimizing transferability between a number of source domains. However, there's a bit of a conceptual gap here, since it is straightforward to say that we optimize a bound on the target accuracy of a particular target distribution that we measure transferability on, but it's less clear how optimizing for transferability on a number of source domains will generalize to a new target.
- In line 242 and equation 13, $\Gamma$, the $\delta_S$-minimal set of $\epsilon_S$, is replaced by a $\delta$-ball in parameter space. This relies on an assumption on the neural network to be Lipschitz in the parameter space. However, for wide neural networks, it seems that small perturbations in parameter space can cause very large changes. This could make the bound in equation 13 very loose. It's also unclear that comparisons wrt to the $\delta$-ball in parameter space like in Figure 3 are valid due to the unknown (and possibly very small) range of valid $\delta$. The overall finding that target accuracy decreases much faster with parameter space perturbations is interesting, however.
- In section 2.1, the paper states that the defined metric is different from IPMs since it relies on an underlying function class, but this is exactly what the "neural network distance" IPMs used in theory of GANs do as well [1].

- Minor: in the self-driving example in the intro, night/day shouldn't be a shift where we should be applying domain generalization techniques, since it's so easy to collect data in both the night and day. Time shift might be more appropriate here (you can't collect data from the future).

Significance

Domain generalization methods generally don't seem to work much better than ERM, according to some empirical works, so advancements in this field are necessary. The overall approach in this paper is nice and comes from a principled point of view, however the empirical results are inconclusive whether this paper's method improves upon the other baselines.

Originality

The final algorithm seems related, though different, to invariance based methods such as IRM or group DRO. The principle seems to be to make the errors on every training domain to be the same within a ball in the parameter space (learn parameter-stable networks). The ball in parameter space seems to be a limitation of the technique / inability to characterize the $\delta$-minimal set of hypothesis, however, and not necessarily fundamental. The measure of transferability is nice, although it requires labeled data in both source and target so it isn't necessarily surprising (whereas using only unlabeled target data to estimate transferability would be tougher). 


Clarity

The paper could be structured more hierarchically - it dives into math as part of the essential explanation. I think the clarity could benefit from giving a high level overview and then going into the details. 
- Fig 1: from the description of transferability, it's unclear why we're considering $\delta$ perturbations in the parameter space. This is only introduced in Section 4.
- Def 2 is named quantifiable transfer measures, but I believe only the third one is computable since it's the only one that doesn't require knowing optimal errors. Whether something should be considered quantifiable if it is not computable is unclear.
- Since there are 3 transfer measures defined, it's a bit hard to keep track of which one is being used at the moment, and what the purpose of each of the measures are. Perhaps the paper could state which one will be the main one to work with, and then using these propositions we can make it computable. 
- In Fig 3, the brown color is used for many different algorithms, consider deduplicating the color.


[1] Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, Yi Zhang. Generalization and Equilibrium in Generative Adversarial Nets (GANs), 2017.
limitations_and_societal_impact: 
The authors mention that their algorithm requires a large number of inner maximization steps, increasing training time, and that moderate hyperparameter tuning is needed for additional hyperparameters. In terms of broader impact, the authors warn that just because an algorithm passes their transferability check, it does not guarantee good domain generalization performance.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper aims to provide a formulation of transferability. The main intuition is that any near-optimal source classifier should be also near-optimal on the target domain. So the authors quantify transferability by measuring the relation between near-optimal classifiers in source and target domains and derive genralization bounds based on this. In addition, new algorithm is proposed based on the intuition.
main_review: Originality: The problem of quantifying transferability is not well studied and the reviewer agrees that proposing a formulation of transferability or invariance is important and novel (though there are many other concurrent papers doing this, e.g., https://arxiv.org/abs/2008.01883, https://arxiv.org/abs/2106.04496, which might be better to be included later)

Quality: The basic formulation of transferability in this paper is like a relaxation of the IRM objective with quantifiable surrogates. The relaxation is reasonable though the quantitative part is similar to the DANN-style discrepancy. Given the improvements on the empirical performance compared to DANN and other distributional matching baselines, the reviewer tends to vote for accept.

Clarity: The writing of this paper is clear.

Significance: The formulation and algorithm proposed in this paper are similar to the previous work in the form but bring new thoughts and improvements to the area.
limitations_and_societal_impact: Not applicable.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Authors introduced the notion of transferability between pairs of domains (i.e. joint distributions over input x output spaces) along with a discrepancy measure, which can be used to assess how dissimilar are two data sources relative to a given class of predictors. Such a quantity may be useful in either assessing how well is a trained model likely to perform on new data, or as a training signal so that more preference is given to predictors able to generalize out of the training data distribution. The main strength of this work in my opinion is the fact that the proposed transferability measure accounts for the joint distributions over data and labels. However, while I appreciate the depth of the discussion and results in terms of the properties of the proposal, I believe a key missing discussion is the effect that commonly used sets of assumptions have in the proposed transferability measure. Moreover, the evaluation lacks in evidencing the benefits of the proposal relative to current strategies. Please refer to the following for further details.
main_review: Strengths:

+The proposed discrepancy measure accounts for the joint distribution over data and labels, which seems to remove the need for assumptions such as covariate shift, required in common alternatives that only account for the data marginal distribution.

+Authors present a comprehensive set of results and in-depth discussion of the properties of the proposed measure.

+An interesting evaluation is carried out where transferability is estimated for features obtained under different settings appearing in recent literature.

+Algorithm 2 requires a single extra classifier to estimate the worst-case pairwise discrepancy (h’). Past similar approaches use N or even N^2 extra classifiers, where N is the number of training domains.

Weaknesses/suggestions:

-The write-up lacks in discussing the effects of different kinds of distribution shifts in the transferability measures. I would expect to see a discussion similar to [1] where the effect of, for instance, conditional shifts would be analyzed. It seems severe conditional shift, c.f. example in figure 1 in [1], would make the transferability measure explode, which would result in no advantage over more standard discrepancy measures over data marginals only such as the H-divergence. Please clarify whether that’s indeed the case.

-Regarding specifically the empirical evaluation, it lacks in evidencing the effectiveness of the proposed measure since it focuses on analyzing the transferability of features obtained under different algorithms. It would be important to see evidence showing that the estimation errors are not significantly affecting results in practice. As a suggestion, it seems all evaluations are carried out in datasets where covariate shift can be assumed (i.e. only data marginals shift). In that case, we would expect the proposed transferability to correlate with, for example, the H-divergence. Verifying that would serve as evidence that the measure and the algorithm used to implement it actually yield informative results, and are not relevantly affected by, for example, sub-optimality of gradient ascent.

-Another missing piece of evidence in my opinion is in how indicative of generalization the proposed transferability measure is. From a practical perspective, it would be relevant to see evidence that lower transferability implies better generalization.

-Finally, the claim that the proposed transferrability measure is helpful in allowing the analysis to shift from the marginals to the joint distributions should be supported by empirical evidence as well. Are there cases where more standard notions such as the H-divergence wouldn't work and the proposal would?

Minor suggestions:

Since transferability is a property of a pair of domains and a hypothesis class, perhaps that should be indicated in the notation; i.e. (\delta_S, \delta_T, H)-transferable should be used instead of (\delta_S, \delta_T)-transferable.

[1] Zhao H, Des Combes RT, Zhang K, Gordon G. On learning invariant representations for domain adaptation. International Conference on Machine Learning 2019 May 24 (pp. 7523-7532). PMLR.

Post rebuttal edit:

I went through the authors' responses which clarified the effect that conditional shift would have on the proposed measure. I decided to update my score to 'accept'. I do like the idea since it's the first domain-divergence measure that I'm aware of that accounts for the joint distribution over data and labels, as opposed to only one of the marginal distributions.

One concern that remains to me is the coverage of the evaluation, which I think is very limited. While the main strength of the approach, in theory, is the fact that it accounts for the joint distributions, all evaluations are carried out on cases where only the data marginals shift. Moreover, only the proposed transferability is evaluated, and it's not compared with more standard divergence measures such as the H-divergence. If the proposal yields meaningful domain divergence results, we would expect a strong correlation between their proposal and the standard measures in the considered evaluations, and that would give confidence that the proposal works in practical settings. For example, on PACS, pairwise H-divergences could be compared with the proposed transferability, and ideally, they would correlate more or less strongly depending on the label marginals shift.

In summary, I think the idea is very interesting, novel, and solves an important issue, the theoretical results are strong and insightful, but the evaluation is limited. All in all, I think the merits of the paper outweigh its flaws.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper focuses on transferability in context of OOD generalization. The authors define the term, point out connection and differences with current discrepancy methods and then prove that this measure can be transferability can be estimated with enough samples and give a new upper bound for the target error based on our transferability. Empirically, they evaluate the transferability of the feature embeddings learned by existing algorithms for domain generalization. They also propose a new method for learning transferable features and test it over some benchmarks. The idea is interesting and the theoretical investigations are worthwhile. The authors addressed the questions and concerns raised by reviewers and I believe their responses are satisfactory. I request the authors to update the paper as promised in the discussion period and add the requested information and clarifications as well as comparison with the related works raised by the reviewers. 
Decision: Accept (Poster)