Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper motivates a series of information based analysis and methods. The authors first propose an information theoretic analysis framework for out-of-distribution generalization problem. Some corresponding generalization bounds are brought forward. Three different kinds of regulators are further studied. The effectiveness of proposed methods are demonstrated through numerical experiments.
main_review: The writing of the main paper and appendix are clear.  The reviewer lists some concerns here.

1. Why the authors claim that VREx is a relaxation of sufficiency criterion? VREx requires the stability of joint distribution of $(y, z)$ to achieve a stable loss value across domains. 

2. The ablation about combination of different criterions is lacked, which would be an interesting direction.

3. How to numerically estimate the mutual information?
limitations_and_societal_impact: The reviewer doesn't foresee a limitation on societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper investigates distribution shift from an information-theoretic perspective. It outlines a theoretical framework for describing different sources of modeling error, including that due to distribution shift, in the presence of a latent representation. It uses the framework to categorise existing approaches to reducing distribution shift according to the type of regularization criteria they use. It explains the assumptions underlying the success of these approaches, and demonstrates empirically, on synthetic data, that when these assumptions fail to hold, performance drops. 
main_review: **Originality**

I’m not an expert in this area, but to the best of my knowledge the theoretical presentation is original, as is the demonstrated experiments. It appropriately cites the selected approaches, which are investigated experimentally. I would have liked to see references given in section 3.3 on the separation criteria, as in 3.1, 3.2 (this is given in related work, but good to have it here too in my opinion). 

**Quality**

The quality of the theory is very good. Ideally I would have liked to see the experiments section pushed further, in particular on the neural network approaches. Only one type of architecture for all methods was investigated, and some surprising results were found (VREx performing worse than expected on CMNIST and d-CMNIST; CDANN performing worse than expected on y-CMNIST), which don’t line up well with the theory presented. It’s not really possible with the current experiments to tell whether this is something fundamental about the method, or if it’s something about the architecture, optimizer or other hyperparameters. 

**Clarity**

The paper is well-written overall. 
- One minor suggestion is to move the related work section to either after the introduction, or just before the conclusion (perhaps the latter works better considering the context of the paper is helpful to understand the related work section). In its current position it breaks the flow between theory and experiments.
- I would also prefer to call the ‘OOD’ terms ‘test’ instead, as this is more standard, and perhaps more correct (just because a sample is not in the training set, doesn’t mean it is out of distribution in some fundamental way - although sometimes this is the case). 
- I found equation 9 and its preceding paragraph hard to follow, consider rewriting, and giving an explanation for the derivation of 9. 
- Line 141: doesn’t lambda also trade off aspects of (i), as well as between (ii) and (iii)?
-Line 171: suggest rewrite I(e;y|z) as I(y;e|z) for consistency with eq 11 - even though Mutual Info is symmetric, it will be less confusing to reader. 

**Significance**
The main limit on significance is the use of synthetic datasets - not just because this gives us limited experimental evidence about the effectiveness of methods for distribution shift, but because the theoretical terms would be challenging to estimate in real-world data (the paper acknowledges this shortcoming). Whether the presented analysis is actually useful would have to be shown in follow-up work.

limitations_and_societal_impact: Technical limitations are briefly commented on. No potential negative societal impact is discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors describe the problem of data shift from a information
theoretic perspective. They identify different sources of error and compare
the effect of different regularization objectives on creatively derived datasets
in a controlled setting which cannot be achieved in most normal datasets.
main_review: ## Pros

- the description of the problem of distributional shift as being a result of
  not truly IID sampling the real distribution is informative and intuitive
- The construction of the datasets is sound and captures complex interactions between different dataset selection biases.
- The authors are very thorough in their theoretical framework which makes a compelling argument for analyzing future OOD-robust models under this controlled setting.

# Cons

- The colors in figure three are too hard to distinguish from one another. The
legends are also too small for being able to make out what they are showing at
a glance.

- I find the naming of the criteria in sections 3.1, 3.2, and 3.3 to be quite
confusing. Why is 3.1 called "Independence Criterion" when actually all of the
sections 3.1, 3.2, 3.3 are encouraging some form of independence? I realize
that these are cited as coming from another source, but I still think they
should be clarified so that readers do not become confused.

- It would be good to include some qualitative samples from the CMNIST dataset
in order to gain more intuition about what is being said in the paper.

- Why are none of the models evaluated on cross-criteria? The models seem to be
chosen for the specific inductive biases which predispose them to do better on
certain criteria. If this is the case, then it would be very interesting to see
how they perform on the other criteria, in order to validate that the inductive
biases are correct.

- I think it would be interesting to analyze some popular convolutional models
such as resnets in the controlled setting to see how their performance breaks
down (like equation 3) and then verify if the trend continue on larger datasets
such as CIFAR-10/100 with an OOD dataset for testing. If the best model in the
controlled setting also performed the best in this experiment (on some other standard OOD metrics which can be measured), that would be
extremely strong validation of the main theoretical findings.

## Questions

- In Figure 1 it is not explicitly clear how there is sample bias in the data
$p(y | x_1)$. It seems as if there was actually no selection bias, but just
another feature was added. Is there something I am missing?

- The anonymous code repo in the appendix appears to be incomplete, as I cannot
see the actual training script anywhere in these files. Is there something I am
missing?

- Equation 48 in the appendix seems to show that the encoder is only a linear
function. Is there really no non-linearity in that function?

- Equation 48 says it was used to compute the latent representations for $z$ in
figure 3, but then in section E of the appendix, the authors state that the
encoder has 3 hidden layers. Which one is correct?

- What is the "bottleneck" criterion in figure 3 and how does this relate to
the criteria highlighted in sections 3.1 - 3.3?

# Minor

- L93 I find this confusing because the equations show "information loss" and
  then "latent error," but in the text L93 the authors refer to them in the
  opposite order and then on L94-95 they refer to the "former and latter." I
  think the former and latter should match the order of the terms in the
  equations in order to avoid confusion.

Overall, I am extremely positive on the paper and the main theoretical
motivations and findings. However, I feel there are further experiments which could be
done to further validate the findings (the last two bullet points of the **Cons**
review). I feel the organization and clarity of each section could also use improvement, as I had to read each section multiple times in order to grasp what was being said.
limitations_and_societal_impact: Yes, they have adequately addressed the limitations in the appendix. I am not aware of any negative societal impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper studies the distributional shift setting, in which test and training distribution are different.  The distribution shift is described in terms of mutual information of the data with a binary selection variable $t$ (train or test).
Several propositions describe relations between the distribution of test and train covariates and labels and a latent space.
Based on those observations, several regularization criteria are suggested.
The experiments compare various settings of distributional shifts and popular methods.
main_review: Strengths:
- Distributional Shift is an interesting and important problem.
- The information theory approach gives a novel and intuitive description of the problem. 
- The experiments test different settings of distributional shift and compare the performance of popular approaches.

*******************

Weakness:
- I think the paper lacks significant technical results. I may have missed some insights, but I think that the propositions don't lead to significant and quantifiable conclusions about the distributional shift problem.  
The methods tested in the experiments section are claimed to be a relaxation of the criteria derived from the framework. This can be a very interesting conclusion, but it is not studied thoroughly in the paper (Sect 5.1 doesn’t elaborate the description of the methods.)

- There is also no mention of the finite-sample nature of learning problems. The conclusion from Prop 1. about the maximum-likelihood solution seems unfounded if there is no mention of the data used for the estimation.

- The objective function in Sect 2.5 resembles Variational Autoencoders, but it is not discussed in the paper. 

- Section 2.3 -  Lines 105-106 - “Whenever the JSD is small then the latent OOD error is small” - this doesn’t seem to follow from (8) since the JSD is a lower bound.
 

*******************
Clarity:
The paper is generally well written.
However, the introduction is not informative enough about the contributions of the paper.
The different sections feel disjointed with no clear organizing structure.
For example, section 2 is titled “Framework and Problem Statement” but contains propositions. It is not clear where the main results of the paper start.

*******************
Minor comments: 
- Line 98 - the sentence is not clear.
- Line 191 - typo - “t” 
- It seems to be a problem with the PDF, Fig 3 is missing when viewing the file with Adobe Reader (but it is fine with other software).


limitations_and_societal_impact: The limitations are discussed in the main review. There is no potential negative societal impact.
review_title: null

Meta-review: The authors study out-of-distribution (OOD) generalisation using an information-theoretic framework, decomposing the contributions of concept and covariate shift to the overall error. They go on to show how the framework yields an upper bound on OOD error in the context of a (latent) representation of the input variables. Consistent with related results, this leads to a conclusion that to minimise OOD error necessitates losing some information which is predictive of the training data. The optimal tradeoff relies on data unavailable at training time and so the authors study 3 proxies for minimising concept shift in the representation: independence, sufficiency and separation. In an empirical evaluation, the authors study minimising these criteria in comparison with existing approaches with related objectives. 

Several reviewers appreciated the information-theoretic formulation of OOD error and the theoretical treatment of it. Reviewer S3bF raised concerns about the significance of the technical contribution but this is to some extent compensated by the insights gained by the change in framework compared to related works. Additionally, several comments were made about the clarity of the organisation of the paper. In particular, transitions between sections are often hard to follow (see e.g., Section 2.2. which starts without any lead-in). The authors would do well to guide the reader better through the sections. Nevertheless, given the reviewers appreciation for the work, and the arguably minor concerns that remain after the discussion phase, I believe that a re-organisation of the paper is sufficient.
Decision: Accept (Poster)