Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary
The paper proposes a generalization of the invariant risk minimization objective to the continual learning setting, where environments are observed sequentially. An ADMM strategy for the solution of the resulting bilevel problem is proposed. In extensive experiments on smaller MNIST-like datasets, the method is shown to perform favorable to recent approaches for continual learning.

## Explanation of Rating
The main strength of the paper is that it attempts to tackle an important and open problem using a reasonably principled approach. The work is well-written, and the methods performs well in practice and is evaluated against a large set of competitors. I lean towards accepting this submission. The weaknesses of the paper are the scalability of the approach (see comment #1) and the lack of theoretical guarantees for the ad hoc ADMM scheme (see comment #2). 

## Detailed Comments 
1. An evaluation of the method for larger architectures (e.g. a convolutional network) would make the approach more convincing. At the moment, I get the impression that there are issues with the scalability of the approach to larger data sets and models. 
2. The ADMM formulation seems rather ad-hoc, and since the loss is not convex, it is unclear whether the scheme is numerically stable / convergent. The statement about convergence rates in the strongly convex / convex setting are a bit puzzling, as they do not apply to the problem at hand. What is the main advantage of the ADMM approach over, say, solving the bilevel problem using implicit differentiation? 





review_title: Interesting paper, needs more polishing

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper extends the idea of invariant risk minimization (IRM) initially introduced by Arjovsky et al. (2019) to the setting of continual learning in which environments are observed sequentially rather than concurrently. This extension is implemented under a variational Bayesian and bilevel framework and the optimization is solved using a variant of the alternating direction method of multiplier (ADMM). The authors demonstrate the superiority of the proposed methods on variants of Colored MNIST. 

Pros:
+ The proposed method is a natural and practical extension of IRM and IRMG, which I believe is more applicable to many real-world scenarios. 
+ The resulting bilevel problem can be efficiently addressed using the alternating direction method of multipliers.

Concerns:
- My main concern is about the generalization theory in this continual setting. In both IRM and IRMG, the authors provide the conditions under which the OOD generalization can be guaranteed, as stated in Assumption 8 and Theorem 9 of Arjovsky et al. (2019) and in Assumption 2 and Theorem 2 of Ahuja et al. (2020). I did not see any similar theoretical guarantees in this paper. Without them, it is hard to judge whether or not  the proposed method really generalizes out-of-distribution in a sequential manner. 
- I do not think the contents in Definition 1&2 are just definitions. Instead, they should be treated as propositions, or lemma or so on, because they should be supported by proofs which seem not to be provided yet. 
- The section of Continual IRM by Approximate Bayesian Inference is not friendly with readers unfamiliar with ADMM, which makes that part hard to go through. 
- No section index

review_title: An interesting extension of IRM to the continual learning setting

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
In this work, the authors consider the problem of continual learning with distribution shifts.  The work extends the recent work invariant risk minimization (IRM) from Arjovsky et al. to a continual learning setup. IRM was designed as an offline learning framework. In this work, the authors consider the setting where the different domains arrive sequentially. The authors propose a Bayesian extension of the IRM framework that allows sequential updates as the environments arrive. They provide a justification for how the KL divergence term helps in shrinking the support continually to arrive at an approximately invariant support. Several experiments were carried out on colored MNIST and its variants to show how the proposed scheme is better than existing continual learning methods and existing IRM frameworks. 

Pros: 
I like the problem the authors consider. It is indeed important to understand how to extend IRM type frameworks in settings such as continual learning. I also liked the ADMM based approach taken by the authors as this approach gives another way of learning IRM based predictors in addition to existing approaches. 

Cons:

There are several problems with the paper. I discuss these problems below in different subsections. If these major issues are addressed in the rebuttal, I would be open to changing my score. 

a) Information Projection (Theorem 3)
In the work, the authors show in Theorem 3 as to how when the model is trained sequentially the support of the distribution shrinks. 

Let us consider optimization in 2 time steps

t=1, q_{t} = min_{q_{t}\in P} KL (q_t||q_{t-1}) --> support of q_1 = P \cap support of q_0 

t=2, q_{t} = min_{q_{t}\in P} KL (q_t||q_{t-1}) --> support of q_2 = P \cap support of q_1 = P \cap P \cap support of q_0 = P \cap support of q_0 

It is not clear from just the KL term why support would strictly shrink. The reason why the authors approach works is because the first term based on the IRM loss allows to update and shrink the support and the KL divergence term ensures that there is no need to expand the existing support. Define the first term in the loss associated with IRMv1 as Q_{IRMV1}(q_t)

t=1, q_{t} = min_{q_{t}\in P} Q_{IRMv1}(q_t) + KL (q_t||q_{t-1}) --> support of q_1 \subset P \cap support of q_0 

t=2, q_{t} = min_{q_{t}\in P} Q_{IRMv1}(q_t)  + KL (q_t||q_{t-1}) --> support of q_2 \subset P \cap support of q_1 

It is the combination of the IRM term with KL term that ensures shrinkage and KL term on its own only ensures support does not expand. A proof or an illustration of what I state would have been nice as the current intuition from the authors is incomplete and to some extent a bit misleading.

b) Why penalize the IRM constraints as well?  
The authors arrived at a variational formulation in 9a and 9b with new objective functions defined in 10 a and 10b. It is unclear why it was chosen to impose KL penalty in the constraint (10b) as well. If not a theoretical justification, an experiment in the supplement illustrating why these choices were made would be nice. At least some good intuition needs to be provided. 

c) About the ADMM approach: 
I am happy to see the authors tried the ADMM approach. However, one thing that is unclear is it does not necessarily serve the purpose of solving the problem authors want. From the experiments it seems a continual extension of IRMv1 itself suffices. Is there any other reason why ADMM approach was used. Also, if ADMM approach was proposed as another way to solve standard                                                                                                               IRM in the offline setting, then some experiments explaining any advantage would be useful. At least some offline comparisons would make a better case for using ADMM. 

d) Why not empirically compare with Javed et al.?

As the authors correctly identify there is a recent work from Javed et al., which solves the same problem but makes more assumptions about the data.  I understand that the work uses one hot encoding of color etc. Despite that you can allow your model to also have access to the same data format and then see if your model has any more advantage to offer over and above what was done in Javed et al.

e) Comparisons with offline IRM based methods (IRMv1 Arjovsky et al., IRMG Ahuja et al.) do not seem to have been carried out in a fair manner.
The authors have not provided a very clear description of how IRM based methods were trained. It seems to me that the authors are running offline IRM based methods in the following manner.  They provide the data from the first environment and then keep the offline IRM based models fixed and not update them when the data from the next environment arrives. If they indeed update the offline IRM based methods, then there is no reason why the methods perform so poorly.  What was the reason for not updating these models? The reason I am suspecting is that everytime retraining the entire model on the entire data is computationally expensive. If that is the concern, then authors should have shown a comparison of run-times. The authors should have run experiments allowing all methods the same run-times and then shown that IRM based methods are still not able to perform better. 
For instance if the time to train continual IRM based model proposed by authors for the two environments is a total of 20 seconds. 
Suppose the total time to train a standard IRMv1 is say a total of 15 seconds for the first environment. There is still a five second window for the IRMv1 model to be updated.  If run-time comparisons are non-trivial, then at least there should be a budget on the total number of gradient computations that is fixed across methods. 


f) Compare with Creager et al. 

The reason IRM based offline methods did not perform well is because they were given data from one environment only.  Recently, in Creager et al., it was shown how one can learn to split the data to create new environments and then train an IRM on those environments.  The authors should compare with the Creager et al. as follows. When the data from first environment comes in you can use the approach introduced in the paper http://www.gatsby.ucl.ac.uk/~balaji/udl2020/accepted-papers/UDL2020-paper-045.pdf to learn environments (code available at https://github.com/ecreager/eiil). Therefore, by doing this you can split the data in the first environment into smaller environments and use IRM. This would lead to a lot of performance improvement for the offline IRM based methods.


g) Current experiments do not reflect the true potential of the method proposed by the authors: 

The current experiments do not do a good job of reflecting a continual learning setup. Simple modifications on existing IRM based methods can outperform the method proposed by the authors. However, I believe the method proposed by the authors is nice and much more experimentation is needed. I make some suggestions on how to improve the experiments. 

It seems that in the current experiments after 2 environments there is no more gain from using any method, in fact more environments seem to hurt. I would encourage the authors to do an experiment where adding more environments actually removes spurious correlations more and more and helps.  To this end, there are three suggestions I would like to make:

i) First let us understand is why is two environment sufficing in colored MNIST dataset.  The reason is that the dimension of the spurious factor, i.e., color is small (only two colors). You can increase the number of colors, then more environments will be needed to decorrelate. 

ii) If  adding colors and doing comparisons using raw images is hard, then you can try using the one hot encoded colored MNIST from Javed et al. and add more colors to it. 

iii) Another suggestion would be to try the regression experiments from Arjovsky. In Arjovsky, the authors use two environments only. However, the two environments differ a lot 0.2 and 2.0 variance. If you introduce a sequence of environments from 0.2 to 2.0, say 0.2, 0.4, 0.6, ...2.0., then it is likely that more environments will have some benefit.  

Quality: The solution proposed seems promising. There are problems with theoretical justification. Experiments need a lot of work as I stated above.  

Significance: The problem considered in the paper is interesting. The experiments carried out do not do a justice to the problem being considered. 

Clarity: The writing of the paper can be improved. The experiments need to be described much more clearly and use the space in the supplement to give all the details. Also, the steps of ADMM shoould have been better explained.  




References

Javed et al. "Learning causal models online"

Creager et al. "Environment Inference for Invariant Learning"


 





review_title: Paper proposes an extension of IRM for continual learning. The problem is interesting, the method proposed is natural and makes sense. Theoretical justifications are incomplete and a bit misleading. Experiments are below par and need a lot of work.

final_score: rating: 3: Clear rejection
review_content: # Post-discussion update 

The authors have significantly updated the paper during the discussion period. I have seen the changes, but they unfortunately do not substantiate the claim that the proposed methods are learning anything meaningful. 

In the new results in table 1, the train accuracy now is better than random, but the test accuracy on the unseen environments is worse than a model that makes uniformly random predictions. This implies that the proposed method is not learning to ignore the spurious features at all. It might seem from table 1 that C-VIRMv1 is performing well --- it has 46% accuracy on the test environment after all. However, C-VIRMv1 also has the worse performance on the train set (50% accuracy). In-fact, the test or train performance alone does not mean anything in this benchmark. The goal of the benchmark is to do well on the train set while also generalizing to an unseen environment. Doing well on test set by doing poorly on the training set is not progress. 

The new results in table 3 are equally troubling. The authors claim that the results in table 3 show that C-VIRM with EIIL is performing better than IRM with EIIL, but the data does not support the claim. Both IRM and C-VIRM are performing similarly; all the results are with-in error margins of each others, and the table can not be used to make any claims. 

I would encourage the authors to do a more systematic study of the proposed method. Investigate if the proposed methods are learning to ignore the spurious correlation at all (Table 1 suggests they are not) and only make claims that are supported by the data. In its current form, I cannot recommend this paper for acceptance. 

# Initial review

I'm going to keep the initial review short to point out a fundamental issue with the paper. Until this issue is addressed, I see little point in discussing the paper at length. Hopefully the authors can address the issue in the discussion period and I will update my review to include other aspects of the paper. 
## Issue No 1 ##
 The proposed solution is arguably not learning at all. Both MNIST and FashionMNIST, as used in this paper and earlier IRM papers, are binary classification benchmarks (Five classes have label 0 and remaining five have label 1). A completely randomly initialized neural network gets ~50% accuracy on a binary classification task. This is not just speculation, I have run experiments using a randomly initialized network on these benchmarks to confirm that they do indeed get ~50% +- 2% accuracy on both train and test set. Coincidently, both C-BVIRM and C-VIRMG also get ~50% accuracy on the train/test set. This would imply that the model learned by C-BVIRM and C-VIRMG is indistinguishable from a model that does no learning. Here are some other baselines that will do as well as C-BVIRM: 

1. Set learning rate of ERM to be infinitesimally small so that no learning happens in the given number of steps (set learning rate to $e^{-100}$ for instance). 

2. Make completely random changes in the weight instead of updating the weights using gradients. 

3. Make no changes to the network. 

Clearly 1, 2 and 3 are bad learning algorithms (the first and third are not learning at all whereas the second is making arbitrary updates), yet all three of them would perform as well as the two algorithms proposed in the paper. 

Does this mean the two methods proposed in the paper are bad? Not necessarily. The BIRM formulation of the problem seems technically correct and the authors are tackling and important and interesting problem. However, the poor performance of C-BVIRM does imply that the authors need to look at their experiment results in more detail and do some sanity checks. 

A sanity check for checking if C-BVIRM is learning anything at all is to test it on a benchmark for which both train and test environments have a constant $p_c$ (say 0.1). A good learner should get 90% accuracy on both train and test environment in such a benchmark (Both IRM and ERM would get 90% accuracy). My guess is that C-BVIRM and C-VIRMG would get 50% accuracy on train/test when trained using the same parameters as used to run the experiments in the paper.  

## Issue No 2 ##
Please see the private comment for the second issue. 

Until issue 1 is addressed, discussing the technical details of the paper wouldn't make a difference and I can not recommend an acceptance. 
review_title: The experimental results do not pass basic sanity checks

Meta-review: The authors address the problem of learning environment-invariant representations in the case where environments are observed sequentially.
This is done by using a variational Bayesian and bilevel framework.

The paper is borderline, with two reviewers (R2 and R3) favoring slightly acceptance and two reviewrs (R4 and R1) favoring rejection.

R4 points out that the current experiments do not do a good job of reflecting a continual learning setup and that simple modifications on existing IRM based methods could outperform the method proposed by the authors. The authors are encouraged to take into account the reviewer's
suggestions to improve the paper.

R1 argued initially that the proposed solution is not learning at all since it has errors very close to random guessing. While the authors have improved their method in the revision, the results are still close to random guessing, which questions the practical usefulness of the proposed approach. Also, in the revision, the authors managed to obtain better results when their method is combined with Environment Inference for Invariant Learning (EIIL), but these results are secondary and not the main part of the paper.

The authors should improve the work taking into account the reviewrs' comments.
Decision: Reject