Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The work adopts a continual lifelong learning paradigm for causal effect estimation. In five synthetic datasets, the proposed method achieves similar results to CFR-C, a strong baseline which requires to store all data and needs to be trained from scratch.

I have some concerns about the motivation of this work. A simple question about the claim on the original data: why does the performance of the model trained continually on the original data matter? It is not difficult to save a checkpoint of the model trained on original data and use it for the original data whenever you want, given the fact the neural network based causal inference models are small. Or you can let the model do the inference and save the results periodically. Similarly, I could not get why “catastrophic forgetting” is an issue in this problem. If storing the original data would cause a copyright or privacy issue, I wonder why storing a model or feature representations trained by such data can avoid these issues. The memory issue of storing data is also not likely to happen in real-world since collecting data is often more complicated and expensive than storing them.

In section 2, there is an issue in the strong ignorability assumption. It would be great if the authors can clarify. The notation \mathcal{X}_d assumes that the feature space can change over domains, but the ignorability needs a unified feature space. So, the ignorability assumption actually relies on varying observed covariates space when $d$ changes.

Regarding Table 1, could the authors report the exact setting for the CERL model which achieves the reported numbers? For example, the number of stored feature representations. This is because in  Figure 4, it shows M=10000 has higher PEHE and \epsilon_ATE than the one reported in Table 1.

Since training from scratch with all data is expensive, it would be interesting to show a comparison of storage space, RAM memory and runtime between CERL and CFR-A, B and C.










review_title: The work adopts continual lifelong learning for causal effect estimation. I have concerns on the motivation and the assumption. I also suggest several improvements for the experiments.

final_score: rating: 4: Ok but not good enough - rejection
review_content: I think the problem proposed in the paper is interesting and the method is moderately novel. I did not find the paper's writing very helpful. The method seems reasonably motivated but the discussion around it does not provide enough intuition about the influence of the different pieces on the final prediction, and no ablation experiment are provided.

My main issue with the paper is the limited experimental evidence.

1. Without any real data  (at least semi-synthetic) experiments, I do not think there is a good way to evaluate the significance of the problem and the proposed method.

2. The single synthetic experiment in the paper is  not sufficient to evaluate the effectiveness of the method. I believe at a minimum, different synthetic experiments with different structural models, dataset shifts and rate of shift over time should be considered.

3. No ablation studies are done and no real baseline is considered. For example, a non-trivial baseline would be to do use a traditional continual learning objective with distillation with a new treatment-control population balancing objective with each new dataset. The value of the proposed method's representation-herding would then become clear.


If the authors address these issues, I'm happy to change my score.





review_title: reasonable method; limited experimental evidence;

final_score: rating: 3: Clear rejection
review_content: 
This paper considers adopting continual learning on the problem of causal effect estimation. The paper combines methods and algorithms for storing feature representation and representative samples (herding algorithm), avoiding drifting feature representation when new data is learned (feature representation distillation), balanced representation by regularization, etc. Consequently, the paper presents a system that makes use of existing methods as a loss function (the sum of losses and regularization terms). 

It is difficult to observe the novelty of the method---there is no apparent challenge arose in combining these methods as they can just be simply added. Further, the development of this framework for causal inference is mostly orthogonal to the problem of causal effect estimation (and the use of regularization does not add any novelty.) In addition to the lack of novelty, the paper has many non-negligible mistakes in describing causal inference. For example, consider a phrase "selection bias" between "treatment and control groups" in the first sentence of the second paragraph. "Treatment and control groups" usually are used to describe two groups under the context of RCT. Further, the bias we are talking in the paper is a "confounding bias" not a "selection bias". In Section 2, "Each unit ... received ... one of ... treatments" is also relevant to RCT not observational data. There are many other places the authors mentioning "selection bias".

Question:
What is "Real world evidence"?, which is used in the title, abstract, conclusion, and at the beginning of section 3? Is it just observational data?
review_title: review

final_score: rating: 2: Strong rejection
review_content: The authors propose to estimate individualized treatment effects in a setting they describe as continual learning, batches of data becoming available over time. The technical proposal is to store a feature representation of the presently available data to be updated with new data in the future -- a technique that, together with other regularization methods for selection bias and variable selection, is shown to perform well on synthetic experiments. 

Causal inference is increasingly relevant as machine learning assists decision-making, and an important aspect of this process is to handle information streams over time and update accordingly, just as humans would be expected to. In the context of causality though, I believe this requires a more detailed motivation. Causal effects by their very definition exhibit invariance to interventions on observed variables, and if properly estimated, these should not vary between well-behaved environments. With enough data a priori there should not be large changes in estimation in new environments. This may certainly change if the underlying causal mechanisms change and is a problem that has been studied as data fusion, yet this line of research is not mentioned in the paper.

Writing needs to improve substantially. The use of the words extensibility, adaptability, and accessibility is grammatically incorrect, and the context in which they are used does not clarify the meaning of these ideas either. As I understand it, adaptability refers to domain adaptation yet how domains differ, how to merge them depending on their differences, or even an investigation of these problems is not discussed. Accessibility refers to the amounts of data one needs to deal with. This does not strike me as a problem in causal inference where datasets are typically small. Perhaps examples of applications where each of these aspects is a concern would be helpful.

More importantly, the notation is counter-intuitive and inconsistent. For instance, \mathcal X is defined as the set of all observed variables, but then a specific realization x is written to be an element of \mathcal X (as if \mathcal X was a measurable space of possible realizations of a random variable). Both interpretations cannot be correct. The notation \mathcal X_d is similarly ambiguous, does this mean each domain has different sets of variables or that the spaces in which they are defined differ? 

The experiments are underwhelming and certainly do not back up the claims made in the introduction. Most benchmarks for individualized treatment effects are semi-synthetic (it would be advisable to stick to these data generating processes while modifying them to highlight specific features of your problem). Many more competing algorithms could be evaluated.
review_title: Important problem with some good ideas but the current paper needs substantial revision.

Meta-review: The authors consider the problem of causal inference from multiple conditionally ignorable models that yield different observed data distributions.  This problem is distinct from transportability (which assumes some types of causal invariance across domains, and aims to move causal conclusioned learned in one context to another using this invariance).  The authors adapt a machine learning approach from (Shalit et al, 2017).

Because the authors describe an algorithm rather than a model, it was a bit difficult to understand what assumptions tie the different observed data distributions together (I am guessing there is a way to formulate a 'global model' tying all datasets together in terms of the algorithm hyperparameters but the authors do not discuss this).

The authors evaluate their method via a simulation study.  Moreover, in response to reviewer criticism, the authors uploaded additional results from semi-synthetic data.

Some of the concerns of reviewers were about novelty and scope of evaluation (in addition, some complained about writing and notation).

Decision: Reject