Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a method of dealing with covariate shift in graph learning. To this end, the authors define "causal" and "environmental" features (i.e. features that influence label prediction and ones that are conditionally independent of the labels given the former). Subsequently, desiderata of "environmental diversity" and "causal invariance" are introduced and an adversarial learning scheme is devised that leads to learning graphs that keep their "causal features" which should translate to superior out-of-distribution generalization. This is then confirmed in a series of experiments on benchmark data that compare the proposed approach to competing methods.

strength_and_weaknesses: STRENGTHS

This paper tackles an important problem of robust learning of graph classifiers.

Starting with the definitions in Section 2, the paper is easy to follow.

The experiments are extensive and compare the approach to a number of baselines.


WEAKNESSES

Being unfamiliar with graph classifier learning, I found it difficult to develop an intuition as to what "environmental" and "causal" features are when reading the introduction and Figure 1. It only became clear when definitions were provided.

Having said that, I still think it is a bit of a misnomer to call the features "causal", it appears to me that they are simply the relevant ones when it comes to label prediction, the "environmental" ones being simply conditionally independent (and thus irrelevant for classification).

clarity,_quality,_novelty_and_reproducibility: The paper is well-structured, well-written and easy to follow.
summary_of_the_review: This paper describes a new method of graph classifier learning by adversarially generating distributions that preserve features that have predictive power towards the labels. The authors provide extensive experiments.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper addresses the problem of covariate shift on graphs. This framework is compared with 14 baselines on both synthetic and real-world data. 
strength_and_weaknesses: Strength:
- This paper addresses a rather important and interesting problem.  
- Many baselines are provided for comparison. 
-----------------------------------------------------------
Weakness:
- This paper is poorly written and can be very hard to follow at times. 
- This paper seems to be very incremental. 
- Seems like the problem is not well-justified. For instance, why is covariate shift different from correlation shift in terms of causal features and environment? 
- This paper has many false claims, for example, causal features are NOT invariant across domains.
- While causal features in images are easy to distinguish, how would one define them on graphs? Better examples and justifications are needed. 
- Contributions are not justified. Are you addressing the flaws of the existing frameworks? If that is the case, these frameworks are not designed for the specific task defined in this paper. The same goes for the experiments. 
- Definition 1 seems not to make sense. 
clarity,_quality,_novelty_and_reproducibility: This paper is hard to follow. It is not clear what's being addressed and etc. The novelty is very minor and seems like there are a lot of claims that are not supported. Seems to be reproducible. 
summary_of_the_review: This paper introduces a framework that addresses covariate shift. They provide baselines to compare their framework to. However, some of the baselines are not exactly designed for their task. There are some false claims in the paper that should be justified.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new method for out-of-distribution (OOD) generalization on graphs. The authors claim that the existing works mainly focus on the OOD issue of correlation shift, while another type, covariate shift, remains largely unexplored. Then, a graph augmentation strategy called AdvCA is proposed to handle the covariate shift problem with an adversarial augmenter and a causal generator. Experimental results show that the proposed AdvCA can outperform several baselines.
strength_and_weaknesses: Pros:   
1. The OOD generalization problem of graph machine learning is an important and trending research topic. I appreciate the authors' efforts in formalizing the problem and proposing necessary assumptions for graph covariate shifts.  
2. The authors propose a causal framework to formulate the problem and guide the graph augmentation.  
3.  Both synthetic and real datasets are used to show the empirical effectiveness of the proposed approach.   

Cons:  
However, I have some concerns:  
1. The main concern is regarding technical novelty. I think the problem formulation is similar to previous works, e.g., EERM (Wu et al., ICLR 2022a), so the formulation should not be considered a particular contribution. The authors claim that the existing graph generalization efforts cannot guarantee both environmental diversity and causal invariance, but EERM has explored these key concepts and its formulation for ego-graphs can be straightforwardly extended to the graph-level tasks.   
2. The proposed method is somehow incremental and thus the technical contribution of the paper is limited. In particular:  
(1) The idea of adversarial data augmentation is similar to EERM. More comparisons and discussions regarding related works are expected to further clarify the novelty of the proposed method.   
(2) The authors claim that the existing works cannot guarantee environmental diversity while the proposed AdvCA can. However, why AdvCA can increase diversity than augmentation-based baseline methods such as GREA is not clearly discussed.   
3. The experiments are not convincing enough. The experimental results of baselines seem to be inconsistent with the results in the original papers. The details regarding the hyper-parameters of baselines are not clearly present, which raises potential concerns in fair comparisons. The visualizations should also consider some baselines to verify whether AdvCA finds causal parts more accurately than baselines.  
4. I would also suggest the authors proofread Figure 2 since the plotted environmental features are identical and not diverse.  


clarity,_quality,_novelty_and_reproducibility: The reproducibility is good considering the codes are available, but I do have some concerns about the clarity, quality, and novelty as detailed above.
summary_of_the_review: In summary, this paper focuses on an important problem and proposes an effective method to tackle this problem, but there are concerns regarding the technical novelty, clarity, and quality.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper focus on a specific type of OOD issue, covariate shift. The authors propose Adversarial Causal Augmentation (AdvCA) to alleviate the covariate shift. It adversarially augments the data to explore diverse distributions of the environments. Meanwhile, it keeps the causal features invariant across diverse environments. Extensive experimental results on synthetic and real-world datasets prove the effectiveness of AdvCA.
strength_and_weaknesses: Strengths:
1. The paper focuses on the graph generalization problem under covariate shift, which is of great need but largely unexplored.
2. Comprehensive experiments with 14 baselines are conducted to demonstrate that AdvCA can achieve better generalization ability under covariate shift.

Weaknesses:
1. Experiments on more commonly used datasets like MUTAG and NCI1 are suggested to better demonstrate the effectiveness. 
2. The example “ladder” and “tree” are absent in Figure 1, but are mentioned in paragraph 2, which make readers confused.
3. In Definition 1, since the prediction is independent of the environment features, you can somehow learn to distinguish the causal feature and environment features. Why don’t you only keep the causal feature to make a prediction?


clarity,_quality,_novelty_and_reproducibility: Clarity: Fair

Quality: Good

Novelty: Good
 
Reproducibility: Good
summary_of_the_review: The paper focuses on the graph generalization problem under covariate shift, which is of great need but largely unexplored. However, there are some minor issues to be addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies OOD generalization on graphs by considering covariate shift, i.e., the distribution of graph inputs are different in new domains. To address this problem, the authors propose an adversarial causal augmentation method to explore the diverse distribution of environments. 

While the proposed problem is interesting and the proposed method is interesting, there are several concerns on the technical quality and novelty. First, the authors overuse the term “causality” in the paper. By this paper’s definition, the causal features are the stable features while the environmental features are changing features. It is not necessary to define a causal model as the proposed method does not make use of any property of the causal generative process, e.g., modularity or interventional causal effects. Second, the authors claim that the two set of features can be separated, however, there is no theoretical evidence or empirical results to demonstrate it. Third, it is unclear why the proposed data augmentation strategy could cover possible structures in the new test domain. The assumption under which the data augmentation could work is not provided. Given these concerns, I would recommend rejection of this paper in its current form.  Although we think the paper is not ready for ICLR in this round, we believe that the paper would be a good one if the concerns can be well addressed.

justification_for_why_not_higher_score: unclear motivation of using causality

the assumptions under which the data augmentation method could work is not clear.
justification_for_why_not_lower_score: n/a
Decision: Reject