Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on the problem of debiasing learning and out-of-distribution detection in graph data. The authors argue that existing works typically address these two tasks independently, but the intrinsic connections between training outliers and OOD test samples are overlooked. To this end, the authors propose a novel model called GraphDE to tackle debiasing learning for training data and OOD detection for test data under a unified probabilistic model. Extensive experiments are conducted on different GNN backbones, and the results validate the superiority of the proposed method over the baselines. Theoretical analyses are also provided to justify the effectiveness of the model.
strengths_and_weaknesses: Strengths:
1. This paper focuses on an interesting and important problem. Outliers in training data and OOD samples are very common in the graph domain. Properly handling them is important to ensure the performance of a GNN model in practical usage.

2. The paper manages to solve the problem of graph debiased learning and OOD detection in a novel perspective. That is, a unified framework is proposed to model the generative process of both the training data and the test data, where the two problems are tackled dependently.

3. The paper is well-written and easy for readers to follow.

4. The experimental results show that the proposed method achieves consistent performance improvements over the baselines, which demonstrates its superiority. Also, a detailed theoretical analysis is provided to justify the rationale of GraphDE.

Weaknesses:
1. It would be better if the authors can discuss under what condition the proposed model may not achieve desirable performance. For example, why GraphDE cannot achieve the best performance on Collab in terms of AUPR and on DrugOOD in terms of FPR95?
questions: Why does not GraphDE outperform OCGIN or other baselines on Collab in terms of AUPR and on DrugOOD in terms of FPR95? More discussions on the undesirable results may be helpful.
limitations: No potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this work the authors propose a method for jointly addressing outliers present in training data as well as learning an OOD detection model. Using a simple binary environment variable they learn a unified probability mixture model and demonstrate performance on both the debiased learning and OOD detection tasks.
strengths_and_weaknesses: ### Strengths

**Quality**: The joint perspective on training time outliers and the task of OOD detection at test time is principled and correct, in this reviewer's opinion. The choice of datasets and the analyses are appropriate for the problem setup and the augmentation methods are relevant baselines. The explanatory ablation results at the end help address appropriate questions about the design choices made.

**Originality**: The main novelty comes from the end-to-end manner in which they address weighting examples at training time as well as extracting the OOD model

### Weaknesses

**Clarity**: The section where Propositions 1 and 2 are stated is lacking proper backing for its claims, or rather in its current form, it's unclear whether this section contributes meaningfully. Overall, the concept of predicting the binary environment variable in order to learn a more optimal model conditioned on the environment variable makes sense, but the assumptions on the perfect recognition and perfect generative models are too strong to just state in this section. I don't feel the sections add significant justification for the approach.

**Significance**: Results on more updated large scale data (say derived from OGB) would help demonstrate that the binary environment variable approach doesn't break down at scale - which matters for practical adoption of new methods, especially those focused on real world problems like distribution shift.

Work would generally benefit from a close editing pass from a native english speaker, _but this does not factor into my assessment_.

questions: 1. In line 42, do you actually agree with the statement that test time is just about feature/covariate shift? I would argue the very premise of a need for a method such as yours, plus basic knowledge of real world data characteristics together suggest that joint distribution shift is also a test time issue (I know it's arguably hard to address directly with any method)
2. For the generative structural process you are making the assumption that the connectivity, or the adjacency matrix, is implied (caused) by the node features - cites random graph theory/homophily. Is this a valid assumption? Under this assumption it seems to follow that the distribution shift is therefore caused by node feature shift. Could your method be extended to when this assumption does not hold?
3. Figure 3 and Table 1 seem to display the same kind of data? Could they be visualized the same way, or was this choice made because the bias ratio was fixed for the table? (why was this fixed for that table?)
limitations: 1. This method introduces the requirement for a choice of prior on the cleanliness of the training set - this will have some effect on the final model performance, unknown in a real deployment setting.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper studies debiased learning and OOD detection for GNNs. Specifically, the authors propose GraphDE, a probabilistic generative framework to model the distribution shifts of graph data. The proposed method contains three main modules: the recognition model to infer the environment variables, the structure estimation model to detect outlier and OOD testing data, and the classification GNN model. Theoretical and empirical justifications of the proposed method are provided.
strengths_and_weaknesses: Pros  
[+] The motivations are clearly present. Figure 1 also helps understand the research problem of this paper.  
[+] The proposed method shows improvements in the adopted datasets.   
[+] The sensitivity analysis and ablation studies are provided to gain deeper insights into the proposed method.  

Cons:  
[-] The novelty of the paper is somewhat limited. The essential idea of the proposed method is building a variational inference module on top of the existing GNNs, which are both heavily studied in the literature. Besides, the variational inference to infer the environment variable seems rather general and hardly connected with graph data (i.e., it may also be applied to other data types). However, the authors claim the challenges of non-Euclidean graph data in the introduction.  
  
[-] The authors do not provide time complexity analysis or the time cost in practice, so the efficiency aspect of the proposed method is unclear.  
  
[-] The experiments are not entirely convincing.  
a) Most importantly, the authors do not compare with OOD baselines, including general OOD methods (e.g., IRM, DRO, etc.), which can be directly combined with GNN backbones, and recent methods designed explicitly for graphs (e.g., DIR and OOD-GNN). 
b) The authors only consider GCN and GAT as the backbone but ignore more popular and powerful backbones such as GIN.  
c) Though I acknowledge that some real-world graph benchmarks are adopted, it would make the experiments more convincing if OGB datasets, widely adopted in the literature, are further included.   
  
[-] The writing of the paper could be improved:  
a) Prop 4.1 and 4.2 are in plain language and thus may not be rigorous. It would be better if these propositions are provided in mathematical terms.  
b) Some of the claims are not well supported. For example, one of the claimed contributions is that the modules are optimized in a mutually-promoting manner, which is not verified in the experiments. The necessary discussions on the inferred environments during the training of GNNs (as stated in line 12) have also not been present in detail.   
c) The problem formulation in Section 2 seems to be literally identical to the existing work EERM, e.g., lines 69-73 and Section 2.1 in EERM. Such textual overlap without quotation should be avoided.  
  
[-] The authors claim in line 109: “For the first time to our best knowledge, we formally define and deal with the graph debiased learning problem.”, which misses important related works such as [1-2].  
[1] Shift-Robust GNNs: Overcoming the Limitations of Localized Graph Training Data, NeurIPS 2021  
[2] Debiased Graph Neural Networks with Agnostic Label Selection Bias, TNNLS 2022  

Minor:  
(1) More related works in Appendix B seem to highly relate to the paper and should be moved to the main paper.  
(2) The font of “idx” in line 222 should be kept consistent with that in Eq. (14).  

questions: See above
limitations: N.A.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper aims to address two challenging tasks, i.e. 1) learning debased GNN model from the training data with OOD samples and 2) detecting the OOD samples from testing data, with a unified framework. To this end, a novel method named GraphDE is produced on the basis of variational inference. A probabilistic generative model is introduced to model the distribution of ID and OOD samples, and the classification module and OOD detection module are integrated into the learning framework. Experiments are conducted to verify the performance of GraphDE on both debiased learning and OOD detection tasks.

strengths_and_weaknesses: Pros:
- The research problem is interesting.
Both debiased graph learning and graph OOD detection are attractive and interesting topics in the research community. The authors discuss the similarities and differences of these two tasks and address both tasks with a unified learning framework, which is promising and challenging.
- The results are promising.
In the experiment, the authors conduct quite extensive experiments, and the proposed method achieves very good results, which prove its effectiveness.
Cons:
- The construction of graph generative model is defective.
In the generative model, the distribution of feature p(X) is not modelled specifically, and the generation of feature X is not involved in the generating process. However, the ID and OOD data obviously have different feature distributions. For example, molecules from different domains tend to be formed by different atoms. In this case, it is not practical to model ID/OOD graph data without considering the difference in feature distribution.
- The instantiation of OOD structure distribution p_0 is not practical.
In the instantiation, the authors use a simple distribution p(a=1)=1/2 to model the OOD data. However, in practice, the real-world OOD data can be universal, which cannot be simply modelled by such an impracticable distribution. Moreover, the ID data is also possible to follow "p(a=1)=1/2". In this case, how to distinguish ID/OOD distributions in practice? Therefore, I believe modelling OOD data with more complex or with learnable distribution is more persuasive in this method.
- There is potential leakage in the evaluation of OOD detection.
Since the model is trained on data composed by both ID and OOD data, the patterns of OOD data have already been seen by the model. In this way, a concern raises that the knowledge about OOD data would leak during the learning procedure. Such leakage may lead to unfair comparison in evaluation.
- More baseline methods should be considered for comprehensive comparison.
Also graph debiased learning and graph OOD detection are both new-born directions in graph learning, there are already some pioneering works that focus on these topics. However, the authors consider limited baselines for comparison, which reduces the persuasion of experiments. So, more baselines can be consider for debiased learning (e.g., [*1], [*2]) and OOD detection (e.g., [*3]). More related papers can be found in survey [*4].
[*1] Fan, Shaohua, et al. "Generalizing Graph Neural Networks on Out-Of-Distribution Graphs." arXiv preprint arXiv:2111.10657 (2021).
[*2] Li, Haoyang, et al. "Ood-gnn: Out-of-distribution generalized graph neural network." arXiv preprint arXiv:2112.03806 (2021).
[*3] Ma, Rongrong, et al. "Deep Graph-level Anomaly Detection by Glocal Knowledge Distillation." Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. 2022.
[*4] Li, Haoyang, et al. "Out-of-distribution generalization on graphs: A survey." arXiv preprint arXiv:2202.07987 (2022).
questions: - Is that practical to define the larger graphs in a dataset (i.e. Collab) or the images with gaussian noise (i.e. MNIST-75sp) as the OOD samples? It is hard to say these samples are generated from a different distribution. 
- In Line 134, the authors say "p_\theta(X|A,e)" is composed of an ID component and an OOD component. Is there a typo? Because the assumption is that A is generated from X. 

limitations: The authors have provided the limitations and potential negative impacts of the proposed framework in Appendix G and H, respectively. However, more discussion for potential negative impacts should be given, for example, the debiased learning paradigm may cause the concern about fairness.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a framework called GraphDE, which unifies debiased learning and OOD detection for graph data. GraphDE takes a generative view to model the joint distribution of $\{G, y, e\}$, with an additional recognition model that infers an OODness indicator variable $e$. The loss is carefully designed according to the framework, with interpretable justification. Various and extensive experiments demonstrate the effectiveness of GraphDE, which is expected to provide reference for future related works.
strengths_and_weaknesses: Strength:
1. The framework unifies debiased learning and OOD detection, i.e. the OOD detector is obtained during the debiased learning.
2. The loss is well-interpretable.
3. Experimental results are fairly good.


Weakness:
See my questions below.
questions: I have a few confusions. Please correct me if I am wrong:
1. How do you calculate the threshold in Eq. (3)? 
2. Line 107-108, the authors mention that prior works are limited to feature-label shift but ignore the variation in the feature space. Can GraphDE capture the variation in the feature space?
3. Line 132, why are the two distributions parameterized by the same $\theta$?
4. Line 134, I think there is a typo, i.e. $p_\theta(X|A,e)$ should be $p_\theta(A|X,e)$.
5. In the last term of Eq. (7), should $p(e)$ be $p(e_i)$?
6. I am very confused with line 181-182 which mentions that $p(e)$ is a scalar $\in[0,1]$. If so, why $p(e)$ is not canceled out in Eq. (8)? I think it should be a discrete distribution over ID and OOD input.
7.  Do you have the assumption that training and test data have the same $p(e)$? Because $p(e)$ (I suppose it is obtained in the training set) is used in the inference on test data (in Eq.(8)). If test data have a different $p(e)$, will GraphDE still work?
8. Eq.(9) is used to predict label on test data. Do you still output a label even when OOD is detected?
9. What is the role of GraphDE-v? Since you can compute the posterior analytically, it seems to me GraphDE-a works perfectly, and it is also confirmed in Table 1 that GraphDE-v is entirely outperformed by GraphDE-a.
10. Figure 4 is counter-intuitive to me. Eq.(2) tells me that debiased learning can be reduced to ignoring updates from outliers (because training on outliers will hurt the performance), and proof in Appendix A.2 tells me that optimizing the proposed $L_{cl}$ is equivalent to optimizing Eq.(2). Then I suppose $L_{cl}$ is why GraphDE can boost test accuracy as in Figure 1(a). However, the ablation study shows that removing $L_{cl}$ makes literally no difference to the performance, while removing $L_{reg}$ shows more impact. Do you have any interpretation on this result?
limitations: The authors addressed the limitations and societal impact in their paper (in the end of appendix).
review_title: null

Meta-review: metareview: The authors propose a mixture modeling approach to train GNNs so that out-of-distribution data can be properly down-weighted during training and detected during testing. The reviews were mixed, with some reviewers criticizing the technical novelty and experimental comparison. Indeed, the authors could have explained their contribution more transparently, and emphasized a bit more on the new challenges in the GNN setting, which the response has largely addressed. Perhaps it is also worthwhile to discuss classic works on mixture of experts, as well as variational Bayesian approaches (e.g. https://ieeexplore.ieee.org/document/5563102). As to the experimental comparison, I think the authors made some good explanations in the response and it is perhaps too ambitious for anyone to compare to every possible alternative. 

In the end, we think the application of the mixture modeling approach to GNNs is sufficiently interesting, and the initial experimental results appear to be encouraging. We urge the authors to further revise their work by incorporating all changes during the response and better positioning the contributions in historical context.
confidence: Less certain
Decision: Accept