Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work looks at the problem of link prediction from the lens of causal inference. In particular, the authors introduce counterfactual links, which are links which would have existed under a different treatment (with a running example being the neighborhood identifier). The authors use this definition to define a training procedure which builds on prior art in neural net based causal inference that uses an IPM penalty to ensure balance in representation between treatment and control outcomes. Experimental results are provided which show strong empirical performance for the proposed model.
main_review: Overall, I found this paper to be an interesting approach, but there are a few items that give me pause. Most importantly, it's not entirely clear what the causal estimand is that is being measured. The authors use membership to a community as the running treatment and consider the existence of a link between nodes as the outcome. However, it's not entirely clear to me that what the authors are ultimately doing is measuring a causal effect (the central task does not appear to be measuring a difference between observed and counterfactual networks). Instead, it seems that the proposed procedure is improving performance by enforcing a kind of invariance in representation between the observed network and the closest perturbed network. I find this to be a very interesting and compelling approach, but unfortunately it appears to be a bit obscured by the presentation.  

A few more minor points:

* Sherman & Shpitser (Intervening on Network Ties, UAI 2019) should probably be cited given the relative similarity in task

* It would be helpful if the definition of ATE provided by equations 13 and 14 appeared earlier in the text. It would give a lot more clarity to the presentation to my eyes to have a more formal definition of the estimand provided during the problem setup. 

* The authors define the ATE as the expectation of the ITE. The ITE is also an expectation, but conditional on X, whereas the ATE is an unconditional measure (after marginalizing over X).  

summary_of_the_review: Overall, I think this is an interesting approach, but unfortunately I think the framing of the paper as a causal estimation paper, rather than a paper which uses counterfactuals a mechanism to improve invariance and performance makes the contributions of the paper slightly obscured.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper targets an interesting question in network analysis: what are the main causes leading to the creation of a link between a pair of two nodes in a given network observations. 

Most previous related models assume a latent structure underlying observed network data, and suppose the creation of each link is driven by the latent structure behind two nodes associated with that link. Hence, the main cause effect resulting in the link between two nodes, may not be truly reflected by the underlying community structure.

To address this concern, the paper resorts to a counterfactual learning framework by learning counterfactual links between the most similar node pairs with a different treatment. Experimental results demonstrate the improved link prediction by the counterfactual graph learning method compared with latent community based methods.
main_review: strengths:
- novelty:
To me, the new perspective and the main idea are novel in the context of link prediction. 
- correctness:
To my knowledge, the idea and technical contribution look sound.

weaknesses:
- No obvious weaknesses were found in this paper.
- It is not clear why T^{CF}, A^{CF} = T, A, otherwise in Eq.(4)
- It is not easy to digest "...ITE of neighborhood assignment as 1-1=0..." for someone without causality background!

additional questions:
- can you discuss how to generalize your method to continuous-valued network data in the supplement? say, A_ij \in [0,1], represents the connection strength between nodes i and j.
- if possible, can you discuss or even conduct additional experiments to demonstrate the robustness of your method to missing (false zeros) and noisy link data, which is prevalent in large-scale networks. 
- except AUROC score, have you considered PR score to evaluate the link prediction performance as PR score is only sensitive to non-zero links?

typos: 
- pp.1 only were -> are
- Algo 1 model inferencing -> inference
- many places causal model -> causal model(s)
summary_of_the_review: Overall, I vote for accepting this paper considering its novelty on solving link prediction from a new perspective of counterfactual learning.
Nonetheless, my background on causality does not allow to check all of its correctness in such a short time.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose an approach for link prediction using counterfactual learning. They define a "treatment" for a node pair as whether or not they belong in the same group (e.g. from running a graph clustering algorithm). They then find the most similar node pair with a different treatment and treat its outcome (whether there is a link) as the counterfactual of the outcome of the original node pair. To find the most similar node pair, they use a graph neural network (GNN) based encoder. They use two link decoders (both multi-layer Perceptrons) to predict both the actual and counterfactual outcomes. They demonstrate impressive improvements on link prediction accuracy on 5 real data sets.
main_review: Strengths:

- Highly creative and novel approach, to the best of my knowledge. The closest related work I have seen is to add and remove edges to a graph to try to improve link prediction accuracy, but that is very different to the counterfactual learning approach proposed here.
- Proposed approach is conceptually easy to understand and could lead to many variants in the future.
- Strong empirical performance compared to other recent GNNs.

Weaknesses:

- The proposed counterfactual learning formulation feels like a hammer looking for a nail. The way the treatment is defined seems highly unusual, and there is nothing really principled about it.
- The authors claim that the estimates of average treatment effect (ATE) are generally close to the observed ATE, but I don't see this at all from Tables 3 and 4. The estimated ATEs are close only for a few specific cases. I do agree with the second conclusion about the ranking of estimated ATE being useful to select the treatment, however.

Question regarding a claim from Page 5 section on Complexity:  "When $\gamma$ is set as a small value to obtain indeed similar node pairs, this step (Eq. (3)) uses constant time." Why is this the case? Also, what is $|\mathcal{E}|$? Is this the total number of edges in the graph?

Typo:

- Page 3, 4th paragraph: "Traditional causal inference methods hence statistical learning approaches". "hence" should probably be replaced with "use"

*After discussion period:*
I have lowered my score slightly following the discussions but continue to support the paper. Despite the unusual framing as counterfactual learning as compared to data augmentation, the empirical gains are highly impressive. I think it could inspire future research investigating the use of other structural properties of the graph as the "treatment".
summary_of_the_review: The authors propose a highly creative approach to improve link prediction accuracy by an innovative definition of treatment for a node pair in a counterfactual learning set up. The proposed approach yields highly impressive empirical link prediction accuracy.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work, authors presented a counterfactual graph learning method for link prediction (CFLP), where authors introduced the idea of counterfactual prediction to improve link prediction on graphs.

Authors demonstrate the model performance on link prediction tasks and achieved promising results. Such results shed insights that a good use of causal models (even basic ones) can greatly improve the performance of (graph) machine learning tasks.
main_review: For experiment, I would recommend using more from OGB. I am curious as OGB has many ling prediction dataset, why authors only use one of them? In addition, OGB-DDI authors model should have ranked second best if author submit the results, I wonder why authors do not submit the results in the official leaderboard? I strongly recommending using official leaderboard to illustrate the performance.

CFLP w/ JKNet consistently achieves the best performance across all dataset, while JKNet itself is only very outstanding. Is there any insight on the impact of CFLP for different architecture? 

For the node representation, authors used MVGRL. Is there any ablation study on the impact of the node embedding? This is the early step and if embedding quality is low, the error can propagate through the learning process. 

Also authors mentioned the embedding is learnt from the observed graph, so my understanding the link in validation/testing set is removed during embedding learning?

In page 4, "That is, we want to find the nearest neighbor with the opposite treatment for each observed node pairs and use the nearest neighbor’s outcome as a counterfactual link." I would recommend adding reference to some matching based methods in causal inference.

In page 4, authors first used d for a distance function in (2) between two node pairs, while later used similar d for a distance function for two nodes. This is confusing and I would recommend using another symbol.

For RELATED WORK, it seems not clear how some literatures (especially in the causal inference section) are relevant to this work.
summary_of_the_review: From the quality and novelty perspective, the proposed method is interesting and the experiment is thorough. However, my major concern is the causal model in the paper. I am not very convinced with the causal model with 'node similarity' as a treatment, and 'link' as outcome. The proposed approach is more like a data augmentation, where similar pairs are found with different community-relation to enrich the training set. It would be helpful to spend more effort to illustrate why such causal graph design makes sense.

In addition, the experiment results are exciting, but I didn't observe any submission in the open benchmark website. I highly recommend make a submission for a fair comparison, or explain why the submission has not done yet.
review_title: null

Meta-review: This work proposes an observational (not counterfactual) link prediction method based on clustering and data augmentation. The total score went down during the rebuttal phase. One of the challenges is that the title confused reviewers. The authors proposed other titles. But in the end all proposed titles had variations of "counterfactual graph", which is what makes the title confusing. 

I think if the work had been more empirically-focused and stayed away from a causal model, it would have been accepted. The idea is ingenious and it is likely that it indeed gives the empirical improvement in observational models that the authors claim. The authors have very scant citations to relevant works in the intersection of causality and representation learning anyways. But if the causal model must stay for future submissions, the authors should: (a) more clearly define and differentiate the causal and non-causal aspects of the task (and properly contextualize the work), and (b) more clearly specifies the causal model that is being utilized (and be more mathematically rigorous in general).

Unfortunately, the original description of the causal model in the paper was both sloppy and incorrect. I had to ask a total number of five (5) long clarifying questions (visible to authors and reviewers only) in order to understand the causal assumptions. In my back-and-forth with the authors, the authors proposed a total of 7 different causal DAGs to explain their method. Some of these DAGs contradicted each other and most of the answers were vague. My first advice to the authors is to always be mathematically precise for the benefit of reviewers and readers. Causality requires very clear assumptions and precise notation. For instance, the variables in the causal DAG given in Figures 2 and 3 had little to do with the variables in the actual graph model. The causal DAG must have the variables $A_{ij}$, $T_{ij}$, $C_i$, $C_j$, $z_i$, $z_j$, for all $i,j \in V$. The final causal DAG proposed by the authors in the discussion was:
- $z_i \to A_{ij}$
- $z_j \to A_{ij}$
- $z_i \to T_{ij}$ (may not be needed)
- $z_j \to T_{ij}$ (may not be needed)
- $C_i \to T_{ij}$
- $C_j \to T_{ij}$
- $T_{ij} \to A_{ij}$

Regarding the nature of $z_i$ and $z_j$: They need to be formally described as structural node embeddings of the graph, otherwise the DAG is incorrect. After digging and reading some of the references the authors use, I came to the conclusion that the empirical method indeed relies on structural embeddings (and the authors later confirmed it). This should be made **very** clear in the SCM description.

This final causal model after discussion looks reasonable. And one challenge that remains, however, is sampling or MAP point estimates of the posterior $P(C_i,C_j|A)$. All standard clustering methods that can use this DAG rely on node embeddings $z_i, z_j$ being positional embeddings, not structural ones. So, how are we actually obtaining $P(C_i,C_j|A)$? The authors would need to prove that standard clustering methods can perform this task (I have my doubts this is even true). I fear this issue could again derail the paper in a future submission if the counterfactual justification is used again. One of the reviewers raised concerns about the quality of the embedding already.

The proposed method is observational and does not need a causal description. But if a causal description is provided, it must be correct and the method must be theoretically sound. The causality part of the paper needs a substantial revision, unfortunately. I suggest rejection.
Decision: Reject