Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims at improving the performance of GNN in the test time by transferring the node features and structure.
More specifically, the transformation is optimized by a contrastive surrogate loss with respect to the embedding from the pretrained model.
Extensive experiments are conducted on different settings and datasets, the results somehow prove the effectiveness of the proposed method.
strength_and_weaknesses: **Strength**
- The motivation of this paper is interesting, the authors consider several critical problems in the test time of GNN, including OOD, feature anomaly and adversarial attack.
- The paper is overall easy to follow.
- The experiments are extensive.

**Weakness**
- The overall problem setting is quite similar to the graph domain adaptation(GraphDA) where the source domain graph data provides a pre-trained model and target domain is the test graph without labels. However, the authors do not mention this active research field as well as the baseline methods. In the GraphDA field, the transformation on the target domain is a quite common operation. Concerning the Equation (6), it is directly computed on the pre-trained model and serves as a weight/guidance on the trasnformation learning, this is an explicit domain adaptation step where the knowledge from the source domain is transformed to the target domain. The authors are required to clarify the difference between this paper and GraphDA task.
- The so-called parameter free transformation is quite common in contrastive learning. In this paper, the dropedge is only a classic one and many other transformations such as feature shuffling, subgraph sampling can be explored.
- The Theorem 1. is somehow weired, in the test time the graph is unlabelled and the classification loss is no longer appliable. In this situation, will theorem 1 affect the model training and transformation selection?
- In the OOD setting, the improvements is not significant and the authors are expected to explain on it, especailly without comparing the GraphDA methods.
- In the nosiy feature setting, the proposed method only works on the small dataset and seems not working on the larger dataset, the authors are expected to discuss on it.

clarity,_quality,_novelty_and_reproducibility: The paper is overall well written and easy to follow. Currently, I think the setting is quite similar to GraphDA which limits the technical contribution. However, if the authors can well distinguish this paper with GraphDA, I will reconsider the novelty. The reproducibility is good as the authors provide detailed setting in the appendix.

summary_of_the_review: My major concerns is the difference between the problem setting and the GraphDA methods.
Currently, I think they are quite similar so the technical contribution and baselines are somehow weak to me.
However, if the authors can well address their difference, I will improve the final rating by reconsidering the technical contribution.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: GNN is a new technique that can be used in many areas such as recommendations, drug discovery, social computing, etc. How to learn the  graph representation is key issue in the area of GNNs. This paper proposes a new method to empower the graph representation learning with test-time graph transformation.  Aslo, this paper provides theoretical analysis on the design of the framework and discusses why adapting graph data works better than adapting the model. Extensive experiments on several datasets show its effectiveness. 
strength_and_weaknesses: Strength.
1. The research is novel and interesting. 
2. The paper is well-written and easy to follow. 

Weakness.
1. This paper does not provide runnable code to reproduce the result. 
clarity,_quality,_novelty_and_reproducibility: Clarity.
The  paper is well written and easy to follow. 

Quality. 
The paper proposes a new method to empower the graph representation learning with test-time graph transformation.  Aslo, this paper provides theoretical analysis on the design of the framework. Furthermore, the paper conducts extensive experiments on several datasets and shows better performance compared with several baselines. 

Novelty. 
The idea is interesting and novel. Most of prior work focuses on to improve graph representations via model structure design. Working on the test-time graph is new direction. This paper proposes a new method in this new direction.  



summary_of_the_review: GNN is a new technique that can be used in many areas such as recommendations, drug discovery, social computing, etc. How to learn the  graph representation is key issue in the area of GNNs. This paper proposes a new method to empower the graph representation learning with test-time graph transformation.  Aslo, this paper provides theoretical analysis on the design of the framework and discusses why adapting graph data works better than adapting the model. Extensive experiments on several datasets show its effectiveness. 

Most of prior work focuses on to improve graph representations via model structure design. Working on the test-time graph is new direction. This paper proposes a new method in this new direction, which will be useful to resolve the problem of dynamic graph learning.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a test-time graph transformation approach to improve graph representation learning performance. Specifically, during test time, the structure and features of the test graph will be optimized through a contrastive learning objective. Experiments on node classification under various settings demonstrate the effectiveness of this data-centric approach over model adaptation approach.

strength_and_weaknesses: Strengths:
+ This paper proposes a novel idea that optimizes the graph structure and features of the test graph.
+ The writing is satisfactory and the idea is clear to follow.

Weaknesses:
- Several claims are not fully substantiated.
  1. Theorem 1 gives a criterion on selecting the surrogate objective while no empirical nor theoretical evidence is given to show that the contrastive learning (Eq. 6) approximates the classification loss. It could be better to show, for example on one dataset that the gradient paths of the contrastive learning indeed are close to that of the classification objective.
  2. Theorem 2 relies on very strong assumptions that the mean of $Z$ and $\hat{Z}$ remain the same for each class. Can authors further justify that this assumption is realistic?
  3. For Eq. (6) the authors claim that existing contrastive learning objectives need extra prediction head that will alter the network architecture. It is not true. For example, the JSD loss and the triplet margin loss do not include additional parameters. Also, it is possible to use bootstrapping-based loss to update the graph structure.
  4. There are also several graph contrastive learning approaches that leverage learnable graph augmentation, which should be considered as baselines as well. For example Adversarial Graph Augmentation to Improve Graph Contrastive Learning, NeurIPS 2021
  Considering the above points, I believe the third contribution needs further elaboration. The technical contribution for the parameter-free surrogate objective is undermined.
- From Table 1, it is very hard to say that GTrans achieves significantly better performance over baselines. Almost all performance gains fall into one sigma. Also, I am curious about how to select and how sensitive the model to the parameter B as it controls the overall degree of graph transformation. 

clarity,_quality,_novelty_and_reproducibility: The paper is well written. The proposed approach is somewhat novel. Several technical contributions should be further elaborated. No data or code is provided to examine the reproducibility.
summary_of_the_review: Overall I like the idea of employing test-time transformation to improve graph representation learning performance. However, there are some technical contributions that are slightly overclaimed. Experiments also only show very marginal improvements. Therefore, I recommend rejection at this time.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a data-centric approach for improving the test set performance of GNNs under distribution shift, abnormal features, and adversarial attacks. In particular, the approach assumes the existence of a pre-trained GNN model and aims at improving the test set performance by updating the graph structure and node attributes at the test time with a surrogate loss function. Some theoretical results have been provided to explain why the proposed approach works. A good set of experimental results have been also provided across different setups to show the benefit of the approach empirically.
strength_and_weaknesses: Strengths: 
+ Good set of experiments across different tasks and with different GNNs as backbone
+ Interesting analysis of the results (I like Table 4)
+ Clarity of writing
+ Good set of related work

Weaknesses:
- The surrogate loss function used in Eq. 6 has been previously used in multiple unsupervised graph representation learning approaches and its merit has been established. Moreover, modifying the graph structure and node features has also been vastly studied in previous work, but as far as I know in all works it has been done during training. So the main novelty of the presented work seems to be in using the combination of the above two **only at the test time** (in fact at a high-level the paper could be viewed as a generalization of the SLAPS model [1] applied only at the test time). This is fine given the high performance of the model. But that makes me wonder why the approach has not been tested against other graph structure learning (GSL) approaches when GSL is done both during training, and also during testing with a fixed model (only the structure changing)? This could have been done with SLAPS [1], IDGL [2], LDS [3], etc.

- \Delta A has been modelled as a full matrix of parameters making the approach not scalable to large graphs.

- I was quite excited when I read theorem 2 but when I looked at the proof, I noticed that approximation between Eq. 17 and Eq. 19 is so coarse that I'm not sure if it even means much. While Eq. 17 requires that each node has a similar embedding across different views, Eq. 19 requires that the two embeddings for each pair of nodes within the same class have similar embeddings; the only connection between the two seems to that both embeddings of the same node should belong to the same class which is a very weak connection.

- I found the example in 3.3 to be somewhat weak: while in theory one can modify the data so the model makes correct predictions for x_1 and x_2, in practice the modification is based on some predefined procedure (in the case of the current paper, the procedure being the optimization of Eq 6) so I would expect the data modifications will still result in the model making the same prediction for both nodes.

- Some of the improvements in Table 1 are not statistically significant so the "Rank" column is misleading.

Question:
- It was not clear to me why the alternating optimizations are used. Why not optimize both losses at the same time?

- How do you decide how many gradient descent steps to use at the test time?

[1] SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks
[2] Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings
[3] Learning Discrete Structures for Graph Neural Networks
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear. A good amount of details have been included for reproducibility and the code has been promised to be made publicly available. Regarding novelty, refer to the first weakness I raised.
summary_of_the_review: The paper shows strong results with a combination of two existing techniques (graph structure learning, and a contrastive surrogate loss function for self-supervised learning) during test time. While the empirical aspect of the paper is strong (expect for some possible GSL baselines), the theoretical aspect (and intuitions) are somewhat lagging.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method which can adapt a testing graph for best performance with some pretrained GNN model.  It finds this change to the test graph by jointly optimizing the feature and adjacency matrices via projected gradient descent to maximize a self-supervised loss (basically drop edge + DGI) over the test graph itself.  Extensive experimental results are performed some synthetic distribution shift scenarios to illustrate the model's capabilities.

The paper seems like it could potentially be a nice contribution, but it could use a little refinement.

strength_and_weaknesses: Strengths:
+ Interesting problem setting
+ Nice experimental evaluation on a variety of real datasets.
+ Theory almost convinces me.

Weaknesses
- All experiments are on datasets which use manually created distribution shifts.  The scale of the shifts (and therefore the difficulty of the task) are impossible to ascertain without reporting more about these tasks.  For instance, the distributional shift could be quantized for each dataset (per task).
- One method which seemingly might work well in this scenario (SR-GNN) is suspiciously avoided.  But I think the method (or some similar domain shift regularization) would make a very natural baseline, and illustrate the difference between the case where the test is available (for SR-GNN) and when it is not (this paper).
- Another method which assumes there is test data available (AirGNN-t) is similarly not 
- Theorem 1 doesn’t provide much value (and I can’t believe it's unique to this paper -- it's just a general statement about learning with a surrogate loss).  Might I suggest citing some sources for this here?  
- I wonder if some more baselines could be added (even simpler ones).  This would add to the result.

Minor comments
- I would love a stronger emphasis on the connection between Theorem 1 and 2.  I’m not sure what to do here, but I had to revisit this a couple times.

- $\Delta A$ just feels hard to read, could you use a subscript or something? (e.g. $\Delta_A$ or $A_\Delta$)

- There's some missing related work on graph structure learning (all of which makes the same sparsity assumption as this work iirc)

  - “Grale: Designing networks for graph learning” - Halcrow et al
  - “Pathfinder Discovery Networks for Neural Message Passing” - Rozemberczki et al
  - "SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks" - Fatemi et al

clarity,_quality,_novelty_and_reproducibility: Clarity: Mostly clear, but some improvements could be made.

Quality:  Generally high quality, with a lot in the appendix.  

Originality:  There's recently been a lot of work in the OOD/transfer learning space with GNNs.  The paper does a pretty good job of positioning itself, but there is room for improvement.

Reproducibility:  There's no supplementary material, so this is hard to judge.  The authors promise to release the code however, and I don't think this method is too hard to implement.
summary_of_the_review: The authors have addressed the majority of my concerns and provided substantially more evidence to support their method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new method to empower the graph representation learning with test-time graph transformation for improving the test set performance of GNNs under distribution shift, abnormal features, and adversarial attacks. During test time, the structure and features of the test graph will be optimized through a contrastive learning objective. In particular, the approach assumes the existence of a pre-trained GNN model and aims at improving the test set performance by updating the graph structure and node attributes at the test time with a surrogate loss function. It finds this change to the test graph by jointly optimizing the feature and adjacency matrices via projected gradient descent to maximize the self-supervised loss (basically drop edge + DGI) over the test graph itself. The paper provides theoretical analysis on the design of the framework and discusses why adapting graph data works better than adapting the model. Experiments on node classification under various settings demonstrate the effectiveness of this data-centric approach over model adaptation approach. The reviewers raised some concerns including better differentiation from graph domain adaptation(GraphDA), justification on contrastive learning to approximate the classification loss, adding graph contrastive learning approaches as baselines. The authors did a nice job addressing reviewers’ comments and adding suggested experiments. Overall, the reviewers have positive assessment after author responses.
justification_for_why_not_higher_score: Modest novelty. 
justification_for_why_not_lower_score: The reviewers raised some concerns including better differentiation from graph domain adaptation(GraphDA), justification on contrastive learning to approximate the classification loss, adding graph contrastive learning approaches as baselines. The authors did a nice job addressing reviewers’ comments and adding suggested experiments. Overall, the reviewers have positive assessment after author responses.
Decision: Accept: poster