Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new perspective of graph learning by regarding node features, graph structures, and node labels as three different views. Different from the previous GNN models, the paper brings in a new latent variable to be learned and shared across all three views. Through a progressive optimization framework, the paper claims the new framework can obtain better computational and memory efficiency. 
strength_and_weaknesses: The regularization framework  $min_F tr(F^T L F) + ||F-Y||^2$, which is used to derive the graph learning models, is not new and it has been well known for decades. Early works such as manifold ranking [1] already used it; in the era of graph neural networks, as the paper mentioned, some works [2] proposed the unified optimization view of GNNs and also used it; in addition, from the perspective of graph signal processing, there are also some works [3,4] which used it as a special case. These works generally directly obtained the solution expression of F, and then use MLP(X) to take place of F. This paper differs in that it added || MLP(X) – F||^2 as another regularization and unroll the optimization process (the methodology of [4] is somehow similar but used a different objective). It does provide a new perspective and still looks a little bit trivial. The understanding of updating F as propagation and updating \Theta as pseudo label generation is interesting.

 However, the biggest problem is I did not see much advantage of the new formulation and optimization. The analysis of the time complexity if also doubtable.  If Multiview4GNN and APPNP only have dimension $c$ in time complexity because of the feature dimension transformation. Why do not we use a different step for SGC? $XW$ part can be first calculated (so the feature dimension is changed to c) and then do the feature aggregation with \sigma(A^n XW). Let us look back at Multiview4GNN, as other methods contains the computation of backpropagation, why is the backpropagation of the MLP in equation (9) not counted in Multiview4GNN? This MLP generally does not only have one layer so the dimension $d$ cannot be omitted, and the backpropagation also takes $t$ steps. 

[1] Zhou et al. Learning with Local and Global Consistency. In NIPS 2003.
[2] Ma et al. A Unified View on Graph Neural Networks as Graph Signal Denoising. In CIKM 2021.
[3] NT, Maehara. Revisiting graph neural networks: All we have is low-pass filters[J]. arXiv preprint arXiv:1905.09550, 2019.
[4] Chen et al. BiGCN: a bi-directional low-pass filtering graph neural network. arXiv preprint arXiv:2101.05519, 2021.


clarity,_quality,_novelty_and_reproducibility: Clarity: The description of the method is clear but the complexity comparison need to be improved. 
Novelty: it provides a new perspective but maybe not so useful. 
summary_of_the_review: In general, the paper has some interesting insights but the novelty is limited. And the contribution especially the analysis of the advantage of the complexity has some problems.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes using node features, graph structure, and node labels as three different views and maximizing their agreement.  Therefore, the loss function has three terms and gets optimized with an alternating optimization method. 
strength_and_weaknesses: **Strengths**
- The paper is well-written and clear.
- The method shows strong performance in very low node label scenarios.
- MULTIVIEW4GNN is memory and time efficient.

**Weaknesses** 
- D_A assumes that graphs have high homophily ratios (the smooth assumption). What would the model perform when the graph is heterophily?
- The performance is promising mostly in the scenarios where we have very few labels. 
- In my opinion, more rigorous experimentation is needed. The datasets are mostly small, easy to solve, and not diverse. 
- Is it mentioned anywhere in the paper what would be the case if there are no initial node features? 
clarity,_quality,_novelty_and_reproducibility: The manuscript is clear and written with good quality. \
The approach is relatively novel but it's limited. \
MULTIVIEW4GNN is relatively straightforward so reproducibility shouldn't be an issue here.
summary_of_the_review: Most of the points are mentioned in the strength and weaknesses section but in summary, the authors suggest that we can see different types of information as different "views" and try to maximize the agreement between these views with an alternation optimization algorithm. They show that the model is efficient and has good performance, mostly in very low node label scenarios. The datasets in the experiment section are small and not diverse.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a way to map node features $\boldsymbol{X_i}$, graph structure $\boldsymbol{A}$, and node label $\boldsymbol{Y_i}$ into a shared latent feature space $\boldsymbol{F_i}$. Each node $i$ is now represented by a vector $\boldsymbol{F_i}$, which has the same dimension as one-hot encoded label $Y_i$. $\boldsymbol{F_i}$ is treated as the soft pseudo label of node $i$.  

The $X_i \mapsto F_i $ mapping is done by an MLP with learnable $\Theta$ params. The graph structure $\boldsymbol{A}$ is enforced by imposing Laplacian smoothness of $\boldsymbol{F_i}$ against its neighbors $\boldsymbol{F_j}$. The node label $\boldsymbol{Y_i}$ is enforced by otimizing MSE between $\boldsymbol{Y_i}$ and $\boldsymbol{F_i}$ directly.

The paper also proposes an alternative training scheme to learn $\boldsymbol{F}$ and $\Theta$ with various training tricks such as: initial feature diffusion, MLP pre-training that uses only $\boldsymbol{Y_i}$, importance sampling based on $\boldsymbol{F_i}$ entropy, class balancing...

The experiment results show that the proposed methods yield good performance when the label rate is very low.


strength_and_weaknesses: Strength:
  - The paper is well presented, with good support materials in the Appendix, including speudo code, dataset statistics, and parameter setting.
  - The paper conducts relevant experiments, including ablation study and efficiency comparison. 

Weaknesses:
  - The whole training scheme is very similar to the Iscen et al. 2019 "Label Propagation for Deep Semi-supervised Learning" paper, including most of the important training tricks.
  - Experiment results only show good improvement on very low label setting (5-60 examples per class). For most of the time, the improvements are not statistically significant.
  - Hyper-param search space for the proposed method is much larger than baseline models. It includes $\lambda_1$ and $\lambda_2$, which together create 25 to 100 settings (Appendix E). On the contrary, the baseline models use fixed learning rate = 0.01. Apart from that, there are many training params that are used in the training algorithm (Appendix B) but are not mentioned in the paper. They are usually also hard to tune right, including $S$ pre-training steps, $p$ times $F$ update and $t$ times $\Theta$ update per iteration.
  - The paper claims that alternating optimization provides better flexibility than end-to-end training. This is not a well-supported statement. In practice, end-to-end training is much easier to setup, does not require training tricks, and depends less on hyper-params tuning.
clarity,_quality,_novelty_and_reproducibility: Clarity and Quality:
  - The paper is well presented with clear notations and references. The experiment setups are reasonable.

Novelty:
  - Not too much, because the whole training scheme is very similar to the Iscen et al. 2019 "Label Propagation for Deep Semi-supervised Learning" paper, including most of the important training tricks.
summary_of_the_review: The paper's novelty is not significantly strong. Most of its empirical positive results are from training tricks that are introduced by Iscen et al. 2019. The paper claims that alternative training is more flexible and more efficient than end-to-end training, which are not well-supported.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers node features, graph structure, and node labels as three views, and designs three loss components correspondingly. Moreover, the latent $F$ and the model parameters $\Theta$ are proposed to be optimized alternatingly to achieve efficiency and scalability.

strength_and_weaknesses: Strength:

1. The proposed multi-view loss (with *lazy propagation*) is an effective method to achieve the trade-off between performance and cost.
2. The authors conduct extensive ablation experiments to verify each model component.

Weaknesses:

1. The three losses after decoupling are not novel individually. Each component seems to be standard and well-studied in graph semi-supervised learning.
2. The pseudo-label with certainty and class-balancing in graph semi-supervised learning are also discussed in [1]. And there are other works highly related to the core ideas in this paper, e.g., [2,3,4], should be discussed.

References:

[1]. Self-Enhanced GNN: Improving Graph Neural Networks Using Model Outputs. Han Yang, Xiao Yan, Xinyan Dai, Yongqiang Chen, James Cheng. IJCNN '2021

[2]. Rethinking and Scaling Up Graph Contrastive Learning: An Extremely Efficient Approach with Group Discrimination. Yizhen Zheng, Shirui Pan, Vincent Cs Lee, Yu Zheng, Philip S. Yu. NeurIPS ’22.

[3]. Combining Label Propagation and Simple Models Out-performs Graph Neural Networks. Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, Austin R. Benson. ICLR ’21.

[4]. On Graph Neural Networks versus Graph-Augmented MLPs. Lei Chen, Zhengdao Chen, Joan Bruna. ICLR ’21.

clarity,_quality,_novelty_and_reproducibility: This paper clearly presents the multi-view loss design and provides solid experiment results. The technical novelty of the proposed method is a bit limited.
summary_of_the_review: This paper decouples the supervised loss and optimizes it alternatingly to achieve efficiency and scalability. However, the technical contribution is a bit limited.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper formulates the graph learning task from a multi-view perspective, where the node features $\mathbf{X}$, graph structure $\mathbf{A}$, and node labels $\mathbf{Y}$ are regarded as three views. Latent variables $\mathbf{F}$ and node feature encoder parameters $\mathbf{\Theta}$ are learned in an alternating scheme (allowing for a lazy propagation approach) minimizing three loss components corresponding to the three views.

In general, reviewers agreed that the paper is generally well-written, and offers interesting insight in its perspective of graph learning, and in its training scheme. However, there are a number of key issues raised by the reviewers, particularly:

- Novelty and comparison to related work: reviewers note the similarity to existing work, particularly [1], as well as [2,3]. While the authors note some differences such as the modified optimization approach in this paper, it would still be better to include a clearer description of the relationship between the current and the closely related works, particularly [1] which is only briefly mentioned in the paper.

- Evidence for benefits of proposed framework: regarding experimental performance, reviewers noted that the performance benefits are only relatively clear on low label rates, but not on high label rates. The authors note in their response that the method also provides improvements in computational and memory efficiency; however, one issue is that the (time / memory) efficiency comparison is only clearly favorable for Multiview4GNN-k (but not Full), while the accuracy performance of Multiview4GNN-k is not shown, making the benefits less clear.

- Other issues: reviewers noted that the proposed approach is not suitable for heterophilic graphs (as agreed by the authors), and significantly increases the amount of hyperparameter tuning (due to $\lambda_1, \lambda_2$ and potentially some other hyperparameters). The authors provide further details on the tuning procedure in their response, but there are still some slightly unclear parts (how $e$ and $k$ are chosen). 

In the end, PCs and AC agree that while the work has intriguing ideas, due to the above issues, the work is not yet ready for publication at ICLR. The reviews offer a number of helpful suggestions for improvement, so I encourage the authors to continue improving the paper based on the reviews for future submissions.


[1] Iscen et al., Label Propagation for Deep Semi-supervised Learning, CVPR 2019.
[2] Zhou et al. Learning with Local and Global Consistency. NIPS 2003. 
[3] Ma et al. A Unified View on Graph Neural Networks as Graph Signal Denoising. In CIKM 2021.
justification_for_why_not_higher_score: The 2 main issues were the novelty and need for clearer comparison to related work, and the need for stronger evidence of the benefits of the proposed framework, as mentioned in my review.
Decision: Reject