Kati8/ICLR_2023_2025 huggingface dataset:
authors: Ruiqi Feng
authors: Zhichao Hou
authors: Tyler Derr
authors: Xiaorui Liu
reviews: 
Comments: The paper reviews the estimation bias of several representative GNNs. It proposes a quasi-Newton iterative reweighted LS algorithm to optimize a GNN model, whose loss is based on the minimax concave penalty. This penalty alleviates the bias in typical GNN models and is also more robust.
Strengths: The paper proposes a quasi-Newton iterative reweighted LS algorithm to optimize a GNN model, whose loss is based on the minimax concave penalty. This penalty alleviates the bias in typical GNN models and is also more robust.
Weaknesses: 1. The paper is not well-written, with notations that are not always defined. E.g., the superscript (0) first appears in eq. (1) but has never been defined. In addition, the paper uses a strange definition of $\ell_1$ norm to mean either the usual 1-norm or the 2-norm. The $\ell_2$ "norm" in this paper is defined to be the square of the usual 2-norm. This is technically not a norm!

1. The contributions are weak as the formulation is essentially based on TWIRLS with the MCP penalty by Zhang, 2010. Robustness issues in Lasso regression have also been well-studied.

1. Most of the cited works are from 2021 or before. The authors are missing recent GNN robustness works like “On the robustness of graph neural diffusion to topology perturbations", NeurIPS 2022 and “Graph-coupled oscillator networks", ICLR 2022.

1. The experiments are limited with tests only on Cora and CiteSeer. Attacks like TDGIA and MetaGIA are not considered.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 2 fair
reviews: 
Comments: This paper proposes a robust model (RUNG) based on unbiased aggregation. Specifically, the authors first unify a few robust GNNs through the lens of $l_1$-based graph smoothing, while they still suffer from accuracy degradation when subjected to large attack budgets. To build a robust model against attacks with a large budget, authors introduce a robust and unbiased estimator by minimizing an objective function with a minimax concave penalty. Subsequently, an efficient Quasi-Newton iterative algorithm is used to optimize the objective function, from which RUNG is derived. Experimental results demonstrate that RUNG outperforms the baseline (defense) GNNs on CoraML and Citeseer, under most attack settings.
Strengths: - Overall, the paper is well-written.
- The unified view of robust estimation on prior GNNs is interesting.
- Authors have considered adaptive attacks.
Weaknesses: - The proposed method relies on a strong assumption of graph homophily, raising concerns about the performance of RUNG on heterophilic datasets. Thus, conducting additional experiments on heterophilic graphs is necessary. Otherwise, authors should explicitly discuss this limitation in the manuscript.
- Authors leverage QN-IRLS to approximate the inverse Hessian matrix to address the scalability issue. However, the experiments are only conducted on two small graphs, making it unclear whether the proposed approach would still work on large graphs.
- The accuracy improvement under small attack budgets is less convincing. For instance, the accuracy gap between RUNG and the runner-up model is often smaller than the standard deviation, as observed under local attacks with a 20% budget on CoraML. My concern is that RUNG might underperform the baselines with a new random seed, considering that the authors obtained averaged accuracy using only 5 different random splits.
- Authors have not performed sensitivity analyses on critical hyperparameters such as $\lambda$ and $\gamma$. It's unclear how these hyperparameters affect the performance of RUNG.
- Some recent defense models (e.g., [1, 2, 3]) are not compared in this work.
- Typo: the runner-up model is SoftMedian rather than RUNG-$l_1$ under the 5% attack budget in Table 2.

[1]: Li et al., "Reliable Representations Make A Stronger Defender: Unsupervised Structure Refinement for Robust GNN", KDD'22. \
[2]: Deng et al., “GARNET: Reduced-Rank Topology Learning for Robust and Scalable Graph Neural Networks”, LoG'22. \
[3]: Lei et al., "EvenNet: Ignoring Odd-Hop Neighbors Improves Robustness of Graph Neural Networks", NeurIPS'22.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 3 good
reviews: 
Comments: This paper analyzes the robust GNN models including SoftMedian, TWIRLS, and  ElasticGNN from a unified view of robust estimation. Building on this analysis, the authors introduce a robust unbiased aggregation method, further developing an efficient Quasi-Newton iterative reweighted least squares algorithm. However, the empirical validation seems somewhat limited, confined to two small graphs and a single attack setting.
Strengths: The paper provides a unified view of $l_1$-based robust graph signal smoothing for three robust GNNs.

It introduces an unbiased graph signal estimator, which is unfolded into feature aggregation layers, aiming to enhance the robustness of GNNs.
Weaknesses: 1. The definition of $l_1$-based graph smoothing is not clear. The MCP function $\rho_\gamma$ used in Equation 4 and Equation 7 is not defined.

2. The paper does not provide an analysis of the computational complexity of RUNG, leaving its efficiency and applicability to larger datasets, such as Ogbn[1], unclear. Additionally, numerical experiments are limited, being only applied to Cora-ML and Citeseer datasets. It would be beneficial to see how RUNG performs on larger-scale datasets.

3. The description of the attack setting in Section 4.1 lacks clarity. Could you provide more details on how the adaptive evasion attack is designed based [2] you referenced? Specifically, what type of perturbations are performed during the attack - are they feature perturbations, graph topology perturbations, or both?

4. The robustness validation of RUNG in the paper is notably insufficient, solely relying on the PGD attack.
A comprehensive assessment using various attacks, as detailed in references [3][4][5][6], is necessary for a credible demonstration of RUNG's robustness.

5. Implementation code was not provided.

[1]. Hu W, Fey M, Zitnik M, et al. Open graph benchmark: Datasets for machine learning on graphs[J]. Advances in neural information processing systems, 2020, 33: 22118-22133.

[2]. Mujkanovic F, Geisler S, Günnemann S, et al. Are Defenses for Graph Neural Networks Robust?[J]. Advances in Neural Information Processing Systems, 2022, 35: 8954-8968.

[3]. Zheng Q, Zou X, Dong Y, et al. Graph robustness benchmark: Benchmarking the adversarial robustness of graph machine learning[J]. arXiv preprint arXiv:2111.04314, 2021.

[4]. Chen Y, Yang H, Zhang Y, et al. Understanding and improving graph injection attack by promoting unnoticeability[J]. arXiv preprint arXiv:2202.08057, 2022.

[5].D. Zügner and S. Günnemann, “Adversarial attacks on graph neural networks via meta learning,” in Proc. Int. Conf. Learn.Representations, 2019.

[6].D. Zügner, A. Akbarnejad, and S. Günnemann, “Adversarial attacks on neural networks for graph data,” in Proc. Int. Conf. Knowl. Discovery Data Mining, 2018.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 3 good
reviews: 
Comments: The authors propose a unifying perspective on three "successful" graph defenses against adversarial structure perturbations. They argue that all models are instances of the so-called ElasticGNN. Moreover, the authors propose a practical optimization algorithm for the devised non-smooth objective that can approximate an L1 estimator. The resulting GNN-layer is called RUNG. The authors provide empirical evidence regarding the efficacy of their method - especially for large perturbation budgets.
Strengths: 1. Unified perspective on effective and established defenses
2. New optimization approach for an ElasticGNN derivative
3. Method shows strong empirical performance, especially for large perturbation budgets
4. The authors evaluate their defense using adaptive attacks and study the transferability of the perturbations between models
Weaknesses: 1. Empirical evaluation only using Cora ML and Citeseer is insufficient. Consider larger graphs like ogbn-arxiv as well.
1. Computational complexity and cost are neither discussed nor evaluated.
1. The authors should provide an ablation study on how the design choices affect the performance.
1. The authors should show that their method breaks. For example, complement Figure 2 with a setting where the method fails.
1. The authors state "The simulation in Figure 2 verifies that our proposed estimator (η(x) := ργ(∥x∥2)) recovers the true mean regardless of the increasing outlier ratio." Which is somewhat misleading since the asymptotically optimal breakdown point of a location estimator is 50% (under certain assumptions, e.g., without constraining the value range of the estimation).

Minor:
1. It would be interesting to see how the perturbations of RUNG transfer to the other defenses as well
1. Section 2.2 could benefit from some notes on the composition of multiple GNN layers
1. The dimension-wise median, is differentiable almost everywhere (similarly to sorting). There are only differentiability issues if the "center element" changes.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good