Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors propose Deep Graph Ensemble (DGE) to tackle the additional information available in higher order networks $-$ something which standard Message Passing GNNs methods are unable to process without loss of information. The authors propose to do this by leveraging an ensemble of standard MPNNs - which are used to solve the 3 design challenges associated with HONs  $-$ variable number of conditional nodes, different and potentially non-overlapping neighborhoods, and the importance of the nodes. The authors then present results of the DGE (with different aggregation strategies
strength_and_weaknesses: **Strengths:**
1. The idea towards using GNNs to solve problem on HONs is interesting as well as identifying different ways to use the HON to construct the inputs for the GNNs
2. Most sections of the paper are well written and clear

**Weaknesses, corresponding suggestions and questions**
1. Unfortunately, to the reader it appears like the work is present as the first work of using GNN's for higher order networks and has altogether missed relevant GNNs works on Higher order networks - both comparing against them  e.g. [1]. Please compare and contrast with this work - and other appropriate related works.
2. The paragraph on total model size comparison is unclear and appears to be unfair as well. Firstly, why limit the number of GNN layers in the baseline GNN models? The provided image in Fig 4 is also hard to read - and appears like until the budget is increased considerably the baseline models appear to do better.  Moreover, it is unclear how to select the right number of GNNs ($l$) to be used in the ensemble - can this be written as a function of the graph statistics?.
3. Standard GNNs are known to suffer in the link prediction task and are not representative baselines - as has been widely studied and documented. Please add comparison with a method such as [2]
4. What happens if you perform simple GNN ensembling - same input graph for all GNNs in the ensemble - but with neighborhood sampling (using MEAN, POOL, etc) without using your proposed training procedure? How does this compare?

**Minor:**
1. A drawback of the proposed approach - from the results in the appendix - it appears like that the DGE takes longer to converge - in terms of epochs how big is the difference? Are these all with the same parameter budgets as well?
2. DGE-pool and DGE-concat (Even without shared parameters) seems to perform worse than the baselines models in most cases - is there any reasoning for this? Please address this in Section 4.3

**References:**
1. Jin, Di, et al. "Graph Neural Network for Higher-Order Dependency Networks." Proceedings of the ACM Web Conference 2022. 2022.
2. Zhang, Muhan, and Yixin Chen. "Link prediction based on graph neural networks." Advances in neural information processing systems 31 (2018).
clarity,_quality,_novelty_and_reproducibility: Reproducibility - Code provided by authors 

Novelty and quality - Please see weakness 1

Clarity - Please see strength 2


summary_of_the_review: Unfortunately, in the current state , the weaknesses of the paper out weigh the strengths. However, I am happy to increase my scores if the authors can answer the questions appropriately.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an ensemble of GNNs that exploits variance in the neighborhood subspaces of nodes in graphs with higher-order dependencies and consistently outperforms baselines on semisupervised and supervised learning tasks.

The title and abstract looked interesting so I was expecting to see some interesting results, but the content is quite disappointing. The method/theory is too vague to be useful. Authors should answer my following concerns

1. Just given a normal graph from arbitrary domains, what do you do?
2. The conditional node, how do you decide if you want to condition?
3. What's the benefit of condition? The similar argument doesn't make sense
4. For regular gnn, how do you use ensemble to make them better?
strength_and_weaknesses: Title and abstract looked interesting



1. Just given a normal graph from arbitrary domains, what do you do?
2. The conditional node, how do you decide if you want to condition?
3. What's the benefit of condition? The similar argument doesn't make sense
4. For regular gnn, how do you use ensemble to make them better?
clarity,_quality,_novelty_and_reproducibility: N/A
summary_of_the_review: This paper proposes an ensemble of GNNs that exploits variance in the neighborhood subspaces of nodes in graphs with higher-order dependencies and consistently outperforms baselines on semisupervised and supervised learning tasks.

The title and abstract looked interesting so I was expecting to see some interesting results, but the content is quite disappointing. The method/theory is too vague to be useful. Authors should answer my following concerns

1. Just given a normal graph from arbitrary domains, what do you do?
2. The conditional node, how do you decide if you want to condition?
3. What's the benefit of condition? The similar argument doesn't make sense
4. For regular gnn, how do you use ensemble to make them better?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposed Deep Graph Ensemble (DGE) to learn effective representations of higher-order networks (HONS). HON is the networks that encode sequential higher-order dependencies. The authors conduct node classification and link prediction to evaluate DGE in comparison to graph neural networks on six datasets.
strength_and_weaknesses: **Strengths**

(1) This paper addresses interesting research topic : higher-order networks (HONs), which encoder sequential higher-order dependencies.

**Weakness**

(1) I think that the novelty of this paper is limited. The proposed method is similar to path-based graph neural networks (e.g., pathGCN [1]) or random-walk based graph neural networks (e.g., RAW-GNN [2]). 

(2) Also, the authors construct graph from the sequential data (paths) and then use GrowHON to construct HONs. I think HON seems very similar to paths (a sequence of nodes). Please clarify what the difference is between HON and paths.  

(3) Since the input datasets used in experiments are sequential data, it would be better to compare proposed DGE with the model for representing sequential data. Also, more comparison with path-based graph neural networks such as pathGCN [1] have to be included.

(4) Can you show qualitative analysis for DGE to verify why it is appropriate approach to deal with HON?

---

[1] Eliasof, Moshe, Eldad Haber, and Eran Treister. "pathGCN: Learning General Graph Spatial Operators from Paths." ICML 2022.

[2] Jin, Di, et al. "RAW-GNN: RAndom Walk Aggregation based Graph Neural Network." IJCAI 2022.
clarity,_quality,_novelty_and_reproducibility: **************Clarity**************

The paper is clearly written to understand overall method.

**************Quality**************

More baselines have to be included to show the significance of proposed models.

**********Novelty**********

The proposed method has a limited novelty. 

******************************Reproducibility******************************

The authors share their source code in the supplement. The paper has good reproducibility.
summary_of_the_review: Overall, I am leaning towards rejection. My major concern is the novelty and the lack of comparison with related works. If you address my concerns, I will raise my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper discusses and proposed one ensemble method of GNN. The key idea is based on the high-order relationship of a target node. The proposed idea is relatively easy to understand and implement, which indicates that future researchers can easily generalize it. The proposed methods also outperform baselines.
strength_and_weaknesses: The high-level idea is easy to understand, traditional GNN work can't handle the high-order relationship. In this work, the authors proposed a new ensemble method, which indicates that the aggregation of the target node's neighbor is a biased estimator of the aggregation of the target node. The proof is also technically sound. The paper also builds a connection between ensemble methods and GNN, which can be useful to large-scale graphs in future research.

Questions: 

Actually, the high-order relationship (i.e., high-order WL test) has been discussed in many previous works. Previous work, e.g., [1], [2], and other works, both discuss the high-order relationship. These works already show better performance compared with the 1-WL model(e.g., GIN, GCN, ...). However, the problem is that we don't have enough computational resources for the higher-order relationship(since we need to sample a subgraph for a target node, which is both time and space consuming). Thus, I wonder does the sampling part in this paper share the same limitations. Or maybe it's more efficient/faster than other high-order WL test based models.

Another concern is the expression power of the proposed model. More discussion regarding the expression power will be appreciated. By far we only know that it's a biased estimator. 

[1] Link Prediction Based on Graph Neural Networks
[2] Distance Encoding: Design Provably More Powerful Neural Networks for Graph Representation Learning
clarity,_quality,_novelty_and_reproducibility: This paper is easy to understand and well-organized. It's novel since few previous works discussed the correlation between high-order relationships and ensemble methods. The authors also provide the code and data they used.
summary_of_the_review: It's a good paper for me. The paper provides some insight and builds the connection between high-order relationships and ensemble methods. But I think more discussion is needed. Meanwhile, the experiment part only includes the 1-WL test model, which is not persuasive since it's obvious a higher-order relationship will have a better performance. Also, the runtime comparison may also show the advantage of the proposed model.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers were split about this paper and did not come to a consensus: on one hand they appreciated the clarity of the paper, on the other they had doubts about the baseline comparisons. After going through the paper and the discussion I have decided to vote to accept for the following reason: the main arguments against accepting made by the reviewers are 1) missing related work, 2) unfair total model size comparison, 3) missing ensemble comparison, 4) missing link prediction baseline, and 5) longer convergence. However, the authors responded to each one of these points convincingly. For 1), the reviewers pointed out HO-GNN, pathGCN, and RAW-GNN that should have been discussed, compared against, and hurts novelty. The authors pointed out the recent publication of these methods (all 2022, pathGCN and RAW-GNN were published in July, two months before the ICLR deadline). This addresses novelty. Further they have added a description of these and comparison against HO-GNN and pathGCN. This point seems resolved. For 2), the reviewers wondered if baselines are allowed to use more layers in order to match the ensemble size of the proposed approach, would they perform better? The authors show results for increased number of layers, which seem to make no difference or deteriorate the result. It seems that it is easy to overfit these datasets. For 3), the reviewers wonder if ensembling using neighborhood sampling would perform as well as the proposed method. The authors show in Appendix E that this is not the case. This result should really go in the main text somewhere, it’s a criticial point to show that the proposed method is doing more than just variance reduction via bagging. For 4), the reviewers pointed out that standard GNNs do poorly at link prediction and that a method such as SEAL should be compared against instead. The authors do this, and the method underperforms their approach. For 5), the authors agree that a downside of the approach is slower convergence. I’m glad the authors include a discussion on this. I would really like to see this mentioned in Section 5. For these reasons, the authors have convinced me the paper is worth accepting. Authors: you've already indicated that you've updated the submission to respond to reviewer changes, if you could double check their comments for any recommendation you may have missed on accident that would be great! The paper will make a great contribution to the conference!
justification_for_why_not_higher_score: The contribution of the work, while useful, is not ground-breaking.
justification_for_why_not_lower_score: The work has all the hallmarks of a useful contribution to the community, accepting it would improve ICLR.
Decision: Accept: poster