Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Authors provide the meta-learning framework for graph learning models that estimates the model performance based on the meta-graph features such as network structures. The proposed method optimizes the numerical performance values based on the top-1 probability based cross-entry loss function. Experimental results show that the proposed method can be robustly optimal for various datasets.
strength_and_weaknesses: * Strength
- The proposed meta-learning estimates the performance and leverages the predictive power for more robustly good model learning.
- The proposed method is a novel way of ensembling graph learning algorithms.
- The small initial overhead results in more reliable performance.

* Weakness
- The model performance matrix P is a random matrix since the performance is not deterministic. It would be great to study the sensitivity to the variance of the performance matrix P, not just the observation rate.
- It would be great if the proposed meta-learning is not only used for the model selection, but also for the insights about the performance characteristics of each graph model algorithm -- such as some algorithm working better for a homophily networks or clustering the similar flavor of algorithms, and so on.
- It would be interesting to compare the performance from MetaGL with the actual best performance. Since we have a fixed set of algorithms and perform training for each method, we are able to compare with the "optimal" selection.
clarity,_quality,_novelty_and_reproducibility: - The manuscript is well-written so that readers can easily follow.
- Through analysis has been made so the impact by the proposed meta-learning can be well understood.
- The idea of using graph structure as the meta-learner features is novel enough. It is worth being studied more in the research community.
- Without the shared performance matrix, it may not be trivial to reproduce the results. For the full reproducibility, all the hyperparameters chosen for performance matrix need to be shared.
summary_of_the_review: While GNN becomes popular in the ML area, it often requires a lot of resource to train on large datasets and chooses the best working model. The proposed meta-learning will provide the opportunity to produce more reliable results in a more productive fashion. Also, the thorough study for the given MetaGL provides understanding and confidence of the proposed model.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work focuses on graph model selection without training by matching the meta-feature and model representations. Extensive experiments are conducted to demonstrate the effective and efficiency of the proposed method. Basically, the overall idea has close relation to pioneering work MetaOD. By comparison, this work adopts the key idea to explore the graph model selection task with meta-knowledge.
strength_and_weaknesses: Strength:
1. The presentation and writing are good enough for clearly presenting the motivation and corresponding solution.
2. Meta-features for graph data are clearly defined by fully extracting the graph structural knowledge.
3. Extensive experiments are conducted to demonstrate the superiority of proposed method to state-of-the-art meta model selection method.

Weakness:
1. The novelty is limited to applying the principles of MetaOD to graph data by matching meta-features of graph data with model representations. It's interesting to compare the way to run "meta-learning" in this work with the optimization-based meta-learning methods like MAML [1]. The common thing shared by them is that finally selected model should have impressive prediction performance. The principles emphasized in this work pays attention to select the best one from the stored models which are trained on historical data. While optimization-based meta-learning can capture the unique knowledge existing in the input data just by fine-tuning models. I strongly suggest to have a discussion on the difference between them and make a comparison over their prediction performance.

2. Though the proposed method can fast search one model from the candidates, it's still difficult to believe that only comparing the meta-features extracted from graph structures can select the exact model for the input data. What if the extracted meta-feature can not represent the node features distribution differences existing in the graphs? Is it still the best choice for model selection? or is the learning-free model selection suitable to deal with the knowledge gap between the new graph and those graphs in the data pool?

3. Last but not least, I still have concern on the generalization of the proposed method. It's well known that graph consists of unique node sets. How is possible that the models trained on different graphs can be applied to a graph with totally different nodes? Maybe authors should present some concrete applications that are suitable to be applied the proposed method.  


References:
1. Finn, Chelsea, Pieter Abbeel, and Sergey Levine. "Model-agnostic meta-learning for fast adaptation of deep networks." In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
clarity,_quality,_novelty_and_reproducibility: The presentation is overall clear enough the catch the main idea. However, I have concerns on the novelty of the proposed method.
summary_of_the_review: This work attempts to facilitate model selection just by matching the meta-knowledge extracted from graph data and the trained models. It's a very interesting idea. But graph data raises unique challenges over the knowledge transferring across different graphs. Considering this point, it's difficult to believe the generalization of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of how to select an effective GL model for link prediction task on a new graph without training/evaluating process. To deal with this problem, the authors propose a meta-learning based matrix factorization (MF) approach MetaGL. MetaGL adopt the meta graph features to represent graphs, which can convert the transductive MF to an inductive manner for a new graph.
The proposed method is evaluated on 301 graphs via 5-fold cross validation with 412 candidate models.

strength_and_weaknesses: ### Strengths:
1. This paper first considers how to select an effective GL link prediction model for a new graph without searching process. It is a new and meaningful problem. 

2. The experiments are solid and can support the contributions: effectiveness of the meta-learning framework and global statistical features.

3. The construction of G-M Network makes sense and further improves the performance. 

### Weaknesses:
1. It seems that the extracting of some meta-features (e.g., triangles and Page Rank score) is time-consuming which limits the application on the large-scale graph.

2. The GRL methods in the model set are limited and the model set may not contain some new GRL methods in recent years.

3. It seems to be missing the training details and metrics of the link prediction subproblem.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and original.
summary_of_the_review: The paper considers a new problem of selecting effective GL models in an evaluation-free manner and proposes an effective approach to handle this problem with solid experiments. However, there are some issues with the scalability and limited GRL methods in the model set.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel framework to conduct evaluation-free model selection for graph learning models, i.e., without having to train/evaluate any model on the new graph. The framework learns latent embeddings for observed models and the corresponding performance on observed graphs. Moreover, meta-graph features are computed for each graph based on specific structures so that the graph similarity can be measured. The extensive experiments demonstrate the effectiveness of the proposed framework.
strength_and_weaknesses: Strengths:
1. The framework solves the novel problem of evaluation-free model selection in graph learning, which is crucial and challenging in real-world scenarios.
2. The experimental results are comprehensive and adequate. The authors also provide additional results regarding the effectiveness of each module and the efficiency of the framework.
3. The paper is well organized and easy to follow.




Weaknesses:

1. The structural information in each graph seems to be only incorporated via the meta-graph features. However, this can also be achieved by a specific GNN with a readout function. That being said, the improvement of the proposed framework is hardly believed to result from the incorporation of structural information.
2. The input for computing the latent embedding of a graph to function f() is not rational. If φ(m) is solely dependent on m, why the input of f() is a combination of m and φ(m), instead of only m or φ(m)? In this way, the information of m can also be preserved. On the other hand, combining both will seem redundant.	
3. The experimental part only includes the link prediction task. Although this task is important in graph mining, the proposed method should be capable of various graph mining tasks (considering that the scenario will generally be applications). However, the lack of evaluation on other graph mining tasks will cause the paper to be less convincing.
4. The paper lacks sufficient theoretical analysis to support the claim of efficacy and efficiency. The empirical results are plentiful, while further analysis would be beneficial.

clarity,_quality,_novelty_and_reproducibility: For the Clarity:
1. There are several typos and grammar errors in this paper. For example, “smarter, more efficient” should be “smarter and more efficient”. “is fully automatic it does not require” lacks the word “since”. “number of wedges” should be “number of edges”.

2. The paragraph starting with “Benchmark Data/Code:” appears twice in the main paper, which is redundant.

For the Quality and Novelty:
The quality seems to be solid and the studied problem is novel. The specific modules in the framework are less novel, while the framework is well designed.

For the Reproducibility:
Following the above, the authors claim that they release the code and datasets, which are not found in the paper. No links are provided.

summary_of_the_review: The paper solves a novel and crucial problem of evaluation-free model selection for graph learning. The experimental results are comprehensive. However, the major drawback lies in the lack of other graph mining tasks except for link prediction. The design is somewhat novel while also lacking theoretical analysis.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a meta-learning framework for evaluation-free model selection of graph link prediction models. The paper proposes a new problem where given a performance matrix of M models on N graphs, and a new unseen graph, select a model among M that performs best on the new graph (without training/evaluation). This becomes a cold-start matrix completion problem with a new column added. Structural meta-graph features, matrix factorization, and graph neural network inference on model-graph bipartite grap are used. The paper uses link prediction as the target graph learning task. Experiments on 301 graphs with 412 candidate models verify the effectiveness of the proposed framework.

I am recommending an accept for this paper, but strongly suggest the authors to address the following problems in their revision:

1. Although the proposed problem is new and interesting, the authors should better explain what the problem's real applications could be. That is, find a senario where people need to repeatedly train models on different graphs and generalize them to new graphs in a evaluation-free setting. As far as I know, industry applications highly focus on model accuracy as it is tied with revenue or user experience. It may be hard to trust a meta-learned strategy for model selection.

2. Holding similar concerns to Reviewer uUz4, without modeling node features, it is questionable whether only comparing the meta-features extracted from graph structures can select the best model for the input graph. Also, there are many transductive network embedding models such as node2vec/LINE in the model pool. The authors may need better explain how one can generalize node embeddings trained in one graph to another graph (with a different set of nodes) without re-training.

3. The authors claim choosing link prediction as a testbed because it is a key task for graph data. However, the link prediction baselines used are mostly outdated or are not specifically targeted for link prediction (known to perform poor on link prediction). The authors should include more modern link prediction baselines, such as SEAL and NBFNet, in their model pool. Without appropriate baselines, it is hard to verify the true effectiveness of the proposed framework.
justification_for_why_not_higher_score: The paper focuses on model selection for link prediction. However, the tested models exclude some of the most well-known link prediction baselines. Without appropriate baselines, it is hard to verify whether the proposed meta-learning framework can actually pick the best model for each graph.
Decision: Accept: poster