Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposed a confidence calibration method for graph neural networks (GNNs) for semi-supervised node classification problems. Based on the existing researches such as temperature scaling, the authors attempt to evaluate the confidence problem of GNNs and find a unique phenomenon that GNNs including GCN and GAT are less confident in their predictions which seem to be constant to other neural networks. From this observation, they proposed a confidence calibration procedure being aware of the topology of a graph structure and prove the accuracy-preserving property of their method. They also provided an effective optimization procedure for training their objective. Intensive experiments demonstrated the improvements of their proposed method with regard ECE and classification accuracy.
main_review: This is a well-written paper. The main question "will the current GNNs follow the same over-confident property as other neural networks?" of this paper is clear and interesting for not only me but other researchers. I enjoyed reading it. 

Strong points:
1. The authors firstly tackled the confidence calibration problem of the GNNs and report under-confident phenomena of GNN that have not been reported in the context of the confidence calibration of other neural networks.
2. To fix the under-confidence problem of GNN, they proposed a new calibration GNN model using an idea based on the homophily property. Although its structure is a bit simple and having no significant difference from the existing GCN, they proved the accuracy-preserving property that assures that no decline is caused by their CaGCN.
3. Intensive experiments showed better performances of CaGCN from other confidence calibration methods such as TS and MS.

Weak points:
1. In l.131 of p. 4, they employed the total variation to assess the confidence distribution and argued that "We can find that the total
variation of confidence does decrease after temperature scaling, which verifies our assumption." It is bit confusing for me. I suppose that if a GCN is well-calibrated, its confidence distribution is also calibrated. And thus a TS operation may not able to decrease the TV. Could you clarify your arguments more?
2. In Table 3, the authors report the node classification accuracy for four datasets. Although the accuracy-preserving property is proved in the aforementioned section, I cannot understand why accuracy of the TS-st and CaGCN-st were improved from the baseline. Is it caused by the calibration procedure or just the increasing number of parameters and layers of baselines?
3. In the context of graph neural networks, node classification is one of the fashionable problems but is a sub-class of the GNN task. Is it possible to enlarge the coverage of their proposed method to the other GNN tasks?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper analyzes the GNN prediction confidence, and reveals that the prediction confidence made by GNN is lower than the accuracy derived from the prediction confidence values, i.e., GNNs are under-confident.  This is different from the previously analyzed modern neural network models, which are often over-confident on the prediction (the prediction accuracy is lower than its confidence). To make the confidence values usable to reflect the true prediction confidence in safety-critical applications, a confidence calibration mechanism is proposed to make the confidence value be exactly equal to the true probability of getting a correct prediction for every node. The calibration follows the classic temperature scaling method, which has been used in the calibration of modern neural networks (ref [10]). 
main_review: It is a novel study on the confidence calibration of GNN models, given that the previous work focused on the calibration of modern neural networks (ref [10]).  However, the paper has to improve on several aspects. First, the description of the model has unclarified facts. Second, the evaluation should be conducted with statistical test to show the effectiveness of the proposed solution. Last, some writing inconsistency should be addressed.  More details can be found below.

In line 175, “, the order between classes of v_i and v’_i is not the same, implying that after calibration by CaGCN, the accuracy of original GCN cannot be preserved. ” But softmax is a monotonic function. The order between classes of v_i and v’_i should be the same.

Comparing to the classic temperature scaling method in the calibration of modern neural networks (ref [10]), the calibration in Eq(5) introduce different temperature t_i for different nodes. This is an interesting idea. As shown in figure 4, the weights W in Eq(5) are shared with the weights in Eq(3) in GCN for getting output V. In this way, t_i is in fact from t_i = \sigma+(v_i).  However, v_i is a vector. A softplus activation function applied on a vector v_i cannot result in a scalar t_i. Clarification is needed. 

Regarding the evaluation results, Table 3 reports the node classification accuracy and the variance (over 10 independent runs) of different models. The variance is quite large and the accuracy difference is small. Statistical test should be done to show if there is a significant improvement between CaGCN-st and baselines.  Table 2 doesn’t show the variance of ECE of different models. Given the small difference between CaGCN and other baselines, it would be necessary to present the variance and run statistical test as well.

Writing to correct: 
1.	In Definition 1,  label index in Y is from 0 to K.  But z_i is a vector with elements from 1 to K (should be from 0 to K). Please check also in many other places, the index is sometimes from 1 to K, and sometimes from 0 to K.
2.	Same for the index of node i. The index below Eq (2) is from 0 to N, for the total N nodes above Definition 1.  In Eq (2), i is indexed rom 1 to N. But in some other places, i is indexed from 0 to N.
3.  a l-layer GCN    etc

limitations_and_societal_impact: Not applicable
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper aims to propose a novel graph neural network model that is trustworthy via confidence calibration. Graph neural networks (GNNs) attract many researchers’ attention, which has achieved excellent achievements. However, can we trust the predicted results of GNNs? Inspired by previous research works showing that model deep neural networks are over-confident with predictions, the authors study the confidence calibration of GNNs. Through many experiments and analyses, they find that the traditional GNNs are under-confident. This observation is different from previous research works related to the confidence calibration of deep neural networks. Thus, how to do the confidence calibration for GNNs is important and highly desired. The authors propose a new calibration GNN model (CaGCN) to learn the calibration function for improving the confidence of GNNs. Specifically, they first validate the confidence distribution in a graph owns homophily characteristics. Then, they transform logits of GNNs to the calibrated confidence of each node. Owing to GNNs is used to deliver confidential information among different nodes during the transformation process, the process preserves the order between classes. Moreover, they apply the CaGCN to a self-training framework, which generates trustworthy pseudo labels for unlabeled data to improve model performance. Finally, the authors conduct extensive experiments to validate the effectiveness from two perspectives: calibration and accuracy.
main_review: Graph neural networks (GNNs) are popular in many domains. But can we trust the predicted result of GNNs? This research problem is interesting and useful. If we have strong and trustworthy GNNs, we can apply these GNNs to many real-world applications, especially safety-critical fields. Thus, from this perspective, we can find that the research topic of this paper is novel and important.

In addition, from the presentation perspective, this paper is well-organized and readers can follow the logic of the paper easily. Specifically, this paper has two steps: (1) finding the under-confident of GNNs, (2) proposing a calibrated GNNs (CaGNN) to make the predicted results more trustworthy. For the first step, the authors find the under-confident of GNNs via experiments, visualizations, and analyses. For the second step, the authors provide detailed explanations.

Moreover, from the experimental perspective, there are two advantages in this paper: (1) the authors conduct extensive experimental results to validate the effectiveness of their framework. Through these experiments, we find that the authors’ framework is not only more stable but also more accurate compared with other baseline models; (2) for each experiment, the authors provide detailed experimental settings including parameter setting, dataset, and so on, which is beneficial for other readers to reproduce experimental results.
limitations_and_societal_impact: There are some limitations in this paper: (1). The authors think that the predictive performance of GNNs is under-confident via analyzing the predictive performance of GNN and GAT. But I think the authors should test more graph neural networks to validate the conclusion. (2). The authors utilize GCNs to implement the confidence propagation part. They think the GCNs can preserve topology information and non-linear relations in graphs. But I think the motivation is weak. The authors should provide stronger reasons to support their model choice. For instance, can we utilize other non-linear graph models to capture these characteristics? (3). Figure 3 cannot reflect the confidence propagation, I suggest the authors replot this figure. (4). The authors can provide more application scenarios to highlight the social impact of the research.
review_title: null

Meta-review: This paper explores an interesting problem that lies at the intersection of two hot topics: graph neural networks and the reliability of NNs.
The observation that GNNs under-estimate themselves, as opposed to regular NNs, is also interesting, and the technical contributions to solving this problem are also solid, and thus can be expected to be of interest to the NeurIPS community.
Decision: Accept (Poster)