Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposed a new method to solve the GNN calibration problem. It identifies several factors that make the calibration different on GNNs from general multi-class classification tasks. Based on the factors a new calibration method is developed which produces nodewise temperature scaling using an attention-based architecture. Experiments demonstrate that the proposed GATS is accurate, data-efficient and expressive.
strengths_and_weaknesses: 
Strengths:

* A thorough study of the factors which impact the calibration error of GNNs
* A new temperature scaling model which considers the factors and mitigates the calibration problem in GNNs

Weakness:

* The writing is generally clear, but as a reader who is not familiar with calibration, it is not easy to understand how the calibration error is calculated and why the GCN model has a different pattern from general multi-class classification. I think it is better to explain it a little bit in the beginning of the method.

* Section 3 and Section 4 should be aligned better. It seems all factors in Sec 3 are used later, but not all of them are directly corresponding a parameter in Eq. (5) (e.g. h in Eq. (2)). Since there are too many new brough-in parameters in Eq(5), it should be explained more. Some notations can be reclaimed (\eta_{c_i}), some notations should be described more clearly (e.g. what does the h mean in \tau_i^h ).


questions: see above weakness questions.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Calibration of neural networks is an important area of research. This paper provides empirical insights on the miscalibration of GNNs for node classification, and then proposes an approach that takes all those factors into account to calibrate the GNN outputs.
strengths_and_weaknesses: Strengths:
+ Well written
+ Good, in-depth analysis of different factors
+ Tested on a suite of 7/8 datasets
Weaknesses:
- Some of the findings from the analyses in section 3 do not seem to match with the ablation study results.
- Some of the identified factors are somewhat expected, and not surprising
questions: Q1: The "distance to training nodes" seems to lead to miscalibration mostly on Cora and Citeseer, and not on the the other graphs. This is also true about the "general under-confident tendency". Is there a reason for this? Is there a correlation between the two factors? Are Cora and Citeseer different in any specific way that makes them more miscalibrated for nodes far away from the training set? 
More evidence may be necessary to back up these two factors.

Q2: From what I understand, "diversity of node distributions" is a direct effect of "general under-confident tendency". That is, if the model is under confident in its probability for the correct class, it means other classes have higher probabilities, and so the entropy is high. Is there anything novel about 3.2 then?

Q3: Among the five factors, I was mostly surprised/impressed by "relative confidence level". It seems like a good finding and the analysis in Fig 3 shows the importance of it. However, I was disappointed when I saw the ablation results; none of the degradations due to removing \delta c_i seem to be statistically significant. Is there a reason for the mismatch between what figure 3 shows regarding the importance of this factor, and what Table 2 shows regarding the unimportance of it? Is it because the proposed calibration algorithm doesn't utilize this factor in the best way?

Q4: Figs 1 and 3 seem to imply that the GCN outputs are well-calibrated for the physics dataset. However, Fig 4 implies that for a subset of the nodes (those with low relative confidence), the outputs are quite miscalibrated. Can you explain how these results can all hold simultaneously? Are there only very few nodes with low relative confidence for Physics (and that's why there is an extremely large std)?

Q5: This is more of an observation than a question, but "distance to training nodes" and "neighborhood similarity" are both factors that affect how "easy" the prediction is for a node, in the sense that, e.g., nodes with many neighbors of the same class are highly probable to belong to the same class (under homophily -- which the studied datasets all exhibit highly). Therefore, it is not quite surprising that these two factors play a role in miscalibration. Nevertheless, seeing empirical backup for the two is definitely nice.
limitations: 1- Among the five factors studied in the paper, the first one has been previously studied, the second one seems to be a direct effect of the first one, the third-one and the fifth one are not quite surprising, and the fourth one is quite interesting but doesn't seem to have a significant effect according to the ablation studies.

2- According to the ablation study, many of the sub-components of the proposed algorithm seem to have negligible effects on the results for many of the datasets (although it is partly understandable because some components may make up for the others).

Nevertheless, I believe a dedicated work for studying multiple factors about the miscalibration of GNNs and providing a calibration approach based on those factors could be beneficial to the community.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper considers confidence calibration for GNN node classification. It conducted a very throughout and careful empirical study of the factors in graph structure that may effect the calibration. It discovered 5 factors that contributing to miscalibration: general under-confidence, diversity of distribution, distance to training, relative confidence level, and neighborhood similarity. After demonstrating each factor independently, the authors proposed a temperature scaling method where each node is assigned with a different temperature, based on a formula that is inspired by the empirical analysis. The proposed GATS method is parameteric efficient but also captures the effects that influences confidence calibration. In particular, GATS out performs other calibration methods for most datasets.
strengths_and_weaknesses: Strength
  - the GNN calibration problem is important but underexplored
  - the empirical study is conducted very carefully, the results are very inspiring
  - the proposed method is very simple, interpretable and effective
  - the ablation study is very informative
  - the logic flow and writing is very clear

Weakness
  - the authors should explain how the GATS parameters are learned
  - the author could compare GATS & GaGCN in terms of the number of parameters
questions: N/A
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work studies prediction calibration for GNNs on node prediction. It performs empirical analysis to identify five factors that are correlated with the GNN prediction confidence. Then an attention-based temperature scaling method is proposed to calibrate GNNs. Experiments show that the proposed method achieves lower ECE compared to baselines.

strengths_and_weaknesses: Strengths:

Studying GNN calibration by analyzing its specific challenges that are different from CNNs is a good contribution.
The result in terms of ECE seems promising.
The paper presentation is mostly clear.

Weaknesses:

I have some concerns about the five factors proposed by the paper.

The definition and analysis for some of the five factors are a bit vague. For example, Section 3.2 is regarding the "diversity of node distributions" factor, which is analyzed through the "entropy distribution" plots in Figure 2. If I understand them correctly, in Figure 2, the entropy of each GNN output is computed, and then a density estimation based on all such entropy values is plotted. In this case, the name "node distribution" can be confusing because the factor is actually the distribution of GNN predictions on nodes. Node distribution should refer to the actual node features/labels, and it is a property of the data. Also, "diverse entropy implies uniform temperature scaling is insufficient" doesn't seem quite right. We can have a very large uniform temperature to make all predictions very uniform or a very small uniform temperature to make all predictions close to deterministic. Both cases will make the entropy distribution very concentrated. The latter can generate a plot close to the GNN case. Furthermore, the plots in the second row seem quite similar to me, which questions whether this factor is always necessary because it is dataset-dependent.

The analysis of these five factors is not thorough given some of these factors may be correlated. For example, the relative confidence level and the neighborhood similarity should be correlated. Intuitively, we are unlikely to have neighbors with the same prediction but very different confidence, or different predictions and both very high confidence. Then doing a more detailed analysis can help to better understand their correlations and importance and maybe disentangle the factors. In the current version, although the ablation study in Table 2 suggests all of the factors help with some ECE reduction, their significants are actually questionable. Like for $\delta c_i$, removing it only increases the ECE by less than 0.1% whereas the std is around 1%.

As a core contribution of the paper, what factors truly influence the GNN prediction and confidence is the key contribution. Ideally, we would want fewer factors and more disentangled factors. I feel the factor definition and analysis can be further improved.

questions: In figure 1, several plots show that the low confidence predictions achieve super high accuracy close to 1. Why is this the case?

limitations: N/A
review_title: null

Meta-review: metareview: This paper studies the calibration problem for GNN node classification. It identifies 5 factors contributing to miscalibration: general under-confidence, diversity of distribution, distance to training, relative confidence level, and neighborhood similarity. A temperature scaling method is proposed where each node is assigned with a different temperature. All reviewers vote for accept. However, multiple reviewers have raised concerns on the validity of the 5 factors. I encourage the authors to thoroughly address them in the revised version.
confidence: Less certain
Decision: Accept