Posterior Label Smoothing for Node Classification

## Abstract

Abstract Soft labels can improve the generalization of a neural network classifier in many domains, such as image classification. Despite its success, the current literature has overlooked the efficiency of label smoothing in node classification with graph-structured data.
In this work, we propose a simple yet effective label smoothing for the transductive node classification task.
We design the soft label to encapsulate the local context of the target node through the neighborhood label distribution.
We apply the smoothing method for seven baseline models to show its effectiveness. The label smoothing methods improve the classification accuracy in 10 node classification datasets in most cases.
In the following analysis, we find that incorporating global label statistics in posterior computation is the key to the success of label smoothing. Further investigation reveals that the soft labels mitigate overfitting during training, leading to better generalization performance.

## 1 Introduction

Adding a uniform noise to the ground truth labels has shown remarkable success in training neural networks for various classification tasks, including image classification and natural language processing (Szegedy et al., ; Vaswani et al., ; Müller et al., ; Zhang et al., ).
Despite its simplicity, label smoothing acts as a regularizer for the output distribution and improves generalization performance (Pereyra et al., ).
More sophisticated soft labeling approaches have been proposed based on the theoretical analysis of label smoothing (Li et al., ; Lienen and Hüllermeier, ).
However, the usefulness of smoothing has been under-explored in the graph domain, especially for node classification tasks.

In this work, we propose a *simple yet effective* smoothing method for transductive node classification tasks.
Inspired by the previous work suggesting predicting the local context of a node (Hu et al., ; Rong et al., ), such as subgraph prediction, helps to learn better representations, we propose a smoothing method that can potentially reflect the local context of the target node.
To encode the neighborhood information into the node label, we propose to relabel the node with a posterior distribution of the label given neighborhood labels.

Under the assumption that the neighborhood labels are conditionally independent given the label of the node to be relabeled, we factorize the likelihood into the product of conditional distributions between two adjacent nodes. To compute the posterior, we estimate the conditionals and prior from a graph’s global label statistics, making the posterior incorporate the local structure and global label distributions. Since the posterior obtained in this way does not preserve the ground truth label, we finally interpolate the posterior with the ground truth label, resulting in a soft label.

The posterior, however, may pose high variance when there are few numbers of neighborhood nodes.
To mitigate the issue with the sparse labels, we further propose iterative pseudo labeling to re-estimate the likelihood and prior based on the pseudo labels.
Specifically, we use the pseudo labels of validation and test sets to update the likelihood and prior, along with the ground truth labels of the training set.

We apply our smoothing method to seven different baseline neural network models, including MLP and variants of graph neural networks, and test its performance on 10 benchmark node classification datasets. Our empirical study finds that the soft label with iterative pseudo labeling improves the accuracy in 67 out of 70 cases despite its simplicity. We analyze the cases where the soft label decreases the accuracy and reveals characteristics of label distributions with which the soft labeling may not work. Further analysis shows that using local neighborhood structure and global label statistics is the key to its success. Through the loss curve analysis, we find that the soft label prevents over-fitting, leading to a better generalization performance in classification.

## 2 Related work

In this section, we introduce previous studies related to our method. We begin by discussing various node classification methods, followed by an exploration of the application of soft labels in model training.

### 2.1 Node classification

Graph structures are utilized in various ways for node classification tasks.
Some studies propose model frameworks based on the assumption of specific graph structures.
For example, GCN (Kipf and Welling, ), GraphSAGE (Hamilton et al., ), and GAT (Veličković et al., ) aggregate neighbor node representations based on the homophilic assumption.
To address the class-imbalance problem, GraphSMOTE (Zhao et al., ), ImGAGN (Qu et al., ), and GraphENS (Park et al., ) are proposed for homophilic graphs.
H2GCN (Zhu et al., ) and U-GCN (Jin et al., ) aggregate representations of multi-hop neighbor nodes to improve performance on heterophilic graphs.
Other studies concentrate on learning graph structure.
GPR-GNN (Chien et al., ) and CPGNN (Zhu et al., ) learn graph structures to determine which nodes to aggregate adaptively.
LDS (Franceschi et al., ), IDGL (Chen et al., ) and DHGR (Bi et al., ) take a graph rewiring approach, learning optimized graph structures to refine the given structure.
Besides, research such as ChebNet (Defferrard et al., ), APPNP (Gasteiger et al., ), and BernNet (He et al., ) focus on learning appropriate filters from the graph signals.

### 2.2 Classification with soft labels

Hinton et al. () demonstrate that a small student model trained using soft labels generated by the predictions of a large teacher model shows better performance than a model trained using one-hot labels. This approach, known as knowledge distillation (KD), is widely adopted in computer vision (Liu et al., ), natural language processing (NLP) (Jiao et al., ), and recommendation systems (Tang and Wang, ) for compression or performance improvement.
In the graph domain, applying KD has been considered an effective method to distill graph structure knowledge to student models.
TinyGNN (Yan et al., ) highlights that deep GNNs can learn information from further neighbor nodes than shallow GNNs, and it distills local structure knowledge from deep GNNs to shallow GNNs.
NOSMOG (Tian et al., ) improves the performance of multi-layer perceptrons (MLPs) on graph data by distilling graph structure information from a GNN teacher model.

On the other hand, simpler alternatives to generate soft labels are considered. The label smoothing (LS) (Szegedy et al., ) generates soft labels by adding uniform noise to the labels.
The benefits of LS have been widely explored.
Müller et al. () show that LS improves model calibration.
Lukasik et al. () establish a connection between LS and label-correction techniques, revealing LS can address label noise.
LS has been widely adopted in computer vision (Zhang et al., ) and NLP (Vaswani et al., ) studies, but has received little attention in the graph domain.

## 3 Method

Figure: Figure 1: Overall illustration of posterior node relabeling. To relabel the node label, we compute the posterior distribution of the label given neighborhood labels. Note that the node features are not considered in the relabeling process.
Refer to caption: x1.png

In this section, we describe our approach for label smoothing for the node classification problem and provide a new training strategy that iteratively refines the soft labels via pseudo labels obtained from the training procedure.

### 3.1 Posterior label smoothing

Consider a transductive node classification with graph $\mathcal{G}=(\mathcal{V},\mathcal{E},\bm{X})$, where $\mathcal{V}$ and $\mathcal{E}$ denotes the set of nodes and edges respectively, and $\bm{X}\in\mathbb{R}^{\lvert\mathcal{V}\rvert\times d}$ denotes $d$-dimensional node feature matrix. For each node $i$ in a training set, we have a label $y_{i}\in[K]$, where $K$ is the total number of classes. We use the notation $\bm{e}_{i}\in\{0,1\}^{K}$ for one-hot encoding of $y_{i}$, i.e., ${e}_{ik}=1$ if $y_{i}=k$ and $\sum_{k}{e}_{ik}=1$. In a transductive setting, we observe the connectivity between all nodes, including the test nodes, without having true labels of the test nodes.

We propose a simple and effective relabeling method to allocate a new label of a node based on the label distribution of the neighborhood nodes. Specifically, we consider the posterior distribution of node labels given their neighbors. Let $\mathcal{N}(i)$ be a set of neighborhood nodes of node $i$. If we assume the distribution of node labels depends on the graph connectivity, then the posterior probability of node $i$’s label, given its neighborhood labels, is

$$ $P(Y_{i}=k|\{Y_{j}=y_{j}\}_{j\in\mathcal{N}(i)})=\frac{{P}(\{Y_{j}=y_{j}\}_{j \in\mathcal{N}(i)}|Y_{i}=k){P}(Y_{i}=k)}{\sum_{\ell=1}^{K}{P}(\{Y_{j}=y_{j}\}_ {j\in\mathcal{N}(i)}|Y_{i}=\ell){P}(Y_{i}=\ell)}\;.$ (1) $$

The likelihood measures the joint probability of the neighborhood labels given the label of node $i$. To obtain the likelihood, we approximate the likelihood through the product of empirical conditional label distribution between adjacent nodes, i.e., $P(\{Y_{j}=y_{j}\}_{j\in\mathcal{N}(i)}|Y_{i}=k)\approx\prod_{j\in\mathcal{N}(i
)}{P}(Y_{j}=y_{j}|Y_{i}=k,(i,j)\in\mathcal{E})$, where $P(Y_{j}=y_{j}|Y_{i}=k,(i,j)\in\mathcal{E})$ is the conditional of between adjacent nodes. The conditional between adjacent nodes $i$ and $j$ with label $n$ and $m$, respectively, is estimated by

$$ $\hat{P}(Y_{j}=m|Y_{i}=n,(i,j)\in\mathcal{E})\coloneq\frac{\lvert\{(u,v)\mid y_ {v}=m,y_{u}=n,(u,v)\in\mathcal{E}\}\rvert}{\lvert\{(u,v)\mid y_{u}=n,(u,v)\in \mathcal{E}\}\rvert}\;.$ (2) $$

The prior distribution is also estimated from the empirical observations. We use the empirical proportion of label as a prior, i.e.,
$\hat{P}(Y_{i}=m)\coloneq{\lvert\{u\mid y_{u}=m\}\rvert}/{\lvert\mathcal{V}\rvert}$.
We also explore alternative designs for the likelihood and compare their performances in Section 4.2.

Note that, in implementation, all empirical distributions are computed only with the training nodes and their labels. The empirical distribution might be updated after node relabeling through the posterior computation, but we keep it the same throughout the relabeling process.

The posterior distribution can be used as a soft label to train the model, but we add uniform noise $\epsilon$ to the posterior to mitigate the risk of the posterior becoming overly confident if there are few or no neighbors. In addition, since the most probable label from the posterior might be different from the ground truth label, we interpolate the posterior with the ground truth label.
To this end, we obtain the soft label $\hat{\bm{e}}_{i}$ of node $i$ as

$$ $\hat{\bm{e}}_{i}=(1-\alpha)\tilde{\bm{e}}_{i}+\alpha\bm{e}_{i}\;,$ (3) $$

where $\tilde{{{e}}}_{ik}\propto P(Y_{i}=k\mid\{Y_{j}=y_{j}\}_{j\in\mathcal{N}(i)})+\beta\epsilon$. $\alpha$ and $\beta$ control the importance of interpolation and uniform noise. By enforcing $\alpha>1/2$, we can keep the most probable label of soft label the same as the ground truth label, but we find that this condition is not necessary in empirical experiments. We name our method as PosteL (Posterior Label smoothing). The detailed algorithm of PosteL is shown in Algorithm 1.

Figure: Algorithm 1 PosteL: Posterior label smoothing

### 3.2 Iterative pseudo labeling

Posterior relabeling is a method used to predict the label of a node based on the labels of its neighboring nodes. However, in transductive node classification tasks where train, validation, and test nodes coexist within the same graph, the presence of unlabeled nodes can hinder the accurate prediction of posterior labels. For instance, when a node has no labeled neighbors, the likelihood becomes one, and the posterior only relies on the prior. Moreover, in cases where labeled neighbors are scarce, noisy labels among the neighbors can significantly compromise the posterior distribution.
Such challenges are particularly prevalent in sparse graphs. For example, 26.35% of nodes in the Cornell dataset have no neighbors with labels. In such scenarios, the posterior relabeling can be challenging.

To address these limitations, we propose to update the likelihoods and priors through the pseudo labels of validation and test nodes.
We first train a graph neural network with the soft labels obtained via Equation 3 and predict the labels of validation and test nodes to obtain the pseudo labels. We choose the most probable label as a pseudo label from the prediction. We then update the likelihood and prior with the pseudo labels, leading to the re-calibration of the posterior smoothing and soft labels. By repeating training and re-calibration until the best validation loss of the predictor no longer decreases, we can maximize the performance of node classification.
We assume that if posterior label smoothing improves classification performance with a better estimation of likelihood and prior, the pseudo labels obtained from the predictor can benefit the posterior estimation as long as there are not many false pseudo labels.

## 4 Experiments

The experimental section is composed of two parts. First, we evaluate the performance of our method for node classification through various datasets and models. Second, we provide a comprehensive analysis of our method, investigating the conditions under which it performs well and the importance of each design choice.

### 4.1 Node classification

In this section, we assess the enhancements in node classification performance across a range of datasets and backbone models. Our aim is to validate the consistent efficacy of our method across datasets and backbone models with diverse characteristics.

#### Datasets

We assess the performance of our method across 10 node classification datasets. To examine the effect of our method on diverse types of graphs, we conduct experiments on both homophilic and heterophilic graphs. Adjacent nodes in a homophilic graph are likely to have the same label. Adjacent nodes in a heterophilic graph are likely to have different labels. For the homophilic datasets, we use five datasets: the citation graphs Cora, CiteSeer, and PubMed (Sen et al., ; Yang et al., ), and the Amazon co-purchase graphs Computers and Photo (McAuley et al., ). For the heterophilic datasets, we use five datasets: the Wikipedia graphs Chameleon and Squirrel (Rozemberczki et al., ), the Actor co-occurrence graph Actor (Tang et al., ), and the webpage graphs Texas and Cornell (Pei et al., ). Detailed statistics of each dataset are illustrated in Appendix A.

**Table 1: Classification accuracy on 10 node classification datasets. $\Delta$ represents the performance improvement achieved by PosteL compared to the backbone model trained with the ground truth label. All results of the backbone model trained with the ground truth label are sourced from He et al. ().**
|  | Cora | CiteSeer | PubMed | Computers | Photo | Chameleon | Actor | Squirrel | Texas | Cornell |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GCN | 87.14±1.01 | 79.86±0.67 | 86.74±0.27 | 83.32±0.33 | 88.26±0.73 | 59.61±2.21 | 33.23±1.16 | 46.78±0.87 | 77.38±3.28 | 65.90±4.43 |
| +LS | 87.77±0.97 | 81.06±0.59 | 87.73±0.24 | 89.08±0.30 | 94.05±0.26 | 64.81±1.53 | 33.81±0.75 | 49.53±1.10 | 77.87±3.11 | 67.87±3.77 |
| +KD | 87.90±0.90 | 80.97±0.56 | 87.03±0.29 | 88.56±0.36 | 93.64±0.31 | 64.49±1.38 | 33.33±0.78 | 49.38±0.64 | 78.03±2.62 | 63.61±5.57 |
| +PosteL | 88.56±0.90 | 82.10±0.50 | 88.00±0.25 | 89.30±0.23 | 94.08±0.35 | 65.80±1.23 | 35.16±0.43 | 52.76±0.64 | 80.82±2.79 | 80.33±1.80 |
| $\Delta$ | $+1.42({\color[rgb]{1,0,0}\uparrow})$ | $+2.24({\color[rgb]{1,0,0}\uparrow})$ | $+1.26({\color[rgb]{1,0,0}\uparrow})$ | $+5.98({\color[rgb]{1,0,0}\uparrow})$ | $+5.82({\color[rgb]{1,0,0}\uparrow})$ | $+6.19({\color[rgb]{1,0,0}\uparrow})$ | $+1.93({\color[rgb]{1,0,0}\uparrow})$ | $+5.98({\color[rgb]{1,0,0}\uparrow})$ | $+3.44({\color[rgb]{1,0,0}\uparrow})$ | $+14.43({\color[rgb]{1,0,0}\uparrow})$ |
| GAT | 88.03±0.79 | 80.52±0.71 | 87.04±0.24 | 83.32±0.39 | 90.94±0.68 | 63.13±1.93 | 33.93±2.47 | 44.49±0.88 | 80.82±2.13 | 78.21±2.95 |
| +LS | 88.69±0.99 | 81.27±0.86 | 86.33±0.32 | 88.95±0.31 | 94.06±0.39 | 65.16±1.49 | 34.55±1.15 | 45.94±1.60 | 78.69±4.10 | 74.10±4.10 |
| +KD | 87.47±0.94 | 80.79±0.60 | 86.54±0.31 | 88.99±0.46 | 93.76±0.31 | 65.14±1.47 | 35.13±1.36 | 43.86±0.85 | 79.02±2.46 | 73.44±2.46 |
| +PosteL | 89.21±1.08 | 82.13±0.64 | 87.08±0.19 | 89.60±0.29 | 94.31±0.31 | 66.28±1.14 | 35.92±0.72 | 49.38±1.05 | 80.33±2.62 | 80.33±1.81 |
| $\Delta$ | $+1.18({\color[rgb]{1,0,0}\uparrow})$ | $+1.61({\color[rgb]{1,0,0}\uparrow})$ | $+0.04({\color[rgb]{1,0,0}\uparrow})$ | $+6.28({\color[rgb]{1,0,0}\uparrow})$ | $+3.37({\color[rgb]{1,0,0}\uparrow})$ | $+3.15({\color[rgb]{1,0,0}\uparrow})$ | $+1.99({\color[rgb]{1,0,0}\uparrow})$ | $+4.89({\color[rgb]{1,0,0}\uparrow})$ | $-0.49({\color[rgb]{0,0,1}\downarrow})$ | $+2.12({\color[rgb]{1,0,0}\uparrow})$ |
| APPNP | 88.14±0.73 | 80.47±0.74 | 88.12±0.31 | 85.32±0.37 | 88.51±0.31 | 51.84±1.82 | 39.66±0.55 | 34.71±0.57 | 90.98±1.64 | 91.81±1.96 |
| +LS | 89.01±0.64 | 81.58±0.61 | 88.90±0.32 | 87.28±0.27 | 94.34±0.23 | 53.98±1.47 | 39.44±0.78 | 36.81±0.98 | 91.31±1.48 | 89.51±1.81 |
| +KD | 89.16±0.74 | 81.88±0.61 | 88.04±0.39 | 86.28±0.44 | 93.85±0.26 | 52.17±1.23 | 41.43±0.95 | 35.28±1.10 | 90.33±1.64 | 91.48±1.97 |
| +PosteL | 89.62±0.84 | 82.47±0.66 | 89.17±0.26 | 87.46±0.29 | 94.42±0.24 | 53.83±1.66 | 40.18±0.70 | 36.71±0.60 | 92.13±1.48 | 93.44±1.64 |
| $\Delta$ | $+1.48({\color[rgb]{1,0,0}\uparrow})$ | $+2.00({\color[rgb]{1,0,0}\uparrow})$ | $+1.05({\color[rgb]{1,0,0}\uparrow})$ | $+2.14({\color[rgb]{1,0,0}\uparrow})$ | $+5.91({\color[rgb]{1,0,0}\uparrow})$ | $+1.99({\color[rgb]{1,0,0}\uparrow})$ | $+0.52({\color[rgb]{1,0,0}\uparrow})$ | $+2.00({\color[rgb]{1,0,0}\uparrow})$ | $+1.15({\color[rgb]{1,0,0}\uparrow})$ | $+1.63({\color[rgb]{1,0,0}\uparrow})$ |
| MLP | 76.96±0.95 | 76.58±0.88 | 85.94±0.22 | 82.85±0.38 | 84.72±0.34 | 46.85±1.51 | 40.19±0.56 | 31.03±1.18 | 91.45±1.14 | 90.82±1.63 |
| +LS | 77.21±0.97 | 76.82±0.66 | 86.14±0.35 | 83.62±0.88 | 89.46±0.44 | 48.23±1.23 | 39.75±0.63 | 31.10±0.80 | 90.98±1.64 | 90.98±1.31 |
| +KD | 76.32±0.94 | 77.75±0.75 | 85.10±0.29 | 83.89±0.53 | 88.23±0.38 | 47.40±1.75 | 41.32±0.75 | 32.58±0.83 | 89.34±1.97 | 91.80±1.15 |
| +PosteL | 78.39±0.94 | 78.40±0.71 | 86.51±0.33 | 84.20±0.55 | 89.90±0.27 | 48.51±1.66 | 40.15±0.46 | 33.11±0.60 | 92.95±1.31 | 93.61±1.80 |
| $\Delta$ | $+1.43({\color[rgb]{1,0,0}\uparrow})$ | $+1.82({\color[rgb]{1,0,0}\uparrow})$ | $+0.57({\color[rgb]{1,0,0}\uparrow})$ | $+1.35({\color[rgb]{1,0,0}\uparrow})$ | $+5.18({\color[rgb]{1,0,0}\uparrow})$ | $+1.66({\color[rgb]{1,0,0}\uparrow})$ | $-0.04({\color[rgb]{0,0,1}\downarrow})$ | $+2.08({\color[rgb]{1,0,0}\uparrow})$ | $+1.50({\color[rgb]{1,0,0}\uparrow})$ | $+2.79({\color[rgb]{1,0,0}\uparrow})$ |
| ChebNet | 86.67±0.82 | 79.11±0.75 | 87.95±0.28 | 87.54±0.43 | 93.77±0.32 | 59.28±1.25 | 37.61±0.89 | 40.55±0.42 | 86.22±2.45 | 83.93±2.13 |
| +LS | 87.22±0.99 | 79.70±0.63 | 88.48±0.29 | 89.55±0.38 | 94.53±0.37 | 66.41±1.16 | 39.39±0.73 | 42.55±1.11 | 87.21±2.62 | 84.59±2.30 |
| +KD | 87.36±0.95 | 80.80±0.72 | 88.41±0.20 | 89.81±0.30 | 94.76±0.30 | 61.47±1.23 | 40.68±0.50 | 43.88±1.97 | 84.75±3.61 | 83.61±2.30 |
| +PosteL | 88.57±0.92 | 82.48±0.52 | 89.20±0.31 | 89.95±0.40 | 94.87±0.25 | 66.83±0.77 | 39.56±0.51 | 50.87±0.90 | 86.39±2.46 | 88.52±2.63 |
| $\Delta$ | $+1.90({\color[rgb]{1,0,0}\uparrow})$ | $+3.37({\color[rgb]{1,0,0}\uparrow})$ | $+1.25({\color[rgb]{1,0,0}\uparrow})$ | $+2.41({\color[rgb]{1,0,0}\uparrow})$ | $+1.10({\color[rgb]{1,0,0}\uparrow})$ | $+7.55({\color[rgb]{1,0,0}\uparrow})$ | $+1.95({\color[rgb]{1,0,0}\uparrow})$ | $+10.32({\color[rgb]{1,0,0}\uparrow})$ | $+0.17({\color[rgb]{1,0,0}\uparrow})$ | $+4.59({\color[rgb]{1,0,0}\uparrow})$ |
| GPR-GNN | 88.57±0.69 | 80.12±0.83 | 88.46±0.33 | 86.85±0.25 | 93.85±0.28 | 67.28±1.09 | 39.92±0.67 | 50.15±1.92 | 92.95±1.31 | 91.37±1.81 |
| +LS | 88.82±0.99 | 79.78±1.06 | 88.24±0.42 | 88.39±0.48 | 93.97±0.33 | 67.90±1.01 | 39.72±0.70 | 53.39±1.80 | 92.79±1.15 | 90.49±2.46 |
| +KD | 89.33±1.03 | 81.24±0.85 | 89.85±0.56 | 87.88±1.11 | 94.23±0.51 | 66.76±1.31 | 42.00±0.63 | 53.26±1.07 | 94.26±1.48 | 88.52±1.97 |
| +PosteL | 89.20±1.07 | 81.21±0.64 | 90.57±0.31 | 89.84±0.43 | 94.76±0.38 | 68.38±1.12 | 40.08±0.69 | 53.54±0.79 | 93.28±1.31 | 92.46±0.99 |
| $\Delta$ | $+0.63({\color[rgb]{1,0,0}\uparrow})$ | $+1.09({\color[rgb]{1,0,0}\uparrow})$ | $+2.11({\color[rgb]{1,0,0}\uparrow})$ | $+2.99({\color[rgb]{1,0,0}\uparrow})$ | $+0.91({\color[rgb]{1,0,0}\uparrow})$ | $+1.10({\color[rgb]{1,0,0}\uparrow})$ | $+0.16({\color[rgb]{1,0,0}\uparrow})$ | $+3.39({\color[rgb]{1,0,0}\uparrow})$ | $+0.33({\color[rgb]{1,0,0}\uparrow})$ | $+1.09({\color[rgb]{1,0,0}\uparrow})$ |
| BernNet | 88.52±0.95 | 80.09±0.79 | 88.48±0.41 | 87.64±0.44 | 93.63±0.35 | 68.29±1.58 | 41.79±1.01 | 51.35±0.73 | 93.12±0.65 | 92.13±1.64 |
| +LS | 88.80±0.92 | 80.37±1.05 | 87.40±0.27 | 88.32±0.38 | 93.70±0.21 | 69.58±0.94 | 39.60±0.53 | 52.39±0.60 | 91.80±1.80 | 90.49±1.48 |
| +KD | 87.78±0.99 | 81.20±0.86 | 87.59±0.41 | 87.35±0.40 | 93.96±0.40 | 67.75±1.42 | 41.04±0.89 | 51.25±0.83 | 93.61±1.31 | 90.33±2.30 |
| +PosteL | 89.39±0.92 | 82.46±0.67 | 89.07±0.29 | 89.56±0.35 | 94.54±0.36 | 69.65±0.83 | 40.40±0.67 | 53.11±0.87 | 93.93±1.15 | 92.95±1.80 |
| $\Delta$ | $+0.87({\color[rgb]{1,0,0}\uparrow})$ | $+2.37({\color[rgb]{1,0,0}\uparrow})$ | $+0.59({\color[rgb]{1,0,0}\uparrow})$ | $+1.92({\color[rgb]{1,0,0}\uparrow})$ | $+0.91({\color[rgb]{1,0,0}\uparrow})$ | $+1.36({\color[rgb]{1,0,0}\uparrow})$ | $-1.39({\color[rgb]{0,0,1}\downarrow})$ | $+1.76({\color[rgb]{1,0,0}\uparrow})$ | $+0.81({\color[rgb]{1,0,0}\uparrow})$ | $+0.82({\color[rgb]{1,0,0}\uparrow})$ |

#### Experimental setup and baselines

We evaluate the performance of PosteL across various backbone models, ranging from MLP, which ignores underlying structure between nodes, to six widely used graph neural networks: GCN (Kipf and Welling, ), GAT (Veličković et al., ), APPNP (Gasteiger et al., ), ChebNet (Defferrard et al., ), GPR-GNN (Chien et al., ), and BernNet (He et al., ).
We follow the experimental setup and backbone implementations of He et al. (). Specifically, we use fixed 10 train, validation, and test splits with ratios of 60%/20%/20%, respectively, and measure the accuracy at the lowest validation loss. We report the mean performance and 95% confidence interval. The model is trained for 1,000 epochs, and we apply early stopping when validation loss does not decrease during the last 200 epochs. For all models, the learning rate is validated within $\{0.001,0.002,0.01,0.05\}$, and weight decay within $\{0,0.0005\}$. The search spaces of the other model-dependent hyperparameters are provided in Appendix B. We validate two hyperparameters for PosteL: posterior label ratio $\alpha\in\{0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0\}$ and uniform noise ratio $\beta\in\{0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9\}$.

We compare our method with two different soft labeling methods, including label smoothing (LS) (Szegedy et al., ) and knowledge distillation (KD) (Hinton et al., ).
For KD, we use an ensemble of average logits from three independently trained GNNs as a teacher model.
The temperature parameter for KD is set to four following the previous work (Stanton et al., ).

#### Results

In Table 1, the classification accuracy and 95% confidence interval for each of the seven models across the 10 datasets are presented. In most cases, PosteL outperforms baseline methods across various settings, demonstrating significant performance enhancements and validating its effectiveness for node classification. Specifically, our method performs better in 67 cases out of 70 settings against the ground truth labels. Furthermore, among these settings, 39 cases show improvements over the 95% confidence interval. Notably, on the Cornell dataset with the GCN backbone, our method achieves a substantial performance enhancement of 14.43%.
When compared to the other soft label methods, PosteL performs better in most cases as well. The knowledge distillation method shows comparable performance with the GPR-GNN baseline, but even in this case, there are marginal differences between the two approaches.

### 4.2 Analysis

In this section, we analyze the main experimental result from various perspectives, including design choices, ablations, and computational complexity.

#### Learning curves analysis

Figure: Figure 2: Loss curve of GCN trained on PosteL labels and ground truth labels on the Squirrel dataset.
Refer to caption: x2.png

We investigate the influence of soft labels on the learning dynamics of GNNs by visualizing the loss function of GCNs with and without soft labels.
Figure 2 visualizes the differences between training, validation, and test losses with and without the PosteL labels on the Squirrel dataset.
From the training loss, we observe that the cross entropy with the PosteL labels converges to a higher loss than that with the ground truth labels. The curve shows that predicting soft labels is more difficult than predicting ground truth labels.
On the other hand, the validation and test losses with the soft labels converge to lower losses than those with the ground truth labels. Especially, up to 200 epochs, we observe that no overfitting happens with the soft labels.
We conjecture that predicting the correct PosteL label implies the correct prediction of the local neighborhood structure since the PosteL labels contain the local neighborhood information of the target node. Hence, the model trained with PosteL labels could have a better understanding of the graph structure, potentially leading to a better generalization performance. A similar context prediction approach has been proposed as a pertaining method in previous studies (Hu et al., ; Rong et al., ).
We provide the same curves for all datasets in Figure 8 and Figure 10 in Appendix D. All curves across all datasets show similar patterns.

#### Influence of neighborhood label distribution

Figure: (a) PubMed
Refer to caption: x5.png

Our approach assumes that the distribution of neighborhood labels varies depending on the label of the target node. If there are no significant differences between the neighborhood’s label distributions, the posterior relabeling assigns similar soft labels for all nodes, making our method similar to the uniform noise method.

Figure 3 shows the neighborhood label distribution
for three different datasets.
In the PubMed and Texas datasets, we observe a notable difference in the conditionals when w.r.t the different labels of a target node. The PubMed dataset is known to be homophilic, where nodes with the same labels are likely to be connected, and the conditional distributions match the characteristics of the homophilic dataset.
The Texas dataset, a heterophilic dataset, shows that some pairs of labels more frequently appear in the graph. For example, when the target node has the label of 1, their neighborhoods will likely have the label of 5.
On the other hand, the conditionals of the Actor dataset do not vary much regarding the label of the target node.
In such a case, the prior will likely dominate the posterior. Therefore, the posterior may not provide useful information about neighborhood nodes, potentially limiting the effectiveness of our method.
This analysis aligns with the results in Table 1, where the improvement of the Actor dataset is less significant than those of the PubMed and Texas datasets.
The neighborhood label distributions for all datasets are provided in Figure 11 and Figure 12 in Appendix E.

#### Visualization of node embeddings

Figure 5 presents the t-SNE (Van der Maaten and Hinton, ) plots of node embeddings from the GCN with the Chameleon and Squirrel datasets. The node color represents the label. For each dataset, the left plot visualizes the embeddings with the ground truth labels, while the right plot visualizes the embeddings with PosteL labels. The visualization shows that the embeddings from the soft labels form tighter clusters compared to those trained with the ground truth labels.
This visualization results coincide with the t-SNE visualization of the previous work of Müller et al. ().

Figure: (a) Chameleon
Refer to caption: x8.png

#### Effect of iterative pseudo labeling

We evaluate the impact of iterative pseudo labeling by analyzing the loss curve at each iteration.
Figure 6 illustrates the loss curves for different iterations on the Cornell dataset.
As the iteration progresses, the validation and test losses after 1,000 epochs keep decreasing.
In this example, the model performs best after four iteration steps.
We find that the best validation performance is obtained from 1.13 iterations on average. We provide the average iteration steps in Appendix C used to report the results in Table 1.

Figure: Figure 6: The impact of the iterative pseudo labeling: loss curves of GCN on the Cornell dataset.
Refer to caption: x12.png

#### Design choices of likelihood model

**Table 2: Classification accuracy with various choices of likelihood model. PosteL (local-1) and (local-2) indicate that the likelihood is estimated within one- and two-hop neighbors of a target node, respectively. PosteL (norm.), shortened from PosteL (normalized), indicates that the likelihood is normalized based on the degree of a node.**
|  | Cora | CiteSeer | Computers | Photo | Chameleon | Actor | Texas | Cornell |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GCN | 87.14±1.01 | 79.86±0.67 | 83.32±0.33 | 88.26±0.73 | 59.61±2.21 | 33.23±1.16 | 77.38±3.28 | 65.90±4.43 |
| +PosteL (local-1) | 88.26±1.07 | 81.42±0.46 | 89.08±0.31 | 93.61±0.40 | 65.36±1.25 | 33.48±1.03 | 79.02±3.11 | 71.97±4.10 |
| +PosteL (local-2) | 88.62±0.97 | 81.92±0.42 | 88.62±0.48 | 93.95±0.37 | 65.10±1.55 | 34.63±0.46 | 78.20±2.79 | 73.28±4.10 |
| +PosteL (norm.) | 89.00±0.99 | 81.86±0.70 | 89.30±0.39 | 94.13±0.39 | 66.00±1.14 | 34.90±0.63 | 80.33±2.95 | 80.00±1.97 |
| +PosteL | 88.56±0.90 | 82.10±0.50 | 89.30±0.23 | 94.08±0.35 | 65.80±1.23 | 35.16±0.43 | 80.82±2.79 | 80.33±1.80 |

We explore various valid design choices for likelihood models. We introduce two variants of PosteL: PosteL (normalized) and PosteL (local-$H$).
In Equation 2, each edge has an equal contribution to the conditional. The conditional can be influenced by a few numbers of nodes with many connections. To mitigate the importance of high-degree nodes, we alternatively test the following conditional, denoted as PosteL (normalized):

$$ $\hat{P}^{\text{norm.}}(Y_{j}=m|Y_{i}=n,(i,j)\in\mathcal{E})\coloneq\frac{\sum_ {y_{u}=n}\sum_{v\in\mathcal{N}(u)}\frac{1}{\lvert\mathcal{N}(u)\rvert}\cdot \mathbbm{1}[y_{v}=m]}{\lvert\{y_{u}=n\mid u\in\mathcal{V}\}\rvert}\;,$ $$

where $\mathbbm{1}$ is an indicator function.

In PosteL (local-$H$), we estimate the likelihood and prior distributions of each node from their respective $H$-hop ego graphs. Specifically, the likelihood of PosteL (local-$H$) is formulated as follows:

$$ $\displaystyle\hat{P}^{\text{local-$H$}}(Y_{j}=m|Y_{i}=n,(i,j)\in\mathcal{E}) \coloneq\frac{\lvert\{(u,v)|y_{v}=m,y_{u}=n,(u,v)\in\mathcal{E},u,v\in\mathcal {N}^{(H)}(i)\}\rvert}{\lvert\{(u,v)|y_{u}=n,(u,v)\in\mathcal{E},u,v\in\mathcal {N}^{(H)}(i)\}\rvert}\;,$ $$

where $\mathcal{N}^{(H)}(i)$ denotes the set of neighborhoods of node $i$ within $H$ hops. Through the local likelihood, we test the importance of global and local statistics in the smoothing process.

Table 2 shows the comparison between these variants. The likelihood with global statistics, e.g., PosteL and PosteL (normalized), performs better than the local likelihood methods, e.g., PosteL (local-1) and PosteL (local-2) in general, highlighting the importance of simultaneously utilizing global statistics. Especially in the Cornell dataset, a significant performance gap between PosteL and PosteL (local) is observed. PosteL (normalized) demonstrates similar performance to PosteL.

#### Ablation studies

**Table 3: Ablation studies on three main components of PosteL on GCN. PS stands for posterior label smoothing without uniform noise, UN stands for uniform noise added to the posterior distribution, and IPL stands for iterative pseudo labeling. We use ✓to indicate the presence of the corresponding component in training and ✗ to indicate its absence. IPL with one indicates the performance with a single pseudo labeling step.**
| PS | UN | IPL | Cora | CiteSeer | Computers | Photo | Chameleon | Actor | Texas | Cornell |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ✗ | ✗ | ✗ | 87.14±1.01 | 79.86±0.67 | 83.32±0.33 | 88.26±0.73 | 59.61±2.21 | 33.23±1.16 | 77.38±3.28 | 65.90±4.43 |
| ✓ | ✗ | ✗ | 88.11±1.22 | 80.95±0.52 | 88.86±0.40 | 93.55±0.30 | 64.53±1.23 | 33.48±0.62 | 78.52±2.46 | 68.52±4.43 |
| ✗ | ✓ | ✗ | 87.77±0.97 | 81.06±0.59 | 89.08±0.30 | 94.05±0.26 | 64.81±1.53 | 33.81±0.75 | 77.87±3.11 | 67.87±3.77 |
| ✓ | ✗ | ✓ | 88.56±0.90 | 81.64±0.57 | 88.70±0.27 | 93.70±0.37 | 64.25±1.93 | 34.71±0.76 | 80.82±2.79 | 80.16±1.97 |
| ✓ | ✓ | ✗ | 87.83±0.92 | 82.09±0.44 | 89.17±0.31 | 93.98±0.34 | 66.19±1.60 | 34.91±0.48 | 79.51±3.61 | 71.97±5.25 |
| ✓ | ✓ | 1 | 87.96±0.90 | 82.33±0.52 | 89.16±0.30 | 94.06±0.27 | 65.89±1.51 | 34.96±0.48 | 80.16±2.79 | 80.33±1.97 |
| ✓ | ✓ | ✓ | 88.56±0.90 | 82.10±0.50 | 89.30±0.23 | 94.08±0.35 | 65.80±1.23 | 35.16±0.43 | 80.82±2.79 | 80.33±1.80 |

To highlight the importance of each component in PosteL, we perform ablation studies on three components: posterior smoothing without uniform noise (PS), uniform smoothing (UN), and iterative pseudo labeling (IPL). Table 3 presents the performance results from the ablation studies.

The configuration with all components included achieves the highest performance, underscoring the significance of each component. The iterative pseudo labeling proves effective across almost all datasets, with a particularly notable impact on the Cornell dataset. However, even without iterative pseudo labeling, the performance remains competitive, suggesting that its use can be decided based on available resources. Additionally, incorporating uniform noise into the posterior distribution enhances performance on several datasets. Moreover, PosteL consistently outperforms the approach using only uniform noise, a widely used label smoothing method.

#### Complexity analysis

The computational complexity of calculating the posterior label is $O(|\mathcal{E}|K)$. Since the labeling is performed before the learning stage, the time required to process the posterior label can be considered negligible. The training time increases linearly w.r.t the number of iterations with the pseudo labeling. However, experiments show that an average of 1.13 iterations is needed, making our approach feasible without having too many iterations. The proof of computational complexity is in Appendix C.

### 4.3 Training with sparse labels

**Table 4: Accuracy of the model trained with sparse labels. The ratio indicates the percentage of nodes used for training.**
|  | ratio | Cora | CiteSeer | Computers | Photo | Chameleon | Actor | Texas | Cornell |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GCN | 5% | 80.03±0.57 | 70.19±0.49 | 85.32±0.60 | 92.39±0.24 | 45.96±2.48 | 25.20±0.83 | 54.23±6.35 | 50.58±5.84 |
| +PosteL | 80.42±0.64 | 71.08±0.65 | 86.22±0.45 | 92.66±0.21 | 51.35±1.19 | 27.04±0.51 | 57.52±1.97 | 50.36±3.43 |  |
| GCN | 10% | 83.05±0.51 | 72.09±0.46 | 86.68±0.59 | 92.49±0.29 | 51.55±1.67 | 26.78±0.68 | 60.08±2.56 | 53.64±3.49 |
| +PosteL | 83.50±0.36 | 73.76±0.26 | 87.47±0.37 | 92.88±0.30 | 56.33±1.86 | 28.07±0.19 | 61.63±2.87 | 57.75±1.86 |  |
| GCN | 20% | 84.46±0.68 | 73.93±0.69 | 87.12±0.33 | 93.24±0.33 | 55.57±1.18 | 27.42±0.76 | 63.33±2.05 | 52.91±2.65 |
| +PosteL | 85.32±0.65 | 75.73±0.39 | 87.77±0.19 | 93.47±0.18 | 60.91±1.07 | 29.23±0.50 | 64.87±2.74 | 56.92±2.39 |  |
| GCN | 30% | 85.76±0.46 | 75.56±0.44 | 87.02±0.49 | 93.14±0.27 | 59.41±1.08 | 28.81±0.50 | 65.64±4.36 | 60.40±3.96 |
| +PosteL | 86.04±0.37 | 77.30±0.65 | 88.09±0.31 | 93.47±0.27 | 63.64±0.98 | 30.21±0.39 | 69.80±3.86 | 64.95±2.08 |  |
| GCN | 40% | 86.32±0.43 | 77.17±0.52 | 87.88±0.58 | 93.76±0.20 | 60.44±1.20 | 29.71±0.72 | 67.88±2.47 | 62.00±2.12 |
| +PosteL | 86.23±0.37 | 79.22±0.32 | 88.21±0.29 | 93.99±0.24 | 63.82±1.44 | 31.05±0.40 | 73.76±2.59 | 67.41±4.71 |  |

Our method relies on global statistics estimated from training nodes. However, in scenarios where training data is sparse, the estimation of global statistics can be challenging. To assess the effectiveness of the
label smoothing from graphs with sparse labels, we conduct experiments with varying sizes of a training set. We vary the size of the training set from 5% to 40% of an entire dataset and conduct the classification experiments with the same setting used in the previous section. The percentage of validation nodes is set to 20% for all experiments. Table 4 provides the classification performance with sparse labels. Even in scenarios with sparse labels, PosteL consistently outperforms models trained on ground truth labels in most cases. These results show that our method can effectively capture global statistics even when training data is limited.

## 5 Conclusion

In this paper, we proposed a novel posterior label smoothing method, PosteL, designed to enhance node classification performance in graph-structured data. Our approach integrates both local neighborhood information and global label statistics to generate soft labels, thereby improving generalization and mitigating overfitting. Extensive experiments across various datasets and models demonstrated the effectiveness of PosteL, showing significant performance gains compared to baseline methods despite its simplicity.

## Appendix A Dataset statistics

We provide detailed statistics about the dataset used for the experiments in Table 5.

**Table 5: Statistics of the dataset utilized in the experiments.**
| Dataset | # nodes | # edges | # features | # classes |
| --- | --- | --- | --- | --- |
| Cora | 2,708 | 5,278 | 1,433 | 7 |
| CiteSeer | 3,327 | 4,552 | 3,703 | 6 |
| PubMed | 19,717 | 44,324 | 500 | 3 |
| Computers | 13,752 | 245,861 | 767 | 10 |
| Photo | 7,650 | 119,081 | 745 | 8 |
| Chameleon | 2,277 | 31,396 | 2,325 | 5 |
| Actor | 7,600 | 30,019 | 932 | 5 |
| Squirrel | 5,201 | 198,423 | 2,089 | 5 |
| Texas | 183 | 287 | 1,703 | 5 |
| Cornell | 183 | 277 | 1,703 | 5 |

## Appendix B Detailed experimental setup

In this section, we provide the computer resources and search space for model hyperparameters.
Our experiments are executed on AMD EPYC 7513 32-core Processor and a single NVIDIA RTX A6000 GPU with 48GB of memory.
We use the same model hyperparameter search space as He et al. []. Specifically, we set the number of layers for all models to two. The dropout ratio for the linear layers is fixed at 0.5. For the GCN [Kipf and Welling, ], the hidden layer dimension is set to 64. The GAT [Veličković et al., ] uses eight heads, each with a hidden dimension of eight. For the APPNP [Gasteiger et al., ], a two-layer MLP with a hidden dimension of 64 is used, the power iteration step is set to 10, and the teleport probability is chosen from $\{0.1,0.2,0.5,0.9\}$. For the MLP, the hidden dimension is set to 64. For the ChebNet [Defferrard et al., ], the hidden dimension is set to 32, and two propagation steps are used. For the GPR-GNN [Chien et al., ], a two-layer MLP with a hidden dimension of 64 is used as the feature extractor neural network, and the random walk path length is set to 10. The PPR teleport probability is chosen from $\{0.1,0.2,0.5,0.9\}$. For BernNet [He et al., ], a two-layer MLP with a hidden dimension of 64 is used as the feature extractor, and the polynomial approximation order is set to 10. The dropout ratio for the propagation layers in both GPR-GNN and BernNet is chosen from $\{0.0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9\}$.

## Appendix C Complexity analysis

In this section, we provide a detailed analysis of the time complexity of Section 3.1. Specifically, we demonstrate the time complexity of obtaining the prior and likelihood distributions separately. Finally, we determine the time complexity of computing the posterior distribution using these distributions.

First, the prior distribution $\hat{P}(Y_{i}=m)$ can be obtained as follows:

$$ $\hat{P}(Y_{i}=m)=\frac{\lvert\{u\mid y_{u}=k\}\rvert}{\lvert\mathcal{V}\rvert} =\frac{\sum_{u\in\mathcal{V}}e_{um}}{\lvert\mathcal{V}\rvert}.$ (4) $$

The time complexity of calculating Equation 4 is $O(\lvert\mathcal{V}\rvert)$, so the time complexity of calculating the prior distribution for $K$ classes is $O(\lvert\mathcal{V}\rvert K)$.

Next, calculating the empirical conditional $\hat{P}(Y_{j}=m|Y_{i}=n,(i,j)\in\mathcal{E})$ from Equation 2 can be performed as follows:

$$ $\hat{P}(Y_{j}=m|Y_{i}=n,(i,j)\in\mathcal{E})\propto\sum_{u:u\in\mathcal{V},y_{ u}=n}\sum_{v\in\mathcal{N}(u)}e_{vm}.$ (5) $$

The time complexity of calculating Equation 5 for all possible pairs of $m$ and $n$ is $O(\sum_{u\in\mathcal{V}}|\mathcal{N}(u)|K)$. Since $\sum_{u\in\mathcal{V}}\mathcal{N}(u)=2\lvert\mathcal{E}\rvert$, the time complexity for calculating empirical conditional is $O(\lvert\mathcal{E}\rvert K)$.

The likelihood is approximated through the product of empirical conditional distributions, denoted as $P(\{Y_{j}=y_{j}\}_{j\in\mathcal{N}(i)}|Y_{i}=k)\approx\prod_{j\in\mathcal{N}(i
)}\hat{P}(Y_{j}=y_{j}|Y_{i}=k,(i,j)\in\mathcal{E})$. Likelihood calculation for all training nodes operates in $O(\sum_{u\in\mathcal{V}}\lvert\mathcal{N}(u)\rvert K)$ time complexity. So the overall computational complexity for likelihood calculation is $O(\lvert\mathcal{E}\rvert K)$.

After obtaining the prior distribution and likelihood, the posterior distribution is obtained by Bayes’ rule in Equation 1. Applying Bayes’ rule for $\lvert\mathcal{V}\rvert$ nodes and $K$ classes can be done in $O(\lvert\mathcal{V}\rvert K)$. So the overall time complexity is $O\left(\left(\lvert\mathcal{E}\rvert+\lvert\mathcal{V}\rvert\right)K\right)$. In most cases, $\lvert\mathcal{V}\rvert<\lvert\mathcal{E}\rvert$, so the time complexity of PosteL is $O(\lvert\mathcal{E}\rvert K)$.

**Table 6: Average iteration counts of iterative pseudo labeling for each backbone and dataset used to report Table 1.**
|  | Cora | CiteSeer | PubMed | Computers | Photo | Chameleon | Actor | Squirrel | Texas | Cornell |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GCN+PosteL | 2.5 | 2.2 | 1.5 | 1 | 0.9 | 0.9 | 1.1 | 0.7 | 1.8 | 2.5 |
| GAT+PosteL | 1.6 | 1.8 | 1 | 1.2 | 0.7 | 0.8 | 2 | 1.1 | 3.1 | 2.4 |
| APPNP+PosteL | 1.9 | 2 | 1.1 | 0.8 | 1.1 | 1 | 1.1 | 0.9 | 1.4 | 2.9 |
| MLP+PosteL | 1.7 | 2.2 | 0.4 | 0.7 | 0.7 | 0.1 | 0.8 | 0.6 | 0.9 | 2.4 |
| ChebNet+PosteL | 1.6 | 2.1 | 1.2 | 0.6 | 0.6 | 1 | 0.7 | 0.7 | 2 | 2 |
| GPR-GNN+PosteL | 0.8 | 1.1 | 0.8 | 0.5 | 1.3 | 1 | 0.3 | 0.7 | 1.1 | 1 |
| BernNet+PosteL | 1.5 | 1.8 | 0.9 | 0.8 | 1 | 1.5 | 1.5 | 0.5 | 1.2 | 2.1 |

In Section 3.2, iterative pseudo labeling is proposed, which involves iteratively refining the pseudo labels of validation and test nodes to calculate posterior labels. Since this process requires training the model from scratch for each iteration, the number of iterations can be a significant bottleneck in terms of runtime. Consequently, the iteration counts are evaluated to assess this aspect. The mean iteration counts for each backbone and dataset in Table 1 are summarized in Table 6. With an overall mean iteration count of 1.13, we argue that this level of additional time investment is justifiable for the sake of performance enhancement.

## Appendix D Learning curves analysis for all datasets

The learning curves for all datasets are provided in Figure 8 and Figure 10.

Figure: (a) PubMed
Refer to caption: x15.png

Figure: (a) Chameleon
Refer to caption: x30.png

## Appendix E Empirical conditional distribution for all datasets

The empirical conditional distribution for all datasets is provided in Figure 11 and Figure 12.

Figure: (a) Chameleon
Refer to caption: x45.png

Figure: (a) Cora
Refer to caption: x50.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: please refer to the weakness
soundness: 3
strengths: 1. The paper is well-written and the content is presented clearly, making it easy to follow.

2. The experiments are comprehensive and include a variety of different settings.

3. The proposed method is probabilistically driven, offering a more rational approach compared to existing heuristics.
summary: The paper introduces a method called Posterior Label Smoothing (PosteL) designed to improve node classification tasks in graph-structured data. By integrating both local neighborhood information and global label statistics, PosteL generates soft labels that enhance the generalization capabilities of models and mitigate overfitting. The method is applied to various datasets and models, showing that it consistently outperforms traditional label smoothing and other baseline methods.
weaknesses: 1. The contribution of this paper is somewhat incremental - the idea of using neighborhood information to perform label smoothing has already been extensively studied [1, 2]. 

2. "Under the assumption that the neighborhood labels are conditionally independent given the label of the node to be relabeled...", I fail to comprehend this statement, hope authors can further explain this. But in any case, I think the assumption of two adjacent nodes are conditioally independent is too strong. 

3. While the proposed method shows superior emprical performance, this paper fail to provide an in-depth explaination on why label-smoothing performs so well in graph-structured data - does it stems from the same reason as i.i.d. data? What makes the proposed method superior than other label smoothing methods on graph?

4. The improvements over the most competitive baselines are marginal - for most cases the differences are within the standard deviation.

[1] Adaptive Label Smoothing To Regularize Large-Scale Graph Training, Zhou et al., 2021.

[2] Structure-Aware Label Smoothing for Graph Neural Networks, Wang et al., 2021.

---

confidence: 1
contribution: 3
initial_rating: 5
presentation: 3
questions: 1 The author should clarify how different neighborhood configurations impact performance in heterophilic vs. homophilic settings.
2 How does the method perform with highly sparse labels, particularly below 10% labeled data? Are there specific mitigations for sparsity?
3  The aushor should provide details on the computational cost of PosteL relative to other smoothing methods, especially in terms of training duration across datasets?
soundness: 3
strengths: 1. This paper introduces a new idea in label smoothing by using posterior distribution calculations for label smoothing. Unlike traditional uniform noise smoothing methods, the authors propose combining local neighborhood information with global label distribution, allowing labels to reflect the local graph structure of nodes.
2. The method was tested on both standard homogeneous graphs and heterogeneous graphs, highlighting its general applicability. Experiments show that PosteL label smoothing can prevents overfitting and enhances model generalization without increasing model complexity.
summary: This paper proposes a posterior label smoothing method (PosteL) that enhances node classification accuracy by incorporating both local neighborhood information and global label statistics. The approach demonstrates performance improvements on both homogeneous and heterogeneous graphs. Experimental results show that this method prevents overfitting and exhibits generalization capabilities across multiple datasets and baseline models.
weaknesses: 1. While the posterior label smoothing method achieves notable improvements on graph-structured data, it relies on the neighborhood label distribution of nodes, which may lead to unstable posterior distributions in sparse graphs or scenarios with very few labels, resulting in reduced label quality. Although the authors propose a strategy of re-estimating the posterior with pseudo labels, limitations remain, especially on heterogeneous graphs with sparse labels.
2. Although the paper mentions that the computational complexity is linearly related to the number of edges and categories, it lacks sufficient discussion on efficient computation and optimization for large-scale graph structures, especially heterogeneous graphs. In real-world applications, such as large-scale social networks or e-commerce recommendation systems, this method may encounter efficiency bottlenecks.
3. Although the paper compares various existing label smoothing methods, it lacks adequate comparison with the latest graph classification models, particularly on large-scale heterogeneous graphs. This limitation suggests that PosteL’s effectiveness on more complex or dynamic graph tasks may require further validation.

---

confidence: 3
contribution: 2
initial_rating: 8
presentation: 3
questions: 1. According to Section 2.2, PosteL differs from existing works focused on smoothing in the graph domain in terms of assumptions. However, is there any analysis that demonstrates how these assumptions influence performance improvement? Are there specific examples that support the manuscript's assumptions? Providing this would be crucial for substantiating the novelty of the work.
2. According to the appendix, the datasets are all structured data. Can PosteL be applied to a broader range of research in the graph domain?
soundness: 3
strengths: 1. The manuscript is well-structured and easy to comprehend. For example, Fig. 1 is clear and intuitive.
2. The method is simple yet effective. PosteL can be combined seamlessly with existing methods.
3. The manuscript presents a wealth of experimental results that highlight the potential of PosteL for performance enhancement.
summary: The manuscript proposes PosteL, a label smoothing method utilizing posterior distribution for node classification in graph-structured data. It is basically a preprocessing method for GNNs, generating soft labels based on neighborhood context and global label statistics before the training phase.
weaknesses: The significant drawback I perceive in this manuscript is that while it explores the usefulness of smoothing in the graph domain, the underlying principles that contribute to its effectiveness are not sufficiently clarified. Providing a theoretical analysis of its effectiveness would significantly strengthen the authors' claims.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. Please explain how the method proposed in the article addresses the stated problem of heterogeneity.

2. In the setting of this paper,  the connectivity between all nodes, including the test nodes, is assumed to be observed. Is this assumption a bit too strong? Please give some explanations.
soundness: 2
strengths: 1. This paper introduces a simple yet effective label smoothing method for transductive node classification.

2. This paper investigates the performance of the proposed label smoothing method on different datasets and models, and analyzes the key factors for its success.

3. The authors present their ideas in a well-structured manner, making it easy for readers to follow the flow of the research. The figures in the paper are presented clearly and effectively support the text.
summary: The paper proposes a label-smoothing method called PosteL for the transductive node classification task. This method generates soft labels by integrating local neighborhood information and global label statistics, enhancing generalization and reducing overfitting. Specifically, it computes the posterior distribution of node labels by estimating conditional distributions through neighbour nodes and prior distributions through all labeled nodes, and then interpolates the posterior distribution with the ground truth label to obtain a soft label.Experiments on 10 node classification datasets with 7 baseline models demonstrate its effectiveness.
weaknesses: 1. After claiming the effective handling of heterophilic graphs as a highlight, the description of the proposed method fails to emphasize its treatment of heterophilic graphs. For instance, it is not clear why the proposed method is suitable for handling heterophilic graphs and where the efficiency of this treatment is manifested.

2. The motivation for proposing the method and the effects achieved by the method itself seem to be contradictory. In Section 1, the authors state "...their performance on heterophilic graphs, where nodes tend to connect with others that are dissimilar or belong to different classes, still remains questionable." However, the proposed method estimates the posterior of one node based on label frequency of its neighbours, which leads to its prediction similar to the classes of its neighbours. In addition, the estimation of the prior mitigates the problem of class imbalance but does not address the stated problem of heterogeneity.

3. The idea in the method section of the article is similar to that of a naive Bayes approach, but some assumptions of conditional independence need to be clearly stated in Section 3.

4. In the case where the neighbors are not labeled and the neighbors of the neighbors are also not labeled, an iterative process of generating pseudo-labels needs to be reflected in this algorithm part.

Meta-review: This paper proposes a label smoothing method for transductive node classification, which incorporates local context through neighborhood label distribution. However, the reviewers provided negative feedback, raising several concerns, including: (1) the assumption that neighborhood labels are conditionally independent given the node's label is unclear; (2) the improvements over competitive baselines are viewed as marginal; and (3) the method faces limitations in handling heterogeneous graphs with sparse labels. Based on these considerations, the paper is not recommended for acceptance at this time.

Decision: Reject