RECAL: Sample-Relation Guided Confidence Calibration over Tabular Data
Authors: Wang HaoTian
Zhen Zhang
Mengting Hu
Qichao Wang
Liang Chen
Yatao Bian
Bingzhe Wu
Venue: EMNLP 2023 Findings (2023)
Keywords: Confidence Calibration
Tabular Data
Element-Wise Temperature Scaling
TL;DR: A general post-training framework, RECAL, calibrates the confidence of ML models. It models relations between table samples, using an element-wise scaling approach to improve confidence.

Abstract: Tabular-format data is widely adopted in various real-world applications. Various machine learning models have achieved remarkable success in both industrial applications and data-science competitions. Despite these successes, most current machine learning methods for tabular data lack accurate confidence estimation, which is needed by some high-risk sensitive applications such as credit modeling and financial fraud detection. In this paper, we study the confidence estimation of machine learning models applied to tabular data. The key finding of our paper is that a real-world tabular dataset typically contains implicit sample relations, and this can further help to obtain a more accurate estimation. To this end, we introduce a general post-training confidence calibration framework named RECAL to calibrate the predictive confidence of current machine learning models by employing graph neural networks to model the relations between different samples. We perform extensive experiments on tabular datasets with both implicit and explicit graph structures and show that RECAL can significantly improve the calibration quality compared to the conventional method without considering the sample relations.

RECAL: Sample-Relation Guided Confidence Calibration over
Tabular Data
Haotian Wang'
Zhen Zhang' Mengting Hul- Qichao Wang?
Liang Chen?
Yatao Bian' Bingzhe Wu²
1 College of Software, Nankai University
School of Computer Science and Engineering, Sun Yat-Sen University, 3 Tencent Al Lab wanght@mail nankai edu. cn, mthu@nankai.. edu.cc cn
Abstract
Tabular-format data is widely adopted in vari- ous real- world applications. Various machine learning models have achieved remarkable suc- cess in both industrial applications and data- science competitions. Despite these successes, most current machine learning methods for tab- ular data lack accurate confidence estimation, which is needed by some high-risk sensitive applications such as credit modeling and finan- cial fraud detection. this paper, we study the confidence estimation of machine learning models applied to tabular data. The key find- ing of our paper that a real-world tabular dataset typically contains implicit sample re- lations, and this can further help to obtain a more accurate estimation. To this end, we in- troduce general post-training confidence cali- bration framework named RECAL to calibrate the predictive confidence of current machine learning models by employing graph neural net- works to model the relations between different samples. We perform extensive experiments on tabular datasets with both implicit and ex- plicit graph structures and show that RECAL can significantly improve the calibration quality compared to the conventional method without considering the sample relations.
Basic Model
Conversioral Calloration
Basic Model
[ [
REGAL
(a) Conventional calibration
(b) RECAL
Figure 1: While conventional calibration methods focus only on the test sample itself, RECAL focuses on the relations between different samples.
Despite these progresses, the reliability of cur- rent models trained on tabular data still remains largely unexplored. Specifically, the predictive con- fidence of the model is the key aspect of reliability, which measures the probability that predictions will be correct. A high-quality confidence esti- mator is desired to deploy these ML. models risk-sensitive scenarios such as finance and health- care (Amodei et al., 2016). A natural way to con- struct the predictive confidence is based on the max- imum value of the output probability vector (Guo et al., 2017). Unfortunately, current ML models are prone to output inaccurate predictive distributions, which further lead low-quality confidence estima- tion. For example, as prior work shows (Kumar et al., 2018), Transformer models typically pro- duce over-confident predictions (Mao et al., 2021). Besides, even for traditional MI. models such as GBDT and random forests, we also show that there is room for improvement in their predictive confi- dence quality.
1 Introduction
dabular-forman data widely exists in various real- world applications such as healthcare (Avati et al., 2020), credit modeling (Clements et al., 2020), and finical fraud detection (Ngai et al.. 2011). 1). Various machine learning models (Huang et al., 2020: Chen et al., 2019) are proposed for modeling tabular- format data, ranging from the traditional model such GBDT (Friedman, 2001) to advanced deep models such Transformer (Vaswani et al., 2017). These methods have achieved remarkable progress both industrial applications (Fu et al., 2019) and data science competitions (Vanschoren et al., 2014).
To provide accurate predictive confidence esti- mation of ML models, numerous post-hoc calibra- tion techniques are presented for improving the con- fidence estimation the post-training stage with- out modifying the original training procedure (Platt et al., 1999; Wang et al., 2021). The core idea of
"Corresponding author.

---

these techniques is transform the original predic- tive probability into the calibrated one with some learnable transformation parameters. One common method is temperature scaling (Guo et al., 2017), which has been widely adopted for calibrating both deep and traditional ML models (Huang et al., 2017; Abdar et al., 2021). However, most of these methods are designed for general settings without considering the unique properties of tabular-data format.
We demonstrate the effectiveness of RECAL on wide range of ML models and tabular datasets with both implicit and explicit graph structures.
2 Related Work
Graph Neural Networks. Graph Neural Net- works (GNNs) learn latent representations of node v through graph structure and features of nodes for tasks such as regression and classification (Kipf and Welling, 2016). It performs well when node fea- tures have homophily properties (Wu et al., 2020). Graph Convolutional Networ (GCN) is impor- tant branch of GNN. Referring CNNs, modern GCNs are based on the spectrum of graph Lapla- cian and learn local and global structural patterns of graphs by designing convolution and readout functions (Bruna et al., 2014). Current popular GCN networks typically use a neighborhood aggre- gation approach, also known mechanism, in which each node's representation is represented by nonlinear aggregation function applied to its neighbors representation (Fey and Lenssen, 2019). A GCN model usually consists of multiple layers, cach representing a nonlinear message-ppssingg function to aggregate the local neighborhood of nodes.
This paper aims to improve previous calibration techniques over tabular-format data by consider- ing the implicit relations between different sam- ples. The motivation behind our work is based on the natural observation that real-world data tables contain implicit sample relationships, which can often be reconstructed by mining feature similar- ities. Figure shows a real-world tabular dataset used for house price forecasting. In general, the prices of houses with similar latitude and longitude are always correlated. In other words, home prices are more closely related to their location. These sample relationships can be further used to improve the model confidence. Based on this observation, we introduce a general calibration method for ML models RECAL, which provides a confidence calibration model with the ability to aggregate po- tentially correlated data across the table as clues for calibration. To integrate the implicit sample re- lation with the confidence, we build an undirected graph top of them, where nodes are the sam- ples in the table and edges represent the connec- tions between them. A Graph Neural Network (GNN) is employed to aggregate logits from differ- ent samples, calibrating confidence. RECAL learns a unique temperature t for each sample for scaling, hence retaining the accuracy of the basic model. By incorporating these implicit sample relations into the calibration process, we empirically show that the quality of estimated confidence has been consistently improved. Moreover, RECAL demon- strates remarkable robustness and stability against data noise and distribution shift. In summary, the main contribution of this paper is as follows:
message-passingg
Confidence Calibration. Confidence Calibra- tion has been gaining more and more attention recently. Given a class prediction Y and a data set X, the model being perfectly calibrated should satisfy:
IP(Y = P X) = = p. c [0, 1]. (1)
Thus, confidence calibration changes the predic- tive distribution (also called logits) of the original model. For the tree-based model, Malinin et al. (2021) uses Stochastic Gradient Boosting (SGB) or Stochastic Gradient Langevin Boosting (SGLB) to generate ensembles GBDT models. NGBoost (Duan et al., 2020) is based on natural gradients and uses the parameters of the conditional distri- bution as the target for Multi-paramen All of the above methods require to retraining the basic model from scratch. In deep lcarning, Order- Preserving Functions (Rahimi al., 2020) train the calibration function on the basic models using a post-hoe method through a held-out dataset. Scal- ing is the commonly used host-hoe confidence cali- bration technique. On the binary model, Platt scal-
boosting.
To the best of our knowledge, we are the first to explore how to leverage the sample relation to improve the model confidence calibration over the tabular dataset.
For the above problem, we present a GNN- based calibration framework named RECAL.

---

validation datasel
graph structure implicit in the table based on the selected feature ($3.1). Then we use GNN a non-linear calibration function to characterize the topology on the graph ($3.2). Finally RECAL outputs a unique temperature for each sample and calibrates samples using scaling- approach ($3.3).
feature
categorical
value continucos value
Pretrain GBDT for fenture selection
if categorical feature
if continuuus feature
(ef()e)),, 288) (f similar cateorical value
ser(n") (1t"), p!ll... n J
3.1 Sample-Relation Modeling
The locils each node used as the node of gnph
Tabular datasets are widely used in machine Iearn- ing models for training and inference purposes. They exhibit highly structured format, where each column represents specific feature or variable, and the values in different rows of the same col- umn are treated independent entities. However, important to acknowledge that each column often carries its own physical significance, and cer- tain features may have a more intuitive impact on the predictive outcomes. For instance, in the con- text of house price forecasting, houses located in similar geographic regions tend to exhibit similar prices. Hence, is hypothesized that implicit rela- tions exist between samples in the dataset, which could potentially enhance the confidence and accu- racy of predictions. Motivated by this, we model the sample relations in the held-out dataset to build graph. The process is depicted in Figure 2.
G
Figure 2: The process of constructing graph from table. A GBDT model is pre-trained using the valida- tion set select the feature with the largest contribution, and nodes divided into different sets based on the selected feature. Edges exist between nodes in the same set. M indicates that the categorical feature has M dif- ferent possible values. N indicates the total number of nodes.
ing (Platt et al., 1999) transforms logits by predict- ing two scalar parameters. Temperature scaling and Matrix scaling (Guo et al., 2017) are multiclassifi- cation extensions of Platt scaling. CaGCN (Wang et al., 2021) uses neural networks as nonlinear post- hoc calibration functions. However, how to find implicit sample relations from the tabular data and construct generic calibration model for tabular data independent of the basic model remains to be studied.
Pre-train GBDT Specifically, we pre-train a Gra- dient Boosting Decision Tree (GBDT) model using the validation set in the dataset. The utilization of the validation is motivated by the consider- ation of post-hoc calibration scenarios (Guo et al., 2017). In this way, the original training data and the basic model M remain black boxes during the calibration process, thus ensuring user privacy protection. Additionally, the adoption of GBDT is appropriate due to its rapid convergence, low training overhead, and its efficacy decision-maker. Moreover, GBDT exhibits high in- terpretability, accurately expressing the importance of different features (Friedman, 2001).
3 Methodology
In this section, we introduce a post-hoc calibration method called RECAL to calibrate the model ap- plied to the tabular data. Assume in a tabular clas- sification task, we have basic model M, which can classify each row into K classes. Yet M is not well-calibrated, indicating its confidence can- not accurately reflect its performance. Our pur- pose post-calibration, which indicates that the training data is unavailable this stage. Only a small held-out dataset can be used (Guo et al., 2017). Therefore, we aim to train another calibra- tion model, i.e. RECAL C, with the held-out dataset R = (r1, r2, ..., ..., TN): ), where N indicates the num- ber of samples in R. RECAL achieves calibration for each sample by leveraging the graph structure built from implicit sample relations.
auxiliary
Graph Construction Based on the good explain- ability of GBDT, we select the feature that con- tributes the most significantly to the GBDT results as the foundation for our graph construction. If the selected feature is categorical, we establish con- nections between nodes sharing the same categori- cal value for that feature. For continuous features, each sample connected to the top-k samples that demonstrate proximity. By incorporating these im- plicit relations into the model, our objective is to
Specifically, as depicted in Figure 3, RECAL first pre-trains a feature selection model and constructs

---

Streamed test set Calibration sct
#7 min C W.)
Uncalibrated logits
4.3 2.4
2.1
1.8
3.3 6.0
thle
Cahibrated logits
4.5
2.5
9.
2.6
1.0
3.7
5.4
GCN
N-I
Elenment-wiss Temperature
lest
Calibrated logits
Basic Model Innd
logits
Scaling
(c) Nonlinear
(a) Input
(b) Sample-Relation Modeling
(d) Scaling-based Calibration
Calibration Function
Figure 3: The overall framework of RECAL. The ranking of classes is unchanged after being calibrated.
enhance the model's 's ability to generate more reli- able predictions for tabular datasets. As depicted in Figure 2, we now derive G from table, where the adjacency matrix is denoted by A E RNNN
In summary, RECAL takes the logits V and the sample-relation graph G as inputs to calibrate V. During training, we use the validation set as the held-out calibration set R. Then, V and G are used to train the calibration model C. During testing, we consider the real-time online scenario where the data be calibrated comes in streaming fash- ion with only few samples tain more comprehensive graph structure informa- tion, we aggregate each incoming sample with the calibration set to construct a new graph structure Geest Vj 1, 1,..., Retst]
3.2 Nonlinear Calibration Function
To explore the structured knowledge in G, we adopt Graph Convolutional Network (GCN) to propa- gate node features along the network topology and smooth the information between neighboring nodes through nonlinear aggregation function (Gian- nakis et al., 2018). Firstly, the basic model M is adopted to compute logit for each sample in the held-out dataset before the softmax layer,
time. To ob-
3.3 Element- Wise Temperature Scaling
Now, we have nonlinear calibration function to characterize the topological relations the graph. However, the output of GCN directly cali- brated logit cannot meet the requirement of accu- racy maintenance. Temperature scaling currently the most effective calibration method to maintain constant prediction accuracy. But, the temperature scaling performs the same linear transformation for each input data, which does not express the graph 's topology. To this end, we propose element- wise temperature scaling to generate unique tempera- ture for each node in G.
Vi = M((r),
(2)
where i E N] and V [v1,.... VN] is ob- tained.
Then, we use Eq.3 to calculate the symmetric normalized Laplacian matrix:
(3)
where E NxN is the degree matrix of graph G, and I the unit matrix. For a K (-cllasiificttonn task, the logit ví and the prediction probability z; for sample after calibration can be calculated as:
As shown in Figure 3, RECAL first takes the out- put V and the sample-relation graph as input and outputs a temperature vector t, where t, is unique for Vi. Then, we use f, to transform the original logit vi through scaling methods to obtain the cal- ibrated logit v/. Finally, the calibrated prediction result P; is given by:
V = L Le vu =
(4)
Zi (TSM(((a).. ... osM(((
where ( is the weight matrix of 1-th layer in exp(e,.) is the softmax GCN, OSM (u,,)) Σx expixi) )
operation, and the calibrated confidence is p5 = max 2,, h E {1....,,} K}. 1',8 can be any value so that pi can traverse the interval & 1), and the model can be calibrated correctly.
t - o (L (000ww()))) [4....., (N....,,N].
v vi/ti,
(5)
p. max (vi)

---

where o+ (44 log(1 + exp(4)) is known as wlement-wise softplus activation, which guarantees that is always greater than zero. When t + 0, the predicted probability is infinitely close to 1. When 4 -> oc, the probability pi has the greatest uncertainty, and the predicted probability approaches K. Thus, RECAL can output any value on ( 1), and the basic model M can theoretically be perfectly calibrated.
Dataset
Datapoints
Features
labels
Type
Marketing
45211
16
Dense
Qsar- GiodeB
1055
41
Dense
Seismic-Bumg
2583
18
2
Dense
County
3217
7
Dense
House Class
20640
6
Dense
DBLP 14475
5002
Spurse
SLAP
20419
2701
15
Spurse
Table I: Summary of datasets. It is worth noting that DBLP and SLAP datasets have an explicitly global graph structure, respectively. Yet in other datasets, we need to build graph by implicit relations.
RECAL is optimized with negative log- likelihood (NLL) (Hastie et al., 2009). This is a common indicator for measuring probabilis- tic models, minimizing NLL loss benefits uncer- tainty calibration. Given the one-hot label y; 1.. ki, and the prediction probability 2₁, the NLL loss over the calibration set can be calcu- lated as:
data split into training, validation, and testing sets ratio of 80%, 10%, and 10%, respectively.
The details of the dataset are outlined in Table 1. To fully explain the graph construction process on each dataset, we have provided further instructions in Appendix A.2.
N
Cill
Vi. log(z
(6)
4.2 Experimental Settings
ECE (Naeini al., 2015) and Brier Score the evaluation metrics and the classical post-hoc method temperature scaling (TS) (Guo et al., 2017) as the bascline method. In addition, we compare a variant of our method, RECAL-MLP, which turns the GCN into fully connected neural network. This is equivalent to more complex matrix scal- ing (Guo et al., 2017). Appendix A.I provides more detailed description.
Inspired by Wang et al. (2021), we also add a regularization term
Lent
Zim + 2
lerr
eery
Zimm
To demonstrate the generalizability of our method, we chose the classical machine learning method Logistic Regression (Wright, 1995), GBDT (Friedman, 2001), and deep learning model called Tranffofformer (Huang et al., 2020) the basic model.
where |cor| and |err| are the number of samples correctly and incorrectly predicted and 2mm and 24,s are the max and submax prediction probability, to the loss function to explicitly reduce the confidence level of misclassified samples. The overall loss function is defined as:
For Logistic Regression, we use scikit-learn's official interface (Kramer, 2016) and modify the source code to ensure that the returned prediction distribution is the value before the softmax layer. We use the CatBoost (Prokhorenkova et al., 2018) as implementation of GBDT. For each decision tree in CatBoost, we set its depth to 6, the early stopping rounds is 100, the number of epochs is 1000, ||A||2 O. TabTransformer's implementa- tion and parameter selection are consistent with (Huang ct al., 2020). The implementation of tem- perature scaling is based on the original author's 's implementation (Guo et al., 2017). For our RECAL, we build 2-layer GCN with the hidden layer of dimension 16. We set the learning rate as (0.001,
CRECAL = Lill + Xheals
where À is a hyperparameter to balance the effects of the regularization term.
4
Experiments
4.1 Datasets
To validate the effectiveness of RECAL, we con- duct experiments on seven datasets with both im- plicit and explicit graph structures from the real- world -Bak Marketing (Moro et al., 2014), Qsar- Biodeg (Huang et al., 2020). Seismic-Bumps (Man- souri et al., 301 County (Jía and Benson, 2020), Mouse_Class (Pace and Barry, 1997). DBLP (Ren and Liu, 2020) and SLAP (Xiao et al., 2019). The
http:///sikkitllam.ongggege

---

Logistic
GBDT
Tab Transformer
Dataset
Calibration
NTJA()
ECC((L)
(SS(1)
NLL())
ECC(())
(S(L))
NL.(()
RCR(L)
FRS(L)
Uncal
0,26032g
0,024844
0.153%6s
0,2088223
0.0151aa
0,128112
0.211th.
0.0141a.
0.1391aa
TS
4
0.0255227
0,1538
0,207113
0.0158a.
0.1274ea
0,2105
0.00900.
0.1387
Bank Marketing
RECAL
1,259125
0.0208
0.1532
0.20644:
0.0147433
0. 0.16744
0.21022.3
0.00984.3
0.1387a5
RECAL (ours)
0.2537
0.0184
0.15492a
0.2063as
0.01272.5
0.12724.0
0.20872.1
0.00752.3
0.138
Uncal
0,299124
0.0608a.
0.17255as
0.30612e
0.08333.
0,18880
0.28955.
0.1652 0.08166.7
TS
0.298207
0.0580a.
0.17162a
0.2910a.s
0.0737a.
0. 17656..
0.28034.1
0.0704a.s
0. 0.1580g.3
Qsa-Biodeg
RECAL- MIP
1.296115
0.051947
0. 06
0.2847aa
0.063863
0. 073327
0,28694.
0.0725g.a
0. 0.16333.5
RECAL (ours)
0.0504s4
0.1674
0,28165.
0.0502,,
0.17172ss
0.2861s.7
0.0600
0.1520,.1
Uncal
0
0,0299,2
0.089
0.17
0.0317ap
0.087214
0.18033.
0.03994.
0.08886
0.0352g4
0.090%ea
0.1726a.
0.03284,3
0.0675L.
0. 0.13324.1
0.0878:. 0.0324a.5
Seismic Bumps
RECAL- MIP
0.1731a3
0.0226gs
p.089
0. 0.1675
0.0283u2
0.0868..2
0. 0.1607g.
0.0340
0.0865
RECAL (ours)
0.1724os
0.0216:4
0. 0.0899m2
0.1674..
0.0243
0.08736.1
0.1638.,s
0.0212
0.0857
Uncal
1 1.2080g
0.0813g2 0.0509p2
0.98662a
0.0
0.5616a.o
1. 1,1000
0.0518
0. 6.082.24
1.2080
0.0432g.3
0.68 14.7
0.98442e
0.049947
0.5611a.
1.1271
0.0412a.s
0.67972.7
County
RECAL- MLP
1.2086ip
0.0391aa
0.6614.s
0,988
0.0623aa2
0.56192.
1.1277a.1
0.01111.
0.6796..8
RECAL (ours)
1 1
0. 0.0608 0,0385
0
0.0115g.5
0.5604..o
1.1241a.2
0. 0.0389
0.6782).7
Uncal
1.1916s1
0.017874
0.600sos
1.098 L988324
0.0366a.
0.50904s
1.4577a44
0.0220g.1
0.7392
TS
1.193623
0,0275x.7
0. 0.6051ns
1.0904..o
0.0178uo
0.5664
1.45600..
0.0189.
0.7390,.s
House_Ciaa
RECAL- MLP
1.1829 12
0,0153g2
0. .
1,0899
0.0157.a3
0.5608ss
1.458522.
0.0236
0.7407
RECAL (ours)
1.1886
0.0141.3
0.5964 12
1,0860
0.0154as
0.5658..
1 1.45402.
0.0173,
0.7371
Uncal
0.610621
0.0710.2
0.2180
0.5837a.
0. 0.1871a
0.2901
1.186las
12033.s
0.6397s.1
TS
0.6236
0,09205.3
0,25531
0,4748.a
0,0684as
0,2417
1.1845g..7
0.0891a.a
0.6426
DBLP
RECAL- MLP
0.5024
0,0714.9
0,2415
0,1628..
0.0
0,23400,
1.1770gs
0.09737,
0.63944.
RECAL (ours)
0
0.0584.
0.2415
0.45693
0.0377p.4
0,2301
1.16775.9
0.0718
0.6334.7
Uncal
0.69922g
0,2757yz
0.33144,
0,1533,,
0.0817a3
0.0606
0,9016m
0. 0,19833,
0.47988.1
0.6154
0,2144
0,26557.7
0,0878au
0.0234
0.05011.
0,9014zz
0.19266.7
0.4785s.
SLAP
RECAL- MLP
0.0896ss
0.16114.3
0.088%
0.0083e
0.0450g,
0.8630as
0.09322.0
0.3922,.3
RECAL (ours)
0.3231
0.062763
0.1598..s
0.0859.7
0.006163
0.04412.0
0.8581
0.09017.6
0.37651s
Table 2: Summary of results using different calibration methods on different basic models and different datasets. Uncal indicates the basic models without calibration, BS represents the Brier Score. RECAL- MLP is the ablation study, and bold indicates the best result, the subscript of each result refers to the standard deviation (x 10-3).
0.01), weight decay to 5e-3 for all datasets, ls is 5 in top-k. Our evaluation metries are NLL loss, ECE, and Bricr Score, all using official realizations (Naeini et al., 2015; Brier et al., 1950).
cant graph structure. It can be observed that TS performs well on TabTrrnsformer which is due to the fact that the miscalibration of neural networks tends to be low- dimensional (Guo et al., 2017). But, it does not perform consistently enough on the traditional ma- chine learning methods like GBDT and Logistic Regression (only out of datasets are success- fully calibrated). This suggests that TS is unreliable for calibrating traditional machine learning meth- ods. To explore the effect of the implicit sample relations between the data on the calibration results, propose variant RECAL-MLP of RECAL, in which the 2-layer GCN is replaced with 2-layer MLP. On Trannnssfommer, TS performs better than RECAL-MLP. However, on the machine learning method, RECAL-MLP performs better than TS. This shows that applying an identical change to all logits on traditional machine learning methods is not enough.
We use the validation set as the held-out calibra- tion set. In the real world, models often need to predict consistently from streaming data. There- fore, we calibrate each row of data in the test set individually and construct a unique graph for each test sample by aggregating it with the validation set.
4.3 Main Results
The evaluation results of our model and all base- lines are summarized in Table 2. The uncalibrated model usually has the highest values on all met- rics, with ECE is typically between 1.4% to 20%, which indicates that the model used to predict the tabular data is poorly calibrated. This is not based on specific basic model: we observed miscali- bration on linear models, tree-based models, and transformer- baased deep neural networks. In partic- ular, the values of ECE are significantly worse on DBLP and SLAP, two sparse datasets with signifi-
Our method achieves the best results on all datasets. In particular, compared to the temper- ature scaling, our method reduces the ECE by 20%, 32%, 26%, 13%, 17%, 45% and 74% on the

---

Dataset
Calibraction
Noise Rate %
30
Bank Marketing
T8
0.155
0.0248
0111547
01467
0.0672
RECAL
0.0127
0.0212
0,0315
0.057
0,0646
TS
0.0499
0.0524
0,0592
0.551
0.0815
County
RECAL
0.0455
0,0476
0.0464
0.0525
0,0677
House_Class
TS
0.0178
0.0311
0,0623
0.0455
0.1326
RECAL
0.0154
0,0296
0.0566
0.0817
0.1265
County-UneH
(h) Uncal
Table 3: ECE of RECAL and TS with noisy data on GBDT.
Tab Transformer
Dataset
Uncal
TS
RECAI.
Bank Marketing
0,0397 0.0982
0.0488 0,0525
0.0317
Selsmic-Rump
0.0298
Table 4: ECE of RECAL and TS with distribution shift on TabTransformer.
(c) County-RECAM
(d) OBLP-RECAL
Figure 4: Reliability diagrams for GBDT before and af- ter confidence calibration on County and DBLP datasets. Any deviation from a perfect(i.e., the gap) diagonal rep- resents miscalibration.
CAL most closely recovers the desired diagonal function, and each of the bins are well calibrated. This suggests that RECAL can yield better confi- dence estimates, showing it is more trustworthy.
Seismic-Bumy Bump, House_Class, County, DBLP, SLAP datasets, respectively, on the GBDT model. On TabTransformer, compared to the temperature scaling, our method reduces the ECE by 17%, 15%, 35%, 8%, 6%, 19%, and 45%, respectively. On Logistic Regression, compared to the temperature scaling, our method reduces the ECE by 22%, 17%, 39%, 49%, 11% 37%, and %,, respectively. NLL. and Brier Score show the same trend. In brief, our method outperforms the baseline method and the graphs provide valuable information for confidence calibration.
Bank,
4.5 The Robustness of RECAL
We further explored the robustness of RECAL to noisy data and distribution shift against the baseline TS.
Noisy Data. To obtain the noisy data, first select the coninuous feature that has the largest im- pact on the tree-base model among all features. In CatBoost, the impact factor of each feature can be obtained from model.feature ponddelfeattre__ppprttaceee importances_. Next, on the test set, we contaminate the data by replac- ing a number of values with random ones in the selected columns. We input the noisy streamed data into the trained GBDT model to obtain the output and calibrate it using the trained RECAL model. Table gives the results of the calculations on three different datasets. As the noisy rate in- creases, RECAL performs better in ECE and thus is more robust than temperature scaling. We con- jecture that the graph brings robustness because its topology dilutes the effect of noise to certain extent.
4.4 Reliability Diagrams
We further depict the reliability diagrams (Guo et al., 2017) to evaluate the quality of uncertainty estimation. As shown in Figure 4, the confidence range is equally divided into 20 bins. Then we cal- culate the average accuracy of each bin. The red diagonal represents the perfectly calibrated line, and its confidence indicates the probability of the model predicting correctly. For example, the mean value of the diagonal function in the bin [0.95, 1.0] is 0.97, and then the classification accuracy of the nodes in this bin should be 97%.
Distribution Shift. Distribution shift refers to the change in the underlying data distribution be- tween training and testing phases in machine learn- ing, which can affect model performance when the model encounters data that differs significantly from what it has learned during training. We first partition the dataset into two parts with different
We can see that the average precision of most boxes of the uncalibrated model higher than the diagonal, indicating that GBDT always outputs a low confidence, i.c., GBDT are usually under- confident. Compared with uncalibrated GBDT, RE-

---

Diraset
Logistie
GDBT
TahTransformer
Logistic
online
offline
online
offline
caline
offline
Uncal
RECAL
Hank Markeinng
0.01844
0.0178
0.0277
0.0106
0.0075
0.0081
Qsar Biodog
136.5013
129.5814
119.8235
Oser BardBBioog
0.0504
0.0447
0.0502
0.0511
0.0600
0.0479
Bank Marketing
6802.1870
6750,9567
6548.1997
Seismic-Bumps
0.0216
0.0175
0.0243
0.0195
0.0212
0.0203
County
57.3955
56.5785
53,5246
SLAP 7.7824
5,0452
0.0653
County
0.0385
0.0361
0.0115
0.0385
0.0389
0.0413
House Closs
0.0148
0.0544
0.0147
00173
0.0135
Table 6: Total variation confidence before and after calibration for four datasets Logistic Regression. Uncal indicates uncalibrated, TS indicates temperature scaling and bold indicates the best result.
DBLP
0.0884
0.0608
0.077
0.0295
0.0718
0.0835
SLAP
0.0626
0.061
0.0051
0,000
0.0890
Table 5: ECE of RECAL in online and offline scenarios.
distributions based on an artificially chosen feature. We conduct experiments on Bank and Seismic- Bumps. For Bank, we divide it into two subsets with and without houses based on housing. For Seismic- Bumis, partition the dataset according seismic, which grades the hazard of the detected seismic waves. We make the training and valida- tion sets have the same distribution, and the test set obeys a different distribution. Then, we train the basic model using the training set, train RECAL utilizing the validation set, and test it using the test set. Results on the two datasets are presented in Table 4. We can see the same conclusion as in the case of noisy data, i.e. that RECAL shows excellent stability and can still calibrate the model, while TS cannot handle this case.
the graph G. Table 6 presents the results compar- ing the total confidence change before and after calibration. Our research findings demonstrate that applying temperature scaling decreases the total change in confidence. This observation supports the assertion that well-calibrated basic models re- sult increased similarity in confidence levels be- tween nodes with the defined relation after cali- bration. GNN performs massage passing along its topology, making it smoother between neighbor- ing nodes. Thus, our approach of incorporating im- plicit sample relations into the graph construction facilitates more reliable predictions and improves confidence calibration for tabular datasets.
Conclusion
In this paper, we exploit the implicit sample rela- tion tabular data and propose a post-hoc cali- bration framework RECAL, which improves the confidence of the model while maintaining clas- sification accuracy. It is worth noting that our approach is general and applicable to both deep learning models and traditional machine learning models. RECAL uses the GCN as a nonlinear cali- bration function so that the confidence can be prop- agated along the graph between the sample. In addition, RECAL generates unique temperature for each sample to be calibrated using the scaling method, which has the accuracy--ressrrving prop- erty. We perform detailed experiments on binary and multi-classified tabular datasets with implicit and explicit graph. Empirically results demonstrate that RECAL has a better performance compared to traditional calibration models. In addition, exper- iments demonstrate that RECAL has remarkable robustness against data noise and distribution shift. An interesting future research work would be use embedding of samples guide the discovery of graph G. This extension would enable the appli- cation of RECAL not only to tabular data but also to textual and image data.
4.6 Offiine Test Scenario
Further, we conduct an experiment to show that RECAL is equally effective in the offline scenario. In this scenario, uncalibrated samples arrive the same time. We also use the validation set to train the RECAL. Unlike the online scenario that gener- ates a unique graph for each test sample, we use all the test samples to construct one graph and get the calibration results for the whole test set by forward propagation once. As shown in Table 5, RECAL works equally well in the offline scenario and per- forms better than online in most cases. We believe this is because building the graph on the test set allows for more accurate characterization of its sample relation.
4.7
Why RECAL Achieves Well Calibration?
To validate the effectiveness of the graph guided by implicit sample relations in enhancing confidence calibration, we perform the following experiment. We utilize temperature scaling as the calibration function and calculate the total change confi- dence, which represents the cumulative difference in confidence values between adjacent nodes within

---

Limitations
Faisal Bin Ashraf, Abdul Matin, Md Shafiur Raihan Shañ, and Muhammad Usama Islam. 2021. An improved k-means clustering algorithm for multi- dimensional multi-cluster data using meta-heuristics. In 2021 24th International Conference on Computer and Information Technology (ICCIT), pages 1-6. IEF..
Our work is the first attempt to use graph struc- tures to represent the relation between samples in tabular data for improving the reliability of tabular model. Using sample-relation graph, RECAL out- puts unique temperature t for each sample, and then finely calibrates each sample using a scaling- based method to make the model more trustworthy. Further work can apply the idea of modeling im- plicit sample relations from datasets to enhance the task more domains.
Anand Avati, Tony Duan, Sharon Zhou, Kenneth Jung, Nigam H Shah, and Andrew Y Ng. 2020. Countdown regression: sharp and calibrated survival predictions. In Uncertainty in Artificial Intelligence, pages 145- 155. PMLR.
Glenn W Brier et al. 1950. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3.
In our analysis. we select a specific feature by a GBDT model pre-trained on a calibration set as the basis for our definition of implicit sample re- lations. However, in real scenarios, large tables with hundreds thousands of features may exist. Meanwhile, the method of traversing each node to connect neighboring nodes makes the overhead of building the graph large. In future research, this can be done by expressing each sample as an embed- ding vector and then using clustering algorithm (MacQueen, 1967; Ashraf et al., 2021) to build a graph on the dataset. With the use of embedding. the method is also no longer limited to tabular data, which opens up new ideas for improving the trust- worthiness of machine learning.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. 2014. Spectral networks and locally connected networks on graphs. The International Conference on Learning Representations (ICLR). pages 1-14.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang. Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. Tabfact: large-scale dataset for table- based fact verification. The Inter- national Conference on Learning Representations (ICLR), pages 1-17.
Jillian M Clements, Di Xu, Nooshin Yousefi, and Dmitry Efimov. 2020. Sequential deep learning for credit risk monitoring with tabular financial data. arXiv preprint 2rxiv:2012.15330.
Acknowledgements
Tony Duan, Avati Anand, Daisy Ding, Khanh Thai. Sanjay Basu, Andrew Ng. and Alejandro Schuler. 2020. Ngboost: Natural gradient boosting for prob- abilistic prediction. In International Conference on Machine Learning, pages 2690-2700 20000000000011 PMLR.
We sincerely thank all the anonymous review- ers for providing valuable feedback. This work is supported by the National Key Research and Development Program of China (Grant No. 2021YFB0300104), the youth program of Na- tional Science Fund of Tianjin, China (Grant No. 22JCONJC01340)) and the Fundamental Research Funds for the Central University, Nankai University (Grant No. 63221028)
Matthias Fey and Jan Eric Lenssen. 2019. Fast graph
representation learning with pytorch geometric. The International Conference on Learning Representa- tions (ICLR), pages 1-9.
Jerome H Friedman. 2001. Greedy function approx- imation: gradient boosting machine. Annals of statistics, pages 1189-1232.
Fangcheng Fu, Jiawei Jiang, Yingxia Shao, and Bin Cui. 2019. An experimental evaluation of large scale gbdt systems. Proceedings of the VLDB Endowment. 12(11).
References
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Ra- jendra Acharya, Vladimir Makarenkov, and Sacid Nahavandi. 2021. A review of' uncertainty quantifica- tion in deep learning: Techniques, applications and challenges. Information Fusion, 76:243-297.
Georgios Giannakis, Yanning Shen, and Geor- gios Vasileios Karanikolas. 2018. Topology identifi- cation and learning over graphs: Accounting for non- lincarities and dynamics. Proceedings of the IEEE 106(5):787-807
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in ai safety. arXiv preprint 46X1v:1606.006655
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. In International conf conference on machine learn- ing, pages 1321-1330. PMLR

---

Trevor Hastie, Robert Tibshirani. Jerome Friedman. and Jerome Friedman. 2009. The elements of statis- tical learning: data mining, inference, and prediction, volume 2. Springer.
Mahdi Pakdaman Nacini, Gregory F Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated prob- abilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial In- telligence, pages 2901-2907.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected con- volutional networks. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4700
Eric WT Ngai, Yong Hu, Yiu Hing Wong. Yijun Chen. and Xin Sun. 2011. The application of data mining techniques in financial fraud detection: A classifica- tion framework and an academic review of literature. Decision support systems, 50(3):559-569.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin, 2020. Tabtransformer: Tabular data mod- eling using contextual embeddings. arXiv preprint arXiv:2012.06678
R Kelley Pace and Ronald Barry. 1997. Sparse spa- tial autoregressions. Statistics & Probability Letters, 33(3)29112277.
Sergei Ivanov and Liudmila Prokhorenkova. 2021. Boost then convolve: Gradient boosting meets graph neural networks. The International Conference on Learning Representations (ICLR), pages 1-16.
John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized like- lihood methods. Advances in large margin classifiers, 106):61-74.
Junteng Jia and Austion R Benson. 2020. Residual correlation in graph neural network regression. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 588-598.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. 2018. Catboost: unbiased boosting with cat- egorical features. Advances in neural information processing systems (NeurIPS), 31.
Thomas N Kipf and Max Welling. 2016. Semi- supervised classification with graph convolutional networks. The International Conference on Learning Representations (ICLR), pages 1-16.
Amir Rahimi, Amirreza Shaban, Ching-An Cheng, Richard Hartley, and Byron Boots. 2020. Intra order- preserving functions for calibration of multi-class neural networks. Advances in Neural Information Processing Systems (NeurIPS), 33:13456:1131 13467.
Oliver Kramer. 2016. Scikit- In Machine learning for evolution strategies, pages 45-53. Springer.
Yuxiang Ren and Bo Liu. 2020. Heterogeneous deep graph infomax. Workshop of Deep Learning on Graphs: Methodologies and Applications co-located with the Thirty- Fourth AAAI Confer-ence Artificial Intelligence.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. 2018. Trainable calibration measures for neural networks from kernel mean embeddings. International Conference on Machine Learning, pages 2805-28 14. PMLR.
Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. 2014. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):99-60.
MacQueen. 1967. Classification and analysis of mul- tivariate observations. In 5th Berkeley Symp. Math. Statist. Probability, pages 281 281-297. University of California Los Angeles LA USA.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems (NeurIPS), 30.
Andrey Malinin, Liudmila Prokhorenkova, and Alek- sei Ustimenko. 2021. Uncertainty gradient boost- ing via ensembles. The International Conference on Learning Representations (ICLR), pages 1-17.
Kamel Mansouri, Tine Ringsted, Davide Ballabio, Roberto Todeschini, and Viviana Consonni. 2013. Quantitative structure-activity relationship models for ready biodegradability of chemicals. Journal of chemical information and modeling, 53(4)8867-875
Xiao Wang, Hongrui Liu, Chuan Shi, and Cheng Yang. 2021. Be confident! towards trustworthy graph neu- networks via confidence calibration. Advances in Neural Information Processing Systems (NeurIPS),
Yuxin Mao, Jing Zhang, Zhexiong Wan, Yuchao Dai, Aixuan Li, Yunqiu Lv, Xinyu Tian, Deng-Ping Fan, and Nick Barnes. 2021. Generative transformer for accurate and reliable salient object detection, anXiv e-prints, page 1rXiv:2104.1011
Raymond Wright. 1995. Logistic regression. Read- ing and Understanding Multivariate Statistics, pages 217-244
Zonghan Wu. Shirui Pan, Fengwen Chen, Guodong Long, Chenggi Zhang, and S Yu Philip. 2020. A com- prehensive survey graph neural networks. IEEE transactions on neural networks and learning sys- lems, 32(1):4-22
Sérgio Moro, Paulo Cortez, and Paulo Rita. 2014. A data-driven approach to predict the success bank telemarketing. Decision Support Systems, 62:22-31.

---

Yuxin Xiao, Zecheng Zhang, Carl Yang, and Chengxi- ang Zhai. 2019. Non-local attention learning on large heterogeneous information networks. In 2019 IEFF International Conference on Big Data (Big Data), pages 978-987. IEEE.

---

A Appendix
in the mine, which contains 2583 samples. It is a binary dataset. Based on the contribution to the classification results, we use gpuls, the number pulses recorded by the most active geophones the previous transformation, to build graph with each node connecting the five closest nodes according to the value of gpuls.
A.1 Evaluation Metrics and Baseline
Evaluation Metrics. The expected calibration error, also called ECE divides the prediction into B equally spaced bins (Nacini et al., 2015). It calculates the weighted average of the accuracy'l conffidece difference of the samples in each bin. We use ECE as the most important eval- uation indicator. The calculation equation is as follows:
Qsar-Biodeg (Mansouri et al., 2013) dataset counts the chemical structure of molecules predict whether they readily biodegrad- able (RB/NRB), which contains 1058 samples. By pre-training a GBDT model using the val- idation set, we use SpMax_ B as the basis for building the graph.
B
B Riest
ECE
aace [:) conf(B.)), (9)
where |Reest is the total number of test sets, B, represents the i-th bin, [Bj] represents the num- ber of samples in the i-th bin.ace(B,, denotes the accuracy.con(((5 represents the average of confi- dences in the i-th bin. The Brier Score (Brier et al., 1950) is also a measure of the degree of calibration. It portrays the difference in predicted probability from the true label. More precisely,
House_ Class (Pace and Barry, 1997) is col- lected from the 1990 census, which contains 20640 samples. The original dataset is a re- gression dataset with the predicted outcome being the price of the property. To construct a multi-classification tabular dataset, we di- vide the prediction results into five intervals [1, 1.5,,,,, 2.5] (Ivanov and Prokhorenkova, 2021). For each block, connect the five closest blocks within a given distance, measured by latitude and longitude.
R
BS
Σ
fp yn) (10)
RiCst
where yn is the label of the nith sample.
Baseline. As comparison, we select the classi- cal post-hoc method temperature scaling (TS) (Guo al., 2017) as our baseline, which is given by
County (Jia and Benson, 2020) county- level map dataset containing 3217 samples. We divide the unemployment rate into four boxes as prediction labels, <0.4, 0.4-0.5, 0.5- 0.7, >0.7. Each node is county, and if two nodes border each other, they are connected.
p = maxxss
(11)
A.2 Dataset Details
Here, provide detailed description of the dataset used in this paper, as well the basis for creating the graphs.
SLAP (Xiao et al., 2019) is sparse multi- node-type dataset in the domain of bioinfor- matics with node types including genes, dis- cases, chemical compounds, etc. Our goal is predict one of the 15 gene types. contains 20419 samples. We use the same graph struc- ture as Ivanov and Prokhorenkova (2021).
Bank Marketing (Moro et al., 2014) is re- lated to the direct marketing activities of a Portuguese banking institution, which con- tains 45211 samples, and the classification goal is to predict if the client will subscribe (yes/no) term deposit. Guided by the pre- trained GBDT, we select month as the basis for building the graph. Since is a categorical feature, we connect samples with the same value.
DBLP (Ren and Liu, 2020) is a sparse clas- sification dataset with 14475 samples. It pre- dicts the author's s research direction and is di- vided into four groups(database, information retrieval, data mining, and machine learning). We use the same graph structure Ivanov and Prokhorenkova (2021).
Seismic-Bump (Huang al., 2020) predict whether a high energy seismic bump will oc- cur in the next shift by using the detection data


Reviews:
[{"review_id": "OpWENZay3u", "reviewer": "Reviewer_XNjM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Q1:__ A minor concern, but it would have been beneficial to see comparisons with other calibration mechanisms such as isotonic regression, histogram binning, and platt scaling.\n\n__A1:__ We appreciate your kind suggestions. We supplement the experiments using Isotonic Regression（IR） for calibration on the Bank Marketing, Qsar-Biodeg and Seismic-Bumps datasets. The results of the experiments on three basic models are shown in the table below, where  Uncal indicates the basic models without calibration, BS represents the Brier Score, IR represents the Isotonic Regression, and bold indicates the best result, the subscript of each result refers to the standard deviation ( ×10$^{−3}$ ). As can be seen from the results, RECAL outperforms IR in most cases. We will conduct more extensive experimental analyses and refine the paper during the Camera-ready phase.\n\n**Logistic:**\n| Dataset           | Calibration         |    NLL(↓)       |    ECE(↓)    |   BS(↓)   |   \n|-------------------|---------------------|-----------------|---------------|----------------|\n|        | Uncal        |    0.2603$_{2.9}$    |    0.0248$_{3.4}$   |   0.1539$_{0.8}$ |   \n|    Bank Marketing    | IR        |    0.2545$_{1.1}$    |    0.02069$_{0.8}$   |   **0.1491$_{0.4}$** | \n|        | RECAL        |   **0.2537$_{2.1}$**    |    **0.0184$_{1.6}$**   |   0.1549$_{2.2}$  | \n|               |           |           |          |               |\n|               |    Uncal       |     0.2991$_{2.4}$      |     0.0608$_{4.1}$    |        0.1725$_{3.5}$        |\n|       Qsar-Biodeg        |      IR     |      0.2937$_{2.0}$     |      0.0604$_{3.2}$     |       0.1734$_{3.1}$        |\n|               |       RECAL    |     **0.2929$_{1.2}$**     |    **0.0504$_{3.6}$**     |        **0.1674$_{3.7}$**      |\n|               |           |           |          |               |\n|               |     Uncal      |      0.1739$_{2.8}$     |    0.0299$_{4.2}$     |     0.0899$_{0.9}$           |\n|         Seismic-Bumps      |    IR       |     0.1728$_{2.6}$       |     0.0290$_{1.7}$      |        **0.0882$_{0.4}$**       |\n|               |      RECAL     |     **0.1724$_{0.5}$**       |    **0.0216$_{3.4}$**     |     0.0899$_{0.2}$          |\n\n**GBDT:**\n\n| Dataset           | Calibration         |    NLL(↓)       |    ECE(↓)    |   BS(↓)   |   \n|-------------------|---------------------|-----------------|---------------|----------------|\n|        | Uncal        |    0.2082$_{2.3}$    |    0.0159$_{3.7}$   |   0.1281$_{2.2}$ |   \n|    Bank Marketing    | IR        |    0.2031$_{1.3 }$    |    0.0138$_{0.7}$   |   0.1279$_{2.1}$ | \n|        | RECAL        |   **0.2063$_{3.9}$**    |    **0.0127$_{2.5}$**   |   **0.1272$_{4.0}$**  | \n|               |           |           |          |               |\n|               |    Uncal       |     0.3061$_{2.6}$      |     0.0933$_{4.9}$    |        0.1833$_{3.3}$        |\n|       Qsar-Biodeg        |     IR     |   0.2993$_{1.9}$     |      0.0647$_{2.9}$     |       0.1725$_{7.1}$        |\n|               |       RECAL    |     **0.2816$_{3.0}$**     |    **0.0502$_{5.4}$**     |      **0.1717$_{2.8}$**      |\n|               |           |           |          |               |\n|               |     Uncal      |      0.1711$_{3.3}$     |    0.0317$_{5.0}$     |     **0.0872$_{1.4}$**            |\n|         Seismic-Bumps      |    IR       |     0.1683$_{5.4}$       |     0.0277$_{2.6}$      |        0.0881$_{3.2}$      |\n|               |      RECAL     |     **0.1674$_{4.3}$**       |    **0.0243$_{1.1}$**     |     0.0873$_{0.1}$          |\n\n**Tab Transformer:**\n\n| Dataset           | Calibration         |    NLL(↓)       |    ECE(↓)    |   BS(↓)   |   \n|-------------------|---------------------|-----------------|---------------|----------------|\n|        | Uncal        |    0.2119$_{1.4}$    |    0.0141$_{3.1}$   |   0.1391$_{3.1}$ |   \n|    Bank Marketing    | IR        |    0.2117$_{2.5}$    |    0.0111$_{1.8}$   |   0.1390$_{2.5}$ | \n|        | RECAL        |   **0.2087$_{2.1}$**    |    **0.0075$_{2.3}$**   |   **0.1388$_{3.6}$**  | \n|               |           |           |          |               |\n|               |    Uncal       |     0.2895$_{5.2}$      |     0.0816$_{6.7}$    |        0.1652$_{1.9}$        |\n|       Qsar-Biodeg        |     IR     |   0.2845$_{3.5}$     |      0.0796$_{2.6}$     |       0.1602$_{0.7}$        |\n|               |       RECAL    |     **0.2861$_{5.7}$**     |    **0.0600$_{7.6}$**     |      **0.1520$_{2.1}$**      |\n|               |           |           |          |               |\n|               |     Uncal      |      0.1803$_{4.0}$     |    0.0399$_{4.8}$     |     0.0886$_{1.6}$           |\n|         Seismic-Bumps      |    IR       |     0.1642$_{5.2}$       |     0.0261$_{2.3}$      |        0.0866$_{1.5}$      |\n|               |      RECAL     |     **0.1638$_{4.6}$**       |    **0.0212$_{5.6}$**     |     **0.0857$_{1.9}$**          |\n\nHistogram Binning changes the ranking of the model's predictive probabilities, which is inconsistent with our goal of not changing the accuracy of the model itself. Platt Scaling is often used to calibrate binary classification tasks, and our baseline TS has been shown to be superior to Platt Scaling (Guo et al., 2017). We will also keep an eye on the latest developments in scaling-based calibration methods and compare them with RECAL on tabular data.\n\nReferences：\n\nChuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper's central idea of utilizing graph neural networks to comprehend data relationships for aiding calibration is really interesting.\n* The way the paper is presented and explained is clear and easy to follow.\n* It's great to see better results compared to the widely used temperature scaling technique.", "reasons_to_reject": "A minor concern, but it would have been beneficial to see comparisons with other calibration mechanisms such as isotonic regression, histogram binning, and platt scaling.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper discusses the challenge in current machine learning models for tabular data where they lack accurate confidence estimation, crucial for high-risk applications like financial fraud detection. The study reveals that real-world tabular datasets inherently have implicit sample relations that can aid in more precise estimation. To address this, the authors introduce RECAL, a post-training confidence calibration framework that employs graph neural networks. This framework captures the relationships between different samples, improving calibration quality when compared to methods that disregard these sample relations.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "P7WGZ3uqAc", "reviewer": "Reviewer_nFCr", "summary": "", "questions": "Is there another better approach here regarding this area? I think it seems like the idea is fairly easy to understand and think about but really novel, so there really hasn't been any work on it before?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Q1:__ Only two methods were compared in the experiment, There was no comparison with the most recently published method, which may have overstated the promise of this approach\n\n__A1:__ Thanks for pointing this out. Following the suggestion of reviewer 1, we use Isotonic Regression as a calibration function to compare with RECAL on the three datasets: Bank Marketing, Qsar-Biodeg and Seismic-Bumps. The results of the experiments on three basic models are shown in the table below, where Uncal indicates the basic models without calibration, BS represents the Brier Score, IR represents the Isotonic Regression, and bold indicates the best result, the subscript of each result refers to the standard deviation ( ×10−3 ). \n\n**Logistic:**\n| Dataset           | Calibration         |    NLL(↓)       |    ECE(↓)    |   BS(↓)   |   \n|-------------------|---------------------|-----------------|---------------|----------------|\n|        | Uncal        |    0.2603$_{2.9}$    |    0.0248$_{3.4}$   |   0.1539$_{0.8}$ |   \n|    Bank Marketing    | IR        |    0.2545$_{1.1}$    |    0.02069$_{0.8}$   |   **0.1491$_{0.4}$** | \n|        | RECAL        |   **0.2537$_{2.1}$**    |    **0.0184$_{1.6}$**   |   0.1549$_{2.2}$  | \n|               |           |           |          |               |\n|               |    Uncal       |     0.2991$_{2.4}$      |     0.0608$_{4.1}$    |        0.1725$_{3.5}$        |\n|       Qsar-Biodeg        |      IR     |      0.2937$_{2.0}$     |      0.0604$_{3.2}$     |       0.1734$_{3.1}$        |\n|               |       RECAL    |     **0.2929$_{1.2}$**     |    **0.0504$_{3.6}$**     |        **0.1674$_{3.7}$**      |\n|               |           |           |          |               |\n|               |     Uncal      |      0.1739$_{2.8}$     |    0.0299$_{4.2}$     |     0.0899$_{0.9}$           |\n|         Seismic-Bumps      |    IR       |     0.1728$_{2.6}$       |     0.0290$_{1.7}$      |        **0.0882$_{0.4}$**       |\n|               |      RECAL     |     **0.1724$_{0.5}$**       |    **0.0216$_{3.4}$**     |     0.0899$_{0.2}$          |\n\n**GBDT:**\n\n| Dataset           | Calibration         |    NLL(↓)       |    ECE(↓)    |   BS(↓)   |   \n|-------------------|---------------------|-----------------|---------------|----------------|\n|        | Uncal        |    0.2082$_{2.3}$    |    0.0159$_{3.7}$   |   0.1281$_{2.2}$ |   \n|    Bank Marketing    | IR        |    0.2031$_{1.3 }$    |    0.0138$_{0.7}$   |   0.1279$_{2.1}$ | \n|        | RECAL        |   **0.2063$_{3.9}$**    |    **0.0127$_{2.5}$**   |   **0.1272$_{4.0}$**  | \n|               |           |           |          |               |\n|               |    Uncal       |     0.3061$_{2.6}$      |     0.0933$_{4.9}$    |        0.1833$_{3.3}$        |\n|       Qsar-Biodeg        |     IR     |   0.2993$_{1.9}$     |      0.0647$_{2.9}$     |       0.1725$_{7.1}$        |\n|               |       RECAL    |     **0.2816$_{3.0}$**     |    **0.0502$_{5.4}$**     |      **0.1717$_{2.8}$**      |\n|               |           |           |          |               |\n|               |     Uncal      |      0.1711$_{3.3}$     |    0.0317$_{5.0}$     |     **0.0872$_{1.4}$**            |\n|         Seismic-Bumps      |    IR       |     0.1683$_{5.4}$       |     0.0277$_{2.6}$      |        0.0881$_{3.2}$      |\n|               |      RECAL     |     **0.1674$_{4.3}$**       |    **0.0243$_{1.1}$**     |     0.0873$_{0.1}$          |\n\n**Tab Transformer:**\n\n| Dataset           | Calibration         |    NLL(↓)       |    ECE(↓)    |   BS(↓)   |   \n|-------------------|---------------------|-----------------|---------------|----------------|\n|        | Uncal        |    0.2119$_{1.4}$    |    0.0141$_{3.1}$   |   0.1391$_{3.1}$ |   \n|    Bank Marketing    | IR        |    0.2117$_{2.5}$    |    0.0111$_{1.8}$   |   0.1390$_{2.5}$ | \n|        | RECAL        |   **0.2087$_{2.1}$**    |    **0.0075$_{2.3}$**   |   **0.1388$_{3.6}$**  | \n|               |           |           |          |               |\n|               |    Uncal       |     0.2895$_{5.2}$      |     0.0816$_{6.7}$    |        0.1652$_{1.9}$        |\n|       Qsar-Biodeg        |     IR     |   0.2845$_{3.5}$     |      0.0796$_{2.6}$     |       0.1602$_{0.7}$        |\n|               |       RECAL    |     **0.2861$_{5.7}$**     |    **0.0600$_{7.6}$**     |      **0.1520$_{2.1}$**      |\n|               |           |           |          |               |\n|               |     Uncal      |      0.1803$_{4.0}$     |    0.0399$_{4.8}$     |     0.0886$_{1.6}$           |\n|         Seismic-Bumps      |    IR       |     0.1642$_{5.2}$       |     0.0261$_{2.3}$      |        0.0866$_{1.5}$      |\n|               |      RECAL     |     **0.1638$_{4.6}$**       |    **0.0212$_{5.6}$**     |     **0.0857$_{1.9}$**          |\n\nAs can be seen from the results, RECAL outperforms IR in most cases. Additionally, Histogram Binning changes the ranking of the model's predictive probabilities, which is inconsistent with our goal of not changing the accuracy of the model itself. Also, we investigated recent work and to the best of our knowledge, we are the first to use implicit relationships between tabular data to guide model calibration with a scaling-based approach. We found that traditional machine learning methods applied to tabular datasets are also in need of calibration. Unlike current calibration methods that focus only on deep neural networks, our approach is equally valid on traditional machine learning methods.\n\n__Q2:__ The authors do not provide some source code to demo the algorithm so reproducibility is questionable.\n\n__A2:__ Due to time limitation, the code was not organized previously. We will clean up the code in the git repository and make it public after the paper is accepted. In addition, we call the LogisticRegression in sklearn, in order to get the logits before softmax, users need to modify the source code in the environment. To avoid user changing the environment, we are replacing the previous calls to sklearn with our manual implementation of LogisticRegression, which will also be made available in the git repository after the paper is accepted.\n\n__Q3:__ Is there another better approach here regarding this area? I think it seems like the idea is fairly easy to understand and think about but really novel, so there really hasn't been any work on it before?\n\n__A3:__ Thank you for your question. After our careful survey, we believe that we are indeed the first work to focus on model calibration applied to tabular data. Unlike previous work, we are the first to attempt to use implicit relationships between tabular data to guide model calibration, which is generalizable to a wide range of models applied on tabular data. We believe that our work is important for improving the fairness and reliability of machine learning on tabular datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written especially introduction and related work and makes it easy to understand.\n2. I should mention right off the bat that I don't have much experience in this specific field. But I think the prospect of considering the study of relationships between different samples of data from the table is very promising. So if as the authors say they are the first to propose the method, I think it is very novel. It also depends on the opinion of other fellow reviewers. \n3. This article conducts experiments on a large number of datasets to illustrate the validity of its methods. Analysis for confidence estimation of result is complete.\n", "reasons_to_reject": "1.\tOnly two methods were compared in the experiment, There was no comparison with the most recently published method, which may have overstated the promise of this approach\n2. The authors do not provide some source code to demo the algorithm so reproducibility is questionable.\n\n", "questions_to_authors": "Is there another better approach here regarding this area? I think it seems like the idea is fairly easy to understand and think about but really novel, so there really hasn't been any work on it before?", "paper_topic_and_main_contributions": "This paper targets at the confidence estimation of machine learning models applied to tabular data, which contains implicit sample relations. The proposed method introduces a general post-training confidence calibration framework by employing graph neural networks. This model employs existing GBDT to select the most significant feature and then classify given samples with the feature for graph construction. Finally, this method adopt GCN as a nonlinear calibration function to explore the graph information and obtain normalized scores. Different from previous methods, element-wise temperature scaling is proposed to calculate the calibrated prediction result. Extensive experiments on tabular datasets show the superiority of the proposed method against another method(just a method).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "none", "Typos_Grammar_Style_and_Presentation_Improvements": "none"}, "extra_text": {}}, {"review_id": "mLPr9QuUwZ", "reviewer": "Reviewer_tBxh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Q1:__ Utilizing the GBDT features in a Platt-scaling equivalent using a linear regression recalibration is missing\n\n__A1:__ Thanks for pointing this out. Following the suggestion of reviewer 1, we use Isotonic Regression as a calibration function to compare with RECAL on three datasets: Bank Marketing, Qsar-Biodeg and Seismic-Bumps. The results of the experiments on three basic models are shown in the table below, where  Uncal indicates the basic models without calibration, BS represents the Brier Score, IR represents the Isotonic Regression, and bold indicates the best result, the subscript of each result refers to the standard deviation ( ×10$^{−3}$ ). \n\n**Logistic:**\n| Dataset           | Calibration         |    NLL(↓)       |    ECE(↓)    |   BS(↓)   |   \n|-------------------|---------------------|-----------------|---------------|----------------|\n|        | Uncal        |    0.2603$_{2.9}$    |    0.0248$_{3.4}$   |   0.1539$_{0.8}$ |   \n|    Bank Marketing    | IR        |    0.2545$_{1.1}$    |    0.02069$_{0.8}$   |   **0.1491$_{0.4}$** | \n|        | RECAL        |   **0.2537$_{2.1}$**    |    **0.0184$_{1.6}$**   |   0.1549$_{2.2}$  | \n|               |           |           |          |               |\n|               |    Uncal       |     0.2991$_{2.4}$      |     0.0608$_{4.1}$    |        0.1725$_{3.5}$        |\n|       Qsar-Biodeg        |      IR     |      0.2937$_{2.0}$     |      0.0604$_{3.2}$     |       0.1734$_{3.1}$        |\n|               |       RECAL    |     **0.2929$_{1.2}$**     |    **0.0504$_{3.6}$**     |        **0.1674$_{3.7}$**      |\n|               |           |           |          |               |\n|               |     Uncal      |      0.1739$_{2.8}$     |    0.0299$_{4.2}$     |     0.0899$_{0.9}$           |\n|         Seismic-Bumps      |    IR       |     0.1728$_{2.6}$       |     0.0290$_{1.7}$      |        **0.0882$_{0.4}$**       |\n|               |      RECAL     |     **0.1724$_{0.5}$**       |    **0.0216$_{3.4}$**     |     0.0899$_{0.2}$          |\n\n**GBDT:**\n\n| Dataset           | Calibration         |    NLL(↓)       |    ECE(↓)    |   BS(↓)   |   \n|-------------------|---------------------|-----------------|---------------|----------------|\n|        | Uncal        |    0.2082$_{2.3}$    |    0.0159$_{3.7}$   |   0.1281$_{2.2}$ |   \n|    Bank Marketing    | IR        |    0.2031$_{1.3 }$    |    0.0138$_{0.7}$   |   0.1279$_{2.1}$ | \n|        | RECAL        |   **0.2063$_{3.9}$**    |    **0.0127$_{2.5}$**   |   **0.1272$_{4.0}$**  | \n|               |           |           |          |               |\n|               |    Uncal       |     0.3061$_{2.6}$      |     0.0933$_{4.9}$    |        0.1833$_{3.3}$        |\n|       Qsar-Biodeg        |     IR     |   0.2993$_{1.9}$     |      0.0647$_{2.9}$     |       0.1725$_{7.1}$        |\n|               |       RECAL    |     **0.2816$_{3.0}$**     |    **0.0502$_{5.4}$**     |      **0.1717$_{2.8}$**      |\n|               |           |           |          |               |\n|               |     Uncal      |      0.1711$_{3.3}$     |    0.0317$_{5.0}$     |     **0.0872$_{1.4}$**            |\n|         Seismic-Bumps      |    IR       |     0.1683$_{5.4}$       |     0.0277$_{2.6}$      |        0.0881$_{3.2}$      |\n|               |      RECAL     |     **0.1674$_{4.3}$**       |    **0.0243$_{1.1}$**     |     0.0873$_{0.1}$          |\n\n**Tab Transformer:**\n\n| Dataset           | Calibration         |    NLL(↓)       |    ECE(↓)    |   BS(↓)   |   \n|-------------------|---------------------|-----------------|---------------|----------------|\n|        | Uncal        |    0.2119$_{1.4}$    |    0.0141$_{3.1}$   |   0.1391$_{3.1}$ |   \n|    Bank Marketing    | IR        |    0.2117$_{2.5}$    |    0.0111$_{1.8}$   |   0.1390$_{2.5}$ | \n|        | RECAL        |   **0.2087$_{2.1}$**    |    **0.0075$_{2.3}$**   |   **0.1388$_{3.6}$**  | \n|               |           |           |          |               |\n|               |    Uncal       |     0.2895$_{5.2}$      |     0.0816$_{6.7}$    |        0.1652$_{1.9}$        |\n|       Qsar-Biodeg        |     IR     |   0.2845$_{3.5}$     |      0.0796$_{2.6}$     |       0.1602$_{0.7}$        |\n|               |       RECAL    |     **0.2861$_{5.7}$**     |    **0.0600$_{7.6}$**     |      **0.1520$_{2.1}$**      |\n|               |           |           |          |               |\n|               |     Uncal      |      0.1803$_{4.0}$     |    0.0399$_{4.8}$     |     0.0886$_{1.6}$           |\n|         Seismic-Bumps      |    IR       |     0.1642$_{5.2}$       |     0.0261$_{2.3}$      |        0.0866$_{1.5}$      |\n|               |      RECAL     |     **0.1638$_{4.6}$**       |    **0.0212$_{5.6}$**     |     **0.0857$_{1.9}$**          |\n\nAs can be seen from the results, RECAL outperforms IR in most cases. Additionally, Histogram Binning changes the ranking of the model's predictive probabilities, which is inconsistent with our goal of not changing the accuracy of the model itself. Platt Scaling is often used to calibrate binary classification tasks, and our baseline TS has been shown to be superior to Platt Scaling (Guo et al., 2017). We will also keep an eye on the latest developments in scaling-based calibration methods and compare them with RECAL on tabular data.\n\nReferences:\n\nChuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR.\n\n__Q2:__ Typos Grammar Style And Presentation Improvements\n\n__A2:__ We appreciate your kind suggestions. We will revise the manuscript thoroughly and carefully.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Outperforms temperature scaling and MLP -based post-processed calibration efforts on 7 tasks, across 3 base models and metrics", "reasons_to_reject": "Utilizing the GBDT features in a Platt-scaling equivalent using a linear regression recalibration is missing", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes to calibrate the probabilities of a classifier that operates over tabular data by incorporating the structure within the table learnt using gradient boosted decision tree and a graph convolutional network on a validation set. \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Abstract: s/wildly/widely; lack of/lack, \n\nmany other typos throughout the text\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper delves into the problem of sample-relation guided confidence calibration on tabular data and introduces the RECAL framework, a post-training confidence calibration approach utilizing graph neural networks to model sample relations.

The overall consensus among most PC members is that this paper demonstrates good soundness but elicits mixed feelings regarding its level of excitement. After thoroughly reviewing the paper and the accompanying reviews, several notable concerns have emerged. These include:
1. Lack of Comparisons: The paper would benefit from providing comparisons with existing approaches or techniques to contextualize its contributions better.
2. Reproducibility: It is better to provide the source code for better reproducibility.

In my view, while the reviewers generally acknowledge the soundness of this paper, its excitement appears somewhat ambiguous. Consequently, I recommend that the authors carefully address these concerns during their revisions to strengthen the overall quality and impact.
Decision: Accept-Findings