# A Study on the Calibrated Confidence of Text Classification Using a Variational Bayes
## 4 Experimental Results
### Comparison of Improvements in Calibrated Confidence in Document Classification
When the target variable is _Valuation_, there were 566 and 5015 observations with labels Y and N in the training data, respectively. Using Equation (2), \(G(X,\,y)\) generated 5954 observations whose label is Y. When raw data and generated data were concatenated, the number of observations with label Y became 6520. That is, the labels in the data augmented through the proposed method had a Y:N ratio of 1.3 (=optimal value of \(L\)). The model \(f_{X}\) learns the data that contains the merged training data and generated data. Figure 2 shows the distribution of probabilities when the labels in the test data are predicted. In the _Litigation_ cases, the likelihood of the proposed method increased. In the _Valuation_ and _Transfer_ cases, the likelihood was higher when it was \(\geq\)0.4, indicating that the confidence was calibrated. Therefore, the proposed method can calibrate the confidence. Finally, Figure 2(a)-c shows the comparison result of \(\mathcal{L}_{Y}(X_{test};\,\cdot\,)\) obtained when the test data with the actual label Y were applied to the baseline and the proposed method (see Equation (3)). The \(\mathcal{L}_{Y}\big{(}X_{test};\,X_{Gen}^{K_{L}}\big{)}\) obtained through the proposed method tends to be higher than the baseline in all tasks. Figure 2(d)-f shows the distribution of \(\mathcal{L}_{Y}(X_{test};\,\cdot\,)\) obtained through the baseline and the proposed method. The distributions in the figure indicate how well the proposed method secures calibrated confidence compared to the baseline. Therefore, we compare the homogeneity of the two distributions to statistically test Hypothesis 3. Avg_baseline in Table 5 is the average of the probability that an observation with actual label Y is correctly classified as Y by the model. Similarly, Avg_\(X_{Gen}^{K_{L}}\) is a value measured by the proposed method. For example, in Figure 2(e), the mean of \(\mathcal{L}_{Y}(X_{test};X)\), which is a negative log-likelihood, is \(-1.855\) and is 0.157 (=\(e^{-1.855}\)) when converted to a probability. Similarly, the mean of \(\mathcal{L}_{Y}\big{(}X_{tests};\,X_{Gen}^{K_{1,3}}\big{)}\) is \(-1.043\); it is 0.353 when converted to a probability. When the \(L\) of the _Valuation_ case was 1.3, a miscalibration phenomenon that resulted in a large difference between the confidence and the F1-score was observed at the baseline. For example, in the valuation, the existing accuracy was 0.157 and the difference from the F1-score was 0.238, which was very large. Conversely, the difference between the value obtained by the proposed method and the F1-score was very small at 0.042. Calculating with the same logic, the proposed method evidently calibrates the confidence from a minimum of 0.02 to a maximum of 0.04. However, the proposed method showed a similar level of confidence to that of the accuracy. Therefore, our method can calibrate the confidence better than the baseline. Figure 2: Distribution of the prediction probability for the target variables (when the generative model was C-VAE and the predictors were quantitative indicators and text of documents). (**a**) This figure shows the distribution of the prediction probabilities when the target variable was _Litigation_, which shows that the overall likelihood increased when the proposed method was used; (**b**) predicted distribution when the target variable was _Valuation:_ this figure shows that the probability increases by 0.4 or more; (**c**) predicted distribution when the target variable is _Transfer:_ it shows a similar pattern to the distribution in (**b**). Figure 4 shows the results of comparing the differences in accuracy and likelihood depending on the generative models and predictors. When the target variables are _Litigation_ and _Valuation_, the confidence is the most calibrated when the generative model is C-VAE and the predictors are document texts. When the target variable was _Transfer_, the difference between accuracy and likelihood was smallest when the quantitative indicators were used together.