# A Study on the Calibrated Confidence of Text Classification Using a Variational Bayes
## 5 Discussion  
Recently, various applications have pointed out that the confidence of the predictive model does not reflect the statistical consistency between the accuracy and the probability of prediction. TM is one field that has a high uncertainty of predictions for reasons such as the time lag in technological development or biased expert decision making. In particular, patents that reflect the TM contain a lot of noise due to various factors. For example, through technology valuation, companies can find excellent technologies that they can then apply to their business models. However, as the value of any technology changes over time, businesses should consider the uncertainty inherent in data to make the right decisions. Therefore, this study proposes a method that reduces uncertainty about TM. Previous studies have mainly devised visualization methods for comparison of expected and observed accuracy, to approximate expected values for the difference between confidence and accuracy or to estimate calibration errors. Recently, researchers have discovered that a model's confidence can be calibrated through the process of increasing the diversity of data representation. Therefore, this study used a VB-based generative model to augment the document representation in various ways. In addition, we were able to intuitively grasp the degree to which the confidence was calibrated by visualizing the predictive probability and log-likelihood obtained using the proposed method. The experiment of the proposed method was carried out by collecting actual patents. The proposed method calibrated the distribution of prediction probability to be less biased than before (see Figure 2). In particular, the probability of the predictive model correcting the ground truth occurred frequently at \(\geq\)0.4. In addition, the log-likelihood of the data with actual label Y was larger than the baseline in all cases (see Figure 3). Specifically, when the target variable was _Litigation_, the difference between accuracy and F1-score decreased from 0.055 to 0.025. Similarly, when the target variable was _Valuation_ and _Transfer_, the difference between the two measures decreased significantly (see Figure 4). We found  
Figure 4: Comparison of differences between the accuracy and likelihood. (**a**) When the generative model was C-VAE and document texts were used as a predictor, the difference was the smallest; (**b**) it was most suitable when C-VAE and document texts were used as in where the target variable was _Litigation_; (**c**) the difference was the smallest when the generative model was C-VAE and quantitative indicators and document texts were used as predictors. through experiments that the degree to which confidence was calibrated decreased as the proportion of labels became imbalanced. To reduce uncertainty, previous studies have proposed methods for measuring the confidence of models. However, their limitation was that they measured the uncertainty of a model that had already been trained. Therefore, an alternate method was suggested that reduced uncertainty by calibrating the confidence while training the model. Specifically, some methods impose a penalty using a confidence score or by shifting data. Based on these studies, we proposed a method to calibrate confidence in text classification tasks using imbalanced data. To this end, the VB used in this study (i) works well in various fields such as healthcare, (ii) is suitable for multimodal data such as patents, and (iii) is a computationally scalable method. Nevertheless, this study has the following limitations:  
* This paper did not present an optimization method to find the hyperparameter \(L\) in the proposed methodology. The hyperparameter \(L\), which determines how much data are generated, is expected to be related to the precursors of the data. In the experiment, we determined the hyperparameters using a greedy search.