# A Study on the Calibrated Confidence of Text Classification Using a Variational Bayes
## 2 Theoretical Background  
In this study, a generative model is used to calibrate the confidence generated when a predictive model is applied to TM. For versatility in the proposed methodology, we use the Conditional-Variational AutoEncoder (C-VAE), a VB-based generative model that can selectively generate data that belong to a specific class [52].  
Let \(z\) be the latent variable that is generated from the prior distribution \(p_{\theta}(z)\). The input data \(X\) for C-VAE is generated by the distribution \(p_{\theta}(X|z)\) for \(z\), which is \(z\sim p_{\theta}(z)\)Furthermore, \(X=\{x\}_{i=1}^{N}\) consists of N _i.i.d._ samples of the variable \(x\) conditioned on \(x:\ x\sim p_{\theta}(x|z)\).  
Next, let \(y\) be the target variable that is generated from the distribution \(p_{\theta}(y|x,\ z)\). The variable \(y\), having \(M\) number of categories, is expressed as \(\{c_{m}\}_{m=1}^{M}\). Then, the target variable \(y_{i}\) with category \(c_{m}\) is \(e_{m}\). Note that \(e_{m}\) is a vector of the standard base, where \(e_{m}\) denotes a vector with 1 as the m-th element and 0 everywhere else.  
Equation (1) is the conditional log-likelihood of \(\widetilde{\mathcal{L}}_{CVAE}(x,\ y;\ \theta,\ \phi)\) when \(z=g_{\theta}(x,\ y,\ \epsilon)\), \(\epsilon\sim N(0,\ I)\). \(q_{\phi}(z|x)\), known as the recognition model, was introduced to approximate the actual posterior \(p_{\theta}(z|x)\) and was reparametrized to the deterministic differentiable function \(g_{\phi}(\cdot,\ \cdot)\) using the variable \(x\) and the noise variable \(\epsilon\) as arguments. The generative model \(G(X,\ y)\) that maximizes Equation (1) can be divided into an encoder and a decoder. When the number of latent dimensions is \(J\), the encoding result of the i-th sample is \(z_{i}=\{z_{i,j}\}_{j=1}^{J}\). When the category of the i-th target variable \(y_{i}\) is \(c_{m}\), the latent vector of the data is \(z_{i}^{c_{m}}=\left\{z_{i,j}^{c_{m}}\right\}_{j=1}^{J}\). The function \(KL\), called the Kullback-Leibler divergence, computes the difference between two inputs:  
\[\log p_{\theta}(y|x)\geq-KL(q_{\theta}(z|x,\ y)||p_{\theta}(z|x))+\log p_{ \theta}(y|x,\ z) \tag{1}\]  
The encoder of C-VAE represents the input data as a disentangled vector according to the label. The decoder of C-VAE receives a specific label along with the vector, and then generates data. This generative model can be utilized in various applications. In particular, VB shows excellent performance for detecting anomalies in network intrusion [53], credit card fraud [54; 55], and medical diagnoses [56; 57] when labels are imbalanced. Furthermore, many previous studies have demonstrated that VB shows well-calibrated results in healthcare, which is one of the fields sensitive to uncertainty [58; 59; 60; 61].  
The generative model used in this study has the following advantages. First, it is efficient when data labels are imbalanced [62]; the labels for patent type obtained through TM are imbalanced. For example, there are fewer transferred technologies than those that are not. Since patents have these characteristics, there is a high risk of uncertainty in predictions; therefore, VB can be effectively utilized in TM. Second, the VB works well with multimodal data [63]. Patents containing quantitative indicators, such as the number of inventors, and texts, such as abstracts, are multimodal. Since the calibration of confidence is affected by the generative model, the proposed method can be expected to show sufficient performance even with VB. Finally, VB is a computationally tractable method [64]. VB, which approximates the true posterior in Bayesian inference, has a low computational cost for training and a low risk of gradient divergence.