# Complement-Class Harmonized Naive Bayes Classifier
## 2 Background and Related Work
The Discriminatively Weighted Naive Bayes (DWNB) [37] method assigns instances different weights depending on how difficult they are to classify. In [48], the probability estimation problem was modeled as an optimization problem and metaheuristic approaches were used to find a better probability estimation. FTNB [39] was proposed to address the problem of data scarcity for the NB classifier. However, the fine-tuning procedure in FTNB [39] leads to overfitting problems and makes NB less tolerant to noise, therefore, a more noise tolerant FTNB was proposed in [1] and also, a FTNB combined with instant weighting was proposed in [41]. Despite the enhancements of FTNB [1; 39; 41], the fine-tuning procedure is similar to correlation-based attribute weighting methods [27; 29; 30] where calculating the update amount (weight) does not simultaneously incorporate the inter-correlation (between classes) distance measure for each attribute value. More specifically, Information gain \(IG\big{(}C\big{|}a_{ij}\big{)}\) is used to measure the difference between a priori and a posteriori entropies of a class target, \(C\), given the observation of feature a, and intuitively, a feature with higher information gain deserves a higher weight [27]. However, in [27], the author proposed the Kullback-Leibler Measure (KL) Equation (4) as a measure of divergence and as the information content of a feature value \(a_{ij}\) to overcome the possible zero or negative values' limitations of IG as a feature weighting. \[\text{KL}\big{(}C\mid a_{ij}\big{)}=\sum_{c}P(c\mid a_{ij})\log\frac{P\big{(}c \mid a_{ij}\big{)}}{P(c)} \tag{4}\]  
where \(a_{ij}\) corresponds to the \(j\) value of the \(i\)-th feature in training data. Thus, the weight of a feature can be defined as the weighted average of the KL measures across the feature values. \(\text{KL}\big{(}C\mid a_{ij}\big{)}\) and mutual information \(\text{MI}\big{(}C\mid a_{ij}\big{)}\) Equation (5) are employed in [29; 30] as two different base methods to measure the significance (relevance) between each attribute value and class target and consequently, the attribute value weights for the NB classifier. \[I(a_{ij};C)=\sum_{c}P(a_{i},c)\log\frac{P(a_{i},c)}{P(a_{i})P(c)}. \tag{5}\]The expectation is that a highly predictive attribute value should be strongly associated with class (maximum attribute value mutual relevance) [30]. In FTNB [39], every misclassified training instance is fine-tuned by updating its conditional probability terms of actual (ground truth label) and predicted classes. In FTNB [39], conditional probability terms of actual class are increased by an amount that is proportional to the difference between \(p\big{(}a_{j}|c_{actual}\big{)}\) and \(P_{max}\big{(}a_{j}|c_{actual}\big{)}\), and contrarily, the conditional probability terms of predicted class decreased by an amount that is proportional to the difference between \(p\big{(}a_{j}|c_{predicted}\big{)}\) and \(P_{min}\big{(}a_{j}|c_{predicted}\big{)}\), using Equations (6) and (7), respectively. \[\delta_{t+1}\big{(}a_{j},c_{actual}\big{)}=\eta\cdotp\big{(}\alpha\cdotp P_{ max}\big{(}a_{j}|c_{actual}\big{)}-p\big{(}a_{j}|c_{actual}\big{)}\big{)}\cdot error \tag{6}\]  
\[\delta_{t+1}\big{(}a_{j},c_{predicted}\big{)}=-\eta\cdotp\big{(}\alpha\cdotp p \big{(}a_{j}|c_{predicted}\big{)}-P_{min}\big{(}a_{j}|c_{predicted}\big{)} \big{)}\cdot error \tag{7}\]  
where \(\eta\) is a learning rate between zero and one, used to decrease the update step, and \(\alpha\) is constant = 2, and error is the general difference between the two posteriors of the actual and predicted classes. The fine-tuning process will continue as long as training classification accuracy keeps improving. There is a fundamental problem with correlations measures (KL) Equation (4) (MI), Equation (5), and (FTNB) Equations (6) and (7) where they would consider a relatively equally distributed but highly correlated attribute value with two or more classes as a discriminative attribute value.