# Decision Support System for Medical Diagnosis Utilizing Imbalanced Clinical Data
## 3 Proposed Methodology
### Regularized Boosting Approach for Multi-Class Classification  
In each iteration of ensemble multi-class classification model, some examples are classified incorrectly by the current classifier after being classified correctly by the classifier in the previous iteration; in particular, the distribution of multiple classes is imbalanced. A regularization parameter was introduced by [PERSON] et al. (2016) into the convex loss function to calculate the classifier weight. This parameter penalized the weight of the current classifier if the classifier misclassifies examples that were classified correctly by the previous classifier. The regularized multi-class classification method aims to keep the correct classifications of minority examples, control the decision boundary towards minority examples, and prevent the bias derived from the large amount of majority examples.  
After each learning iteration, the weight of current classifier is calculated as follows:  
\[\alpha_{t}=\frac{1}{2}\log(\frac{1-e_{t}}{e_{t}})+\frac{1}{2}\log(\delta_{t}( C-1)) \tag{8}\]  
where the regularization parameter \(\delta_{t}\) is initialized as 1. According to the loss function, the weights of misclassified examples are adjusted to increase while the weights of those classified correctly are adjusted to decrease. The weights of examples are updated as follows:  
\[w_{t}=\left\{\begin{array}{l}w_{t-1}(i)e^{-\alpha_{t}},\forall x_{i},f_{i}(x _{i})=y_{i}\\ w_{t-1}(i)e^{\alpha_{t}},\forall x_{i},f_{i}(x_{i})\
eq y_{i}\end{array}\right. \tag{9}\]  
After updating the weights of examples, the weights would be normalized.  
Misclassified examples are categorized into two classes: (i) second-round-misclassified examples \(X_{c}=\{x_{i};f_{t}(x_{i})\
eq y_{i}\text{ and }f_{t-1}(x_{i})=y_{i}\}\), which are classified incorrectly by current classifier but classified correctly by previous classifier; and (ii) two-rounds-misclassified examples \(X_{m}=\{x_{i};f_{t}(x_{i})\
eq y_{i}\text{ and }f_{t-1}(x_{i})\
eq y_{i}\}\), which are classified incorrectly by both the current classifier and the previous classifier. The weighted error is calculated by misclassified examples as follows:  
\[\begin{array}{l}e_{t}=\sum\limits_{i\in X_{c}}w_{t-1}(i)+\sum\limits_{i\in X _{m}}w_{t-1}(i)\\ =\sum\limits_{i\in X_{c}}w_{t-2}(i)\big{(}\frac{\delta_{t-1}(C-1)(1-e_{t-1})}{ \epsilon_{t-1}}\big{)}^{-\frac{1}{2}}+\sum\limits_{i\in X_{m}}w_{t-2}(i)\big{(} \frac{\delta_{t-1}(C-1)(1-e_{t-1})}{\epsilon_{t-1}}\big{)}^{\frac{1}{2}}\end{array} \tag{10}\]  
The regularization term penalizes the current classifier that had misclassified the second-round-misclassified examples by changing its weight. To derive the regularization term, it assumes that all examples misclassified by the current classifier are also misclassified by the previous classifier. Thus, the exponent in expression of calculating the error of second-round-misclassifiedexamples transfers into positive. In the above assumption, the maximum possible error is computed as follows:  
\[e_{t}^{*}=\sum_{i\in(X_{c}\cup X_{m})}w_{t-2}(i)(\frac{\delta_{t-1}(C-1)(1-e_{t- 1})}{e_{t-1}})^{\frac{1}{2}} \tag{11}\]  
Then, the expression of the actual weighted error is computed as follows:  
\[e_{t}=e_{t}^{*}\delta_{t}^{\frac{1}{2}} \tag{12}\]  
Accordingly, the explicit expression of regularization term can be derived as follows:  
\[\delta_{t}=\frac{e_{t}^{2}e_{t-1}}{(\sum\limits_{i\in X_{c}\cup X_{m}}w_{t-2}( i))^{2}(1-e_{t-1})\delta_{t-1}(C-1)} \tag{13}\]  
Both weighted error and regularization term are used to compute the weight of current classifier as shown in Equation (5). The regularization term is adjusted in each iteration in terms of the performances of the current classifier and the previous classifier. Considering this scheme, the weighted error needs to follow the below equation:  
\[(1-e_{t})\delta(C-1)>e_{t} \tag{14}\]  
Thus, the weighted error boundary of the current classifier \(t\) is as follows:  
\[e_{t}<\frac{1}{1+\delta_{t}^{-1}(C-1)^{-1}} \tag{15}\]