# Predicting Firms' Credit Ratings Using Ensembles of Artificial Immune Systems and Machine Learning – An Over-Sampling Approach

Petr Hájek, Vladimír Olej

► To cite this version:

Petr Hájek, Vladimír Olej. Predicting Firms’ Credit Ratings Using Ensembles of Artificial Immune Systems and Machine Learning – An Over-Sampling Approach. 10th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Sep 2014, Rhodes, Greece. pp.29-38, 10.1007/978-3-662-44654-6_3 . hal-01391290

HAL Id: hal-01391290

https://inria.hal.science/hal-01391290

Submitted on 3 Nov 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Predicting Firms' Credit Ratings using Ensembles of Artificial Immune Systems and Machine Learning – An Over-Sampling Approach

Petr Hájek, Vladimír Olej

Institute of System Engineering and Informatics
Faculty of Economics and Administration, University of Pardubice, Studentská 84
532 10 Pardubice, Czech Republic
{petr.hajek, vladimir.olej}@upce.cz

**Abstract.** This paper examines the classification performance of artificial immune systems on the one hand and machine learning and neural networks on the other hand on the problem of forecasting credit ratings of firms. The problem is realized as a two-class problem, for investment and non-investment rating grades. The dataset is usually imbalanced in credit rating predictions. We address the issue by over-sampling the minority class in the training dataset. The experimental results show that this approach leads to significantly higher classification accuracy. Additionally, the use of the ensembles of classifiers makes the prediction even more accurate.

**Keywords.** Credit rating, artificial immune systems, machine learning, neural networks, classification performance, balanced and imbalanced dataset, SMOTE, AdaBoost.

## 1 Introduction

Credit rating addresses an issuer's overall capacity and willingness to meet its financial obligations, thus reducing the information asymmetry between issuers and investors. However, credit ratings are based on costly analysis performed by professionals, which is why credit rating forecasting has attracted considerable recent interest. Various artificial intelligence (AI) methods have been applied to model the complex non-linear relations between input variables and target classes (see [7] for an example of a review). Recent efforts have shown that approaches integrating feature selection process and an appropriate AI method provide the best classification performance [6]. However, although the ensembles of classifiers have shown promising results in related fields such as credit risk and bankruptcy forecasting [15], significantly insufficient attention has been paid to them in credit rating forecasting. In addition, no research has been found that examines the effect of over-sampling the minority class in credit rating data. Therefore, the aim of this paper is to examine the effect of: (1) over-sampling the minority class in credit rating data; and (2) ensembling base classifiers. The difficulty in predicting credit ratings also stems from the fact that a multitude of
sources is used in the credit rating analysis, involving both quantitative data (accounting and financial data drawn from financial statements) and qualitative assessments. The sentiment analysis of corporate annual reports has recently been used to address the issue of qualitative assessment [8]. We follow this approach and use the chosen sentiment categories in addition to corporate financial indicators as input variables.

We employ two categories of AI methods to examine the given aims, (1) artificial immune systems (AISs) and (2) machine learning (ML). AISs mimic the processes and mechanisms of biological immune systems and we specifically use Artificial Immune Recognition Systems (AIRSs) [16] and the Clonal Selection Classification Algorithm (CSCA) [1]. Out of the second category, we use C4.5 decision trees (DTs) [12], Support Vector Machines (SVMs) [13], Radial Basis Function Neural Networks (RBFs) [9] and Multilayer Perceptrons (MLPs) [9]. The research to date has tended to focus on the ensembles of classifiers of the latter category while little attention has been paid to the ensembles of AISs [5].

This paper is organized as follows. In the next section, we provide the description of the dataset. Given the fact that the classes are imbalanced in the dataset, we use the Synthetic Minority Oversampling Technique (SMOTE) algorithm [10] to modify the training dataset. The third section presents the results of experiments. First, the effect of over-sampling is tested and then we employ the AdaBoost algorithm [4] to generate the ensembles of the base classifiers (AIRS1, AIRS2, AIRS2-p, CSCA; and DT, SVM, RBF, MLP). For both categories, the classification performance is examined on (imbalanced) testing dataset depending on the proportion of training dataset generated. The last section discusses the results and concludes the paper.

## 2 Problem Formulation and Dataset

The prediction of firms' credit ratings was realized as a two-class problem. The classes were represented by investment grade (IG, low default risk) and non-investment grade (NG, high default risk), assigned by a highly regarded Standard & Poor's rating agency in 2011. The investment grade position is critical to many investors due to the restrictions imposed on investment instruments.

We used two main groups of input variables in this study, financial indicators and sentiment indicators (see Table 1). More specifically, we used several subgroups of financial indicators such as size, profitability ratios, liquidity ratios, leverage ratios and market value ratios. Sentiment indicators refer to the business position of a company (business risk, character (reputation), organizational problems, management evaluation, accounting quality, etc.). The financial indicators were drawn from the Value Line database, while sentiment indicators were drawn from annual reports available at the U.S. Securities and Exchange Commission EDGAR System. Both groups of input variables were collected for 520 U.S. companies (selected from the Standard & Poor's database) in the year 2010, 195 classified as IG and 325 as NG. Mining and financial companies were excluded from the dataset since they require specific input variables.
The sentiment indicators required linguistic pre-processing (tokenization and lemmatization) and subsequent comparison with the financial dictionary provided by [11]. Then, the tf.idf term weighting scheme was applied to obtain the importance of terms and an average weight was calculated for each sentiment category (negative, positive, uncertainty, litigious, modal strong and modal weak). See [8] for the detailed information on the collection of data.

<table><caption>**Table 1.** Input and output variables describing the dataset.</caption><thead><tr><th>Variable</th><th></th><th>Variable</th><th></th></tr></thead><tbody><tr><td>x<sub>1</sub></td><td>Enterprise value</td><td>x<sub>11</sub></td><td>Dividend yield</td></tr><tr><td>x<sub>2</sub></td><td>Cash</td><td>x<sub>12</sub></td><td>Payout ratio</td></tr><tr><td>x<sub>3</sub></td><td>Revenues</td><td>x<sub>13</sub></td><td>Standard deviation of stock price</td></tr><tr><td>x<sub>4</sub></td><td>Earnings per share</td><td>x<sub>14</sub></td><td>Frequency of negative terms</td></tr><tr><td>x<sub>5</sub></td><td>Return on equity</td><td>x<sub>15</sub></td><td>Frequency of positive terms</td></tr><tr><td>x<sub>6</sub></td><td>Price to book value</td><td>x<sub>16</sub></td><td>Frequency of uncertainty terms</td></tr><tr><td>x<sub>7</sub></td><td>Enterprise value/earnings</td><td>x<sub>17</sub></td><td>Frequency of litigious terms</td></tr><tr><td>x<sub>8</sub></td><td>Price to earnings per share</td><td>x<sub>18</sub></td><td>Frequency of strong modal terms</td></tr><tr><td>x<sub>9</sub></td><td>Market debt / total capital</td><td>x<sub>19</sub></td><td>Frequency of weak modal terms</td></tr><tr><td>x<sub>10</sub></td><td>High to low stock price</td><td>class</td><td>{IG, NG}</td></tr></tbody></table>

The given dataset was randomly divided into training and testing dataset (2:1). This procedure was repeated five times. Thus, the training dataset $O_{train}$ contained 347 companies, 217 classified as IG and 130 as NG. The testing dataset $O_{test}$ covered 173 companies, 108 as IG and 65 as NG. Both datasets were imbalanced, with the less frequent NG category (minority class). There are several approaches to handle this issue. The under-sampling of the majority class may represent a good way to reduce the sensitivity of classifiers, but in our case this approach resulted in a decrease in classification performance. This may be related to both the small size of the dataset and important decision information stored in most of the objects in the majority class. Another way is to apply the over-sampling of the minority class so that all classes are represented equally in the training dataset. We used the SMOTE procedure [10] to generate additional objects (firms) for the NG class to make the training dataset balanced. Thus, $O_{train}$ contained 433 ($O_{train}^{100}$) companies, 217 classified as IG and 216 as NG. These training datasets $O_{train}^{100}$ were considered to be the base balanced training datasets with 100 % of companies. We further examined the effect of generating additional training datasets, increasing the number by 25 % up to 300 % ($O_{train}^{125}$, $O_{train}^{150}$, ..., $O_{train}^{300}$). The testing set remained imbalanced and fixed for all training sets ($O_{train}^{100}$, $O_{train}^{125}$, $O_{train}^{150}$, ..., $O_{train}^{300}$).

### 3 Modelling Financial and Sentiment Indicators

In this section we employed commonly used artificial immune classification algorithms, AIRS1, AIRS2, AIRS2-p and CSCA. Further, the AISs were compared with
DTs, SVMs, RBFs and MLPs. We used 10-fold cross-validation on training data to find the optimum settings of the classifiers' parameters.

The measures of classification performance are represented by the averages of standard statistics applied in classification tasks [14]: true positives (TP rate), false positives (FP rate), precision (Pre) and recall (Re), F-measure (F-m), the area under the receiver operating characteristic (ROC) curve and misclassification cost (MC). F-m is the weighted harmonic mean of Pre and Re, or the Matthews correlation coefficient, which is a geometric mean of the chance-corrected variants. A ROC is a graphical plot which illustrates the performance of a binary classifier system, which represents a standard technique for summarization classifier performance over a range of tradeoffs between TP and FP error rates. In particular, FP rate is important in this study owing to its possible serious financial consequences. This is due to the significant difference between default rates of IG and NG firms. For example, Standard & Poor's reported 0.03 % for IG and 1.71 % for NG in 2011. Therefore, we designed an MC matrix, where MC = 0.03 was assigned to each false classified IG firm and MC = 1.71 to each false classified NG firm, respectively.

## 3.1 Artificial Immune Classification Algorithms

### 3.1.1 Methods

The group of AIRS algorithms represents AISs using populations [2,3]. The algorithms are based on the principle of Recognition Ball (RB) or Artificial Recognition Ball (ARB), which can be described as recognition areas or artificial recognition areas that combine feature vector (antibody) and vector class. The principle solves the issue of the completeness of AISs. Each antibody is surrounded (in the sense of antibody representation in the state space) by a small area called the RB, in which the antibody recognizes all antigens (training dataset). Further, the AIRS algorithms use the principle of a limited resource. Each ARB area competes for limited resource according to its stimulation level. The classification performance of the AIRS algorithms proposed by [16] depends on several user-defined parameters. We used the AIRS1, AIRS2 and AIRS2-p algorithms to predict the firms' credit ratings. The following parameters of the AIRSs were examined to obtain the best classification performance: affinity threshold scalar = {0.1,0.2, ...,0.9}, clonal rate = {1,2,4, ...,32}, hypermutation rate = {1,2, ...,10}, number of k nearest neighbors = {1,2, ...,10}, initial memory cell pool size = 50, number of instances to compute the affinity threshold = all, stimulation threshold = 0.9 and total resources = 150.

The CSCA [1] uses a fitness function to maximize classification accuracy (minimize misclassification accuracy). The performance of the CSCA depends on the following user-defined parameters: clonal scale factor = 1.0, initial population size = {10,20, ...,100}, number of nearest neighbors = {1,2, ...,10}, minimum fitness threshold = 1, number of partitions = 1 and total generations = {1,5, 10, ...,100}.

### 3.1.2 Results

The best classification results for the AIRSs and CSCA simulations on testing dataset are shown in Table 2.
Here, we present the average performance measures over the five datasets. The measures are represented by the averages of standard statistics applied in classification tasks [14]. The results in Table 2 show that the AIRS2 performed best while the CSCA performed significantly worse (using paired t-test on *P* < 0.01). Further, the sizes of training datasets suggest that the AIRSs require larger datasets to achieve good classification performance (see Fig. 1).

<table><caption>**Table 2.** Best results for AIRS1, AIRS2, AIRS2-p and CSCA on testing dataset.</caption><thead><tr><th></th><th>AIRS1</th><th>AIRS2</th><th>AIRS2-p</th><th>CSCA</th></tr></thead><tbody><tr><th>O<sub>train</sub></th><td>O<sub>train</sub><sup>175</sup></td><td>O<sub>train</sub><sup>225</sup></td><td>O<sub>train</sub><sup>225</sup></td><td>O<sub>train</sub><sup>150</sup></td></tr><tr><th>Accuracy [%]</th><td>83.24</td><td><strong>85.24</strong></td><td>83.27</td><td>80.92</td></tr><tr><th>MC</th><td>41.15</td><td>35.95</td><td>37.89</td><td><strong>22.76</strong></td></tr><tr><th>Class</th><td>IG</td><td>NG</td><td>IG</td><td>NG</td><td>IG</td><td>NG</td><td>IG</td><td>NG</td></tr><tr><th>TP rate</th><td>0.923</td><td>0.778</td><td>0.938</td><td>0.806</td><td>0.892</td><td>0.796</td><td>0.692</td><td>0.880</td></tr><tr><th>FP rate</th><td>0.222</td><td>0.077</td><td>0.194</td><td>0.062</td><td>0.204</td><td>0.108</td><td>0.120</td><td>0.308</td></tr><tr><th>Precision</th><td>0.714</td><td>0.944</td><td>0.744</td><td>0.956</td><td>0.725</td><td>0.925</td><td>0.776</td><td>0.826</td></tr><tr><th>Recall</th><td>0.923</td><td>0.778</td><td>0.938</td><td>0.806</td><td>0.892</td><td>0.796</td><td>0.692</td><td>0.880</td></tr><tr><th>F-m</th><td>0.805</td><td>0.853</td><td>0.830</td><td>0.874</td><td>0.800</td><td>0.856</td><td>0.732</td><td>0.852</td></tr><tr><th>ROC</th><td>0.850</td><td>0.850</td><td>0.872</td><td>0.872</td><td>0.844</td><td>0.844</td><td>0.786</td><td>0.786</td></tr></tbody></table>

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Classification accuracy of AIRS1, AIRS2, AIRS2-p and CSCA on O<sub>test</sub> depending on O<sub>train</sub> size (Source: own)</figcaption></figure>

## 3.2 Machine Learning and Neural Networks

### 3.2.1 Methods

A DT is a tree representation assigning a class to an object based on its attributes (variables), which can be continuous or discrete. An attribute with the best value of a splitting criterion is assigned to each root and intermediate node. Classes are assigned to leaf nodes in a DT. The DT with pruning was employed in this study. The classifi-
cation performance of the DT depends on the following parameters: the confidence factor used for pruning (= 0.25 in this study) and the minimum number of instances per leaf = 2.

SVMs represent an essential kernel-based method with many modifications proposed recently. SVMs use kernel functions to separate the hyperplane between two classes by maximizing the margin between the closest data points. This is done in a higher-dimensional space where the data become linearly separable. We used the SVMs trained by the sequential minimal optimization (SMO) algorithm [13]. The classification performance of the SVM was tested for the following user-defined parameters: kernel functions = {polynomial, RBF}, $\gamma$ = 0.01, the level of polynomial function = 2, complexity parameter $C$ = {1,2,4, ..., 256}, round-off error $\epsilon$ = 1.0E-12 and tolerance parameter = 0.001. The RBF was trained with the Broyden-Fletcher-Goldfarb-Shanno method. The initial centres for the Gaussian RBFs were found using a k-means algorithm. The initial sigma values were set to the maximum distance between any centre and its nearest neighbor in the set of centres. The classification performance of the RBF depends on the following user-defined parameters: maximum number iteration = not limited, minimum standard deviation for the clusters = {0.1,0.2,0.3}, the number of clusters for k-means = {1,2,4, ..., 64} and ridge factor = 1.0E-8.

The MLP was trained using the backpropagation algorithm with momentum. The following parameters of the MLP were examined to achieve the best classification performance: the number of neurons in the hidden layer = {5,10,15, ..., 30}, learning rate = {0.01,0.05,0.1}, momentum = 0.2 and the number of epochs = {100,500,1000}.

### 3.2.2 Results

The best classification results for the DT, SVM, RBF and MLP simulations on testing dataset are shown in Table 3. Here, the best classification performance was achieved by MLP (Accuracy) and DT (MC). Compared with the AISs, the MLP provided significantly better results in terms of both classification accuracy and ROCs (again, tested using paired t-test at $P < 0.01$). Furthermore, less $O_{train}$ were required to be generated in order to learn the MLP compared with the AIRS2 (see Fig. 2). On the other hand, DT provided the lowest MC of all classifiers.

<table><caption>**Table 3.** Best results for DT, SVM, RBF and MLP algorithms on testing dataset.</caption><thead><tr><th></th><th colspan="2">DT</th><th colspan="2">SVM</th><th colspan="2">RBF</th><th colspan="2">MLP</th></tr><tr><th>O<sub>train</sub></th><th>O<sub>train</sub><sup>125</sup></th><th>O<sub>train</sub><sup>150</sup></th><th>O<sub>train</sub><sup>150</sup></th><th>O<sub>train</sub><sup>150</sup></th><th>O<sub>train</sub><sup>125</sup></th><th>O<sub>train</sub><sup>125</sup></th><th>O<sub>train</sub><sup>125</sup></th><th>O<sub>train</sub><sup>125</sup></th></tr></thead><tbody><tr><td>Accuracy [%]</td><td>86.13</td><td>87.28</td><td>84.39</td><td>84.39</td><td>84.39</td><td>88.44</td><td>88.44</td><td>88.44</td></tr><tr><td>MC</td><td><strong>17.59</strong></td><td>30.96</td><td>36.01</td><td>36.01</td><td>24.19</td><td>24.19</td><td>24.19</td><td>24.19</td></tr><tr><th>Class</th><th>IG</th><th>NG</th><th>IG</th><th>NG</th><th>IG</th><th>NG</th><th>IG</th><th>NG</th></tr><tr><td>TP rate</td><td>0.785</td><td>0.907</td><td>0.938</td><td>0.833</td><td>0.908</td><td>0.806</td><td>0.908</td><td>0.870</td></tr><tr><td>FP rate</td><td>0.093</td><td>0.215</td><td>0.167</td><td>0.062</td><td>0.194</td><td>0.092</td><td>0.130</td><td>0.092</td></tr><tr><td>Precision</td><td>0.836</td><td>0.875</td><td>0.772</td><td>0.957</td><td>0.738</td><td>0.935</td><td>0.808</td><td>0.940</td></tr><tr><td>Recall</td><td>0.785</td><td>0.907</td><td>0.938</td><td>0.833</td><td>0.908</td><td>0.806</td><td>0.908</td><td>0.870</td></tr><tr><td>F-m</td><td>0.810</td><td>0.891</td><td>0.847</td><td>0.891</td><td>0.814</td><td>0.866</td><td>0.855</td><td>0.904</td></tr><tr><td>ROC</td><td>0.871</td><td>0.871</td><td>0.886</td><td>0.886</td><td>0.884</td><td>0.881</td><td>0.947</td><td>0.947</td></tr></tbody></table>
3.3 Comparison Across Classifiers

In Fig. 3 we compare the ROC achieved for the balanced and imbalanced datasets. Except for the AIRS2-p and RBF, the performance of the classifiers improved significantly (at *P* < 0.05) when the *O*<sub>train</sub> were balanced using the SMOTE (*O*<sub>train</sub><sup>100</sup>). Depending upon the amount of over-sampling required, neighbors from *k* nearest neighbors are randomly chosen. In our experiments we used *k* = 5 nearest neighbors. The results of the classification of *O*<sub>test</sub> for the original imbalanced and balanced *O*<sub>train</sub> are shown in Table 4.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Classification accuracy of DT, SVM, RBF and MLP on O<sub>test</sub> depending on O<sub>train</sub> size

(Source: own)</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** ROC on testing dataset for original imbalanced and balanced O<sub>train</sub> (Source: own)</figcaption></figure>
We further examined the effect of the ensembles of classifiers. We employed the AdaBoost method [4] that combines many ‘weak’ classifiers to obtain an accurate learning algorithm. More precisely, AdaBoost is a meta-algorithm that can be used in conjunction with many other learning algorithms to improve their performance. This meta-algorithm is adaptive in the sense that subsequent classifiers built are tweaked in favor of those instances misclassified by previous classifiers. The number of iterations to be performed was set to 10. Again, we tested the previously described classification algorithms as base learners.

The results are depicted in Fig. 4. Here, the balanced training dataset was used from prior experiments. The ROC was significantly higher (at *P* < 0.05) for all the classifiers except for the MLP. AdaBoost provided noteworthy improvements for the DTs, AISs and SVMs especially. The results of the classification of the *O*<sub>test</sub> for the original imbalanced and balanced *O*<sub>train</sub> are shown in Table 5. DT with balanced data provided significantly lower MC (at *P* < 0.05) for both individual (Table 4) and ensemble classifiers (Table 5). For details, see confusion matrix in Table 6.

<table><caption>**Table 4.** Classification performance on *O*<sub>test</sub> for the original imbalanced (i) and balanced (b) *O*<sub>train</sub> (weighted average for IG and NG).</caption><thead><tr><th></th><th></th><th>AIRS1</th><th>AIRS2</th><th>AIRS2-p</th><th>CSCA</th><th>DT</th><th>SVM</th><th>RBF</th><th>MLP</th></tr></thead><tbody><tr><td rowspan="2">TP</td><td>i</td><td>0.821</td><td>0.814</td><td>0.834</td><td>0.814</td><td>0.861</td><td>0.851</td><td>0.799</td><td>0.845</td></tr><tr><td>b</td><td>0.853</td><td>0.801</td><td>0.798</td><td>0.801</td><td>0.875</td><td>0.853</td><td>0.786</td><td>0.834</td></tr><tr><td rowspan="2">FP</td><td>i</td><td>0.143</td><td>0.201</td><td>0.174</td><td>0.241</td><td>0.138</td><td>0.163</td><td>0.202</td><td>0.161</td></tr><tr><td>b</td><td>0.148</td><td>0.160</td><td>0.155</td><td>0.160</td><td>0.133</td><td>0.148</td><td>0.196</td><td>0.148</td></tr><tr><td rowspan="2">F-m</td><td>i</td><td>0.822</td><td>0.814</td><td>0.834</td><td>0.810</td><td>0.862</td><td>0.851</td><td>0.800</td><td>0.846</td></tr><tr><td>b</td><td>0.854</td><td>0.804</td><td>0.801</td><td>0.804</td><td>0.876</td><td>0.854</td><td>0.788</td><td>0.835</td></tr><tr><td rowspan="2">MC</td><td>i</td><td>42.42</td><td>45.99</td><td>50.80</td><td>21.56</td><td>21.29</td><td>27.99</td><td>44.80</td><td>36.14</td></tr><tr><td>b</td><td>26.96</td><td>30.54</td><td>29.10</td><td>19.80</td><td><strong>17.89</strong></td><td>24.00</td><td>37.33</td><td>27.33</td></tr></tbody></table>

<table><caption>**Table 5.** Classification performance of the *O*<sub>test</sub> for the original imbalanced (i) and balanced (b) *O*<sub>train</sub> (weighted average for IG and NG) with AdaBoost.</caption><thead><tr><th></th><th></th><th>AIRS1</th><th>AIRS2</th><th>AIRS2-p</th><th>CSCA</th><th>DT</th><th>SVM</th><th>RBF</th><th>MLP</th></tr></thead><tbody><tr><td rowspan="2">TP</td><td>i</td><td>0.839</td><td>0.845</td><td>0.850</td><td>0.819</td><td>0.848</td><td>0.846</td><td>0.828</td><td>0.852</td></tr><tr><td>b</td><td>0.845</td><td>0.836</td><td>0.858</td><td>0.809</td><td>0.872</td><td>0.831</td><td>0.798</td><td>0.836</td></tr><tr><td rowspan="2">FP</td><td>i</td><td>0.171</td><td>0.166</td><td>0.144</td><td>0.233</td><td>0.155</td><td>0.141</td><td>0.180</td><td>0.150</td></tr><tr><td>b</td><td>0.141</td><td>0.147</td><td>0.138</td><td>0.191</td><td>0.134</td><td>0.140</td><td>0.179</td><td>0.138</td></tr><tr><td rowspan="2">F-m</td><td>i</td><td>0.840</td><td>0.857</td><td>0.852</td><td>0.816</td><td>0.849</td><td>0.848</td><td>0.829</td><td>0.853</td></tr><tr><td>b</td><td>0.846</td><td>0.837</td><td>0.860</td><td>0.803</td><td>0.872</td><td>0.833</td><td>0.800</td><td>0.838</td></tr><tr><td rowspan="2">MC</td><td>i</td><td>32.70</td><td>35.14</td><td>26.97</td><td>38.99</td><td>22.51</td><td>39.52</td><td>44.09</td><td>37.83</td></tr><tr><td>b</td><td>27.03</td><td>25.29</td><td>28.97</td><td>19.04</td><td><strong>17.28</strong></td><td>32.04</td><td>35.26</td><td>26.93</td></tr></tbody></table>
<figure><img src="image_5.png" /><figcaption>Fig. 4. ROC for base classifiers and AdaBoost (Source: own)</figcaption></figure>

<table><caption>**Table 6.** Confusion matrix of the O<sub>test</sub> for the balanced O<sub>train</sub> trained using DT with AdaBoost.</caption><thead><tr><th></th><th>IG</th><th>NG</th></tr></thead><tbody><tr><th>IG</th><td>56.8±6.3</td><td>9.2±5.2</td></tr><tr><th>NG</th><td>13.0±6.1</td><td>95.0±6.1</td></tr></tbody></table>

# 4 Conclusion

The aim of this paper was to examine the performance of AISs and ML on a complex classification task where financial indicators are combined with sentiment analysis. We attempted to address an important issue of imbalanced dataset. On the one hand, the over-sampling of the minority class showed promising classification improvement, yet on the other hand, the under-sampling of the majority class resulted in weaker classification performance. We performed many experiments to find the best setting of the learning parameters of individual classifiers to achieve the best classification performance. The results also confirmed that an ensemble of weaker base classifiers may perform better than the individual classifiers, especially in terms of ROC measure. More importantly, the decrease in MC may lead to the substantial financial savings of investors. For example, the ensemble of DTs trained on balanced data reduced the MC from 21.29 to 17.28 (18.8% savings). Thus, this study represents a good basis for further experiments in the field of financial distress forecasting, where the problem of imbalanced datasets is usually to be addressed. Additionally, we encourage future research in multi-class datasets.
The experiments in this study were carried out in Statistica 10 (linguistic pre-processing) and Weka 3.7.5 (Artificial Immune Classification Algorithms and ML) in MS Windows 7 operating system.

**Acknowledgments.** This work was supported by the scientific research project of the Czech Sciences Foundation Grant No: 13-10331S.

## References

1. Brownlee, J.: Clonal Selection Theory Algorithm and CLONALG: The Clonal Selection Classification Algorithm (CSCA). Victoria Australia, Centre for Intelligent Systems and Complex Processes, Swinburne University of Technology, Technical Report ID: 2-01 (2005)

2. Dasgupta, D.: Artificial Immune Systems and their Applications. Springer-Verlag, Berlin (1999)

3. De Castro, L.N., Timmis, J.: Artificial Immune Systems: A New Computational Intelligence Approach. Springer-Verlag, Berlin (2002)

4. Freund, Y., Schapire, R.E.: Experiments with a New Boosting Algorithm. In: 13th International Conference on Machine Learning, San Francisco, 148–156 (1996)

5. García-Pedrajas, N., Fyfe, C.: Construction of Classifier Ensembles by means of Artificial Immune Systems. Journal of Heuristics 14(3), 285–310 (2008)

6. Hajek, P.: Municipal Credit Rating Modelling by Neural Networks. Decision Support Systems 51(1), 108–118 (2011)

7. Hajek, P., Olej, V.: Credit Rating Modelling by Kernel-Based Approaches with Supervised and Semi-Supervised Learning. Neural Computing and Applications 20(6), 761–773 (2011)

8. Hajek, P., Olej, V.: Evaluating Sentiment in Annual Reports for Financial Distress Prediction using Neural Networks and Support Vector Machines. In: 14th EANN Conference on Engineering Applications of Neural Networks, Iliadis, L., Papadopoulos, H., Jayne, Ch. (eds.) Springer-Verlag Berlin Heidelberg, Part II, 1–10 (2013)

9. Haykin, S.: Neural Networks: A Comprehensive Foundation. 2nd edition, Prentice-Hall Inc., New Jersey (1999)

10. Chawla, N.V., Bowyer, K.W. et al.: SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research 16, 321–357 (2002)

11. Loughran, T., McDonald, B.: When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10-Ks. The Journal of Finance 66(1), 35–65 (2011)

12. Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann (1993)

13. Platt, J.C.: Fast Training of Support Vector Machines using Sequential Minimal Optimization. In: Schoelkopf, B., Burges, C., Smola, A. (eds.) Advances in Kernel Methods - Support Vector Learning, MIT Press (1998)

14. Powers, D.M.W.: Evaluation: from Precision, Recall and F-measure to ROC, Informedness, Markedness and Correlation. Journal of Machine Learning Technologies 1(2), 37–63 (2011)

15. Ravi Kumar, P., Ravi, V.: Bankruptcy Prediction in Banks and Firms via Statistical and Intelligent Techniques - A Review. European Journal of Operational Research 180(1), 1–28 (2007)

16. Watkins, B.A., Timmis, J., Boggess, L.: Artificial Immune Recognition System (AIRS): An Immune Inspired Supervised Learning Algorithm. Genetic Programming and Evolvable Machines 5, 291–317 (2004)