# Identifying Features with Concept Drift in Multidimensional Data Using Statistical Tests

Piotr Sobolewski, Michal Woźniak

► To cite this version:

Piotr Sobolewski, Michal Woźniak. Identifying Features with Concept Drift in Multidimensional Data Using Statistical Tests. 10th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Sep 2014, Rhodes, Greece. pp.405-413, 10.1007/978-3-662-44654-6_40. hal-01391341

HAL Id: hal-01391341

https://inria.hal.science/hal-01391341

Submitted on 3 Nov 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Identifying features with concept drift in multidimensional data using statistical tests

Piotr Sobolewski and Michał Woźniak

Wrocław University of Technology
ul. Wybrzeże Wyspiańskiego 27
50-370 Wrocław
{piotr.sobolewski,michal.wozniak}@pwr.edu.pl

**Abstract.** Concept drift is a common problem in the data streams, which makes the classifiers no longer valid. In the multidimensional data, this problem becomes difficult to tackle. This paper examines the possibilities of identifying the specific features, in which concept drift occurs. This allows to limit the scope of the necessary update in the classification system. As a tool, we select a popular Kolmogorov-Smirnov test statistic.

**Keywords:** Concept drift, detection, statistical test

## 1 Introduction

Due to the evolution of the internet and the expansion of the decision making technology, the systems designed for classifying the data streams [2] recently became a popular area of research. In the field of machine learning, data streams are defined as sources of continuous data generation, examples of which can be found in real life e.g., shopping trends, stock market, weather control, surveillance systems or health care. Classification task in these areas is often hindered by various factors which cause undesirable changes in the data classification rules. Such phenomenon is called concept drift [6] and it is a major problem in the classification systems.

There are various methods described in the machine learning literature for defending against concept drift, mostly deploying one of the two popular strategies [4]:

- Adapting a learner at the regular intervals without considering whether the changes have really occurred or not,

- First detecting the concept changes and then adapting a learner to them.

The idea presented in this article has a potential of improving the classifier adaptation process as well as enhancing the efficiency of the concept drift detection algorithms.
## 2 Problem description

We assume that in the multidimensional data, concept drift may influence only some specific features, leaving all other features in the same conceptual distribution model. Identifying these features may improve the adaptation of the classification systems, as well as provide useful information for the sophisticated concept drift detection algorithms, such as LDCnet [9].

Our previous experiments [8][10] have shown, that the popular test statistics, such as the Kolmogorov-Smirnov test [7], Wilcoxon rank sum [11] or Wald-Wolfowitz test [12] are capable of detecting concept drift with a similar efficiency as advanced methods, designed specifically for this purpose, such as the CNF test [3]. The most efficient in our experiments was the Kolmogorov-Smirnov test statistic, therefore we have selected it for further analysis in this article.

Kolmogorov-Smirnov test is a non-parametric statistic, as it makes no assumption about the distribution of data and therefore can be deployed on any data.

For the two-sample test, a Kolmogorov-Smirnov statistic is computed as

$$D_{n,m} = \sup_x |F_{1,n}(x) - F_{2,m}(x)| \quad (1)$$

where $F_{1,n}$ and $F_{2,m}$ are the empirical distribution functions of samples computed as:

$$F_n(t) = \frac{1}{n} \sum_{i=1}^{n} 1\{x_i \le t\}, \quad (2)$$

where $(x_1, ..., x_n)$ are independent and identically distributed (i.i.d.) random variables laying in the real numbers domain with a common cumulative distribution function. The statistic is used to perform a KS-test to reject the null hypothesis at level $\alpha$ by computing:

$$\sqrt{\frac{nm}{n_m}} D_{n,m} > K_{\alpha}, \quad (3)$$

where $K_\alpha$ calculated from:

$$Pr(K \le K_{\alpha}) = 1 - \alpha, \quad (4)$$

and K is a Kolmogorov distribution computed as:

$$K = \sup_{t \in [0,1]} |B(t)|, \quad (5)$$

$B(t)$ being the Brownian bridge [5].

In short, the Kolmogorov-Smirnov test compares the distributions of two samples by measuring a distance between the empirical distribution functions, taking into account both their location and shape.

In this paper we evaluate the possibilities of applying the Kolmogorov-Smirnov test statistic as a tool for identifying the features, which are influenced by concept drift. For this purpose, the tool needs to accurately classify the true positives (sensitivity) as well as the true negatives (specificity).
## 3 Data

Due to limited availability of the real data with concept drift, the data used in experiments is taken from the UCI Repository of datasets [1] and concept drift is simulated by swapping the features with each other.

In mathematical notation, if a reference dataset *DS* is characterized by *n* features *f* then the concept drift is applied by swapping any two features *i* and *j* with each other. Data with swapped features forms a new dataset *DS*<sub>*i*,*j*</sub> and the role of the algorithm is to identify which features are influenced by concept drift (i.e. find the *i* and *j*).

Example swap of features 1 and 2:

$$ DS = [f_1, f_2, ..., f_n] \\ DS_{1,2} = [f_2, f_1, ..., f_n] \tag{6} $$

The swaps are made for each possible pair of features, resulting in (n/2) combinations, where *n* is the dimensionality of the dataset. Each of the dataset is described in general by the number of samples and number of features in Tab. 1.

<table><caption>**Table 1.** Datasets</caption><thead><tr><th>Dataset</th><th># of features</th><th># of samples</th></tr></thead><tbody><tr><td>breast</td><td>9</td><td>683</td></tr><tr><td>credit-australian</td><td>14</td><td>690</td></tr><tr><td>haberman</td><td>3</td><td>306</td></tr><tr><td>heart-c</td><td>13</td><td>297</td></tr><tr><td>heart-statlog</td><td>13</td><td>270</td></tr><tr><td>ionosphere</td><td>34</td><td>351</td></tr><tr><td>kr-vs-kp</td><td>36</td><td>3196</td></tr><tr><td>letter-recognition</td><td>16</td><td>20000</td></tr><tr><td>mfeat-mor</td><td>6</td><td>2000</td></tr><tr><td>nursery</td><td>8</td><td>12960</td></tr><tr><td>optdigits</td><td>64</td><td>3823</td></tr><tr><td>page-blocks</td><td>10</td><td>5473</td></tr><tr><td>pendigits</td><td>16</td><td>7494</td></tr><tr><td>pima-indians-diabetes</td><td>8</td><td>768</td></tr><tr><td>segmentation</td><td>19</td><td>210</td></tr><tr><td>tic-tac-toe</td><td>9</td><td>958</td></tr><tr><td>vehicle</td><td>18</td><td>846</td></tr><tr><td>vote</td><td>16</td><td>232</td></tr><tr><td>waveform</td><td>21</td><td>5000</td></tr><tr><td>yeast</td><td>8</td><td>1484</td></tr></tbody></table>

This method of simulating concept drift is relatively common in the machine learning literature [13].
## 4 Experiments

In the experiments, we use the original dataset *D* as the reference data and the
drifted datasets *D*<sub>*i*,*j*</sub> (*i* and *j* indicate the features which are swapped), with
the samples randomly drawn from the datasets *D*<sub>*i*,*j*</sub> and grouped into data
windows *DW* of various sizes *s*.

The Kolomogorov-Smirnov statistic is evaluated on every feature *f* in the
data window to reject the null hypothesis that the values arise from the same
population as the values of features in the reference dataset *D* with confidence
level of 5%. It means, that if the test returns the p-value lower than 0.05, then
the analyzed feature is considered to be influenced by concept drift and the
detection signal is noted and added to the scores.

In order to evaluate the specificity, i.e. the ability to identify the true nega-
tives, the data windows which do not include any feature swap are evaluated and
if in this test the statistic returns the p-value lower than 0.05, then the algorithm
makes a mistake, as it results in a false positive concept drift detection.

A short description of the experimental process is described in the pseudocode
in Fig. 1.

<table><caption>**Algorithm 1 Pseudo-code of a single loop in experimental series**</caption>
  <thead>
    <tr>
      <th>Notations:</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>DS - original dataset with n features,</td>
    </tr>
    <tr>
      <td>DS<sub>i,j</sub> - dataset with swapped features i and j,</td>
    </tr>
    <tr>
      <td>DW<sup>s</sup><sub>i,j</sub> - data window of size s with features i and j swapped,</td>
    </tr>
    <tr>
      <td>Single loop of experiment series:</td>
    </tr>
    <tr>
      <td>DW<sup>s</sup> = draw s random data samples from DS</td>
    </tr>
    <tr>
      <td>For i = 1 to f</td>
    </tr>
    <tr>
      <td>For j = 1 to f</td>
    </tr>
    <tr>
      <td>DW<sup>s</sup><sub>i,j</sub> = swap features i and j in DW<sup>s</sup></td>
    </tr>
    <tr>
      <td>For k = 1 to f</td>
    </tr>
    <tr>
      <td>Evaluate KS statistic on feature k of DW<sup>s</sup><sub>i,j</sub> and feature k of D</td>
    </tr>
    <tr>
      <td>IF p-value &lt; 0.05</td>
    </tr>
    <tr>
      <td>Note concept drift for feature k</td>
    </tr>
    <tr>
      <td>END IF</td>
    </tr>
    <tr>
      <td>END FOR</td>
    </tr>
    <tr>
      <td>END FOR</td>
    </tr>
    <tr>
      <td>END FOR</td>
    </tr>
  </tbody>
</table>

In the presented way, the sensitivity and specificity of the Kolmogorov-
Smirnov test statistic are evaluated for every possible feature swap and for var-
ious sizes of the data windows.

5 Results

All presented values are averaged from the series of 1000 trials.
Tables 2 and 3 show the percentage of correctly detected concept drifts in certain features for the *breast* dataset (size of data window 20 and 50, respectively), where columns are the base features and rows are the features which swap them. The diagonals are the percentage of detected false positives, the lower the value the higher the specificity of the algorithm.

Tables 4 and 5 show how the window size influences the performance of
algorithm for the *breast* and *credit - australian* datasets. The tables store the
results obtained by swapping the first feature with other features. The results
obtained for the *breast* dataset are also presented on the Diagram 1 for a more
clear view of the efficiency trend in the domain of window size.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Concept drift detection ratio for various window sizes.</figcaption></figure>

Finally, Tab. 6 shows the overall performance of the Kolmogorov-Smirnov
statistic in identifying the features affected by concept drift, divided into the
sensitivity and specificity scores for each of the datasets and for various window
sizes. Specificity is presented only for window size 20, as the results were not
significantly different for other window sizes.
<table><caption>**Table 2.** Concept drift detection ratio in breast dataset, window size = 20</caption><thead><tr><th rowspan="2">Swap feature</th><th colspan="9">Base feature</th></tr><tr><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th></tr></thead><tbody><tr><td>1</td><td>0.02</td><td>0.7</td><td>0.67</td><td>0.86</td><td>0.82</td><td>0.89</td><td>0.27</td><td>0.91</td><td>1</td></tr><tr><td>2</td><td>0.65</td><td>0.02</td><td>0</td><td>0.02</td><td>0.93</td><td>0.01</td><td>0.5</td><td>0</td><td>0.47</td></tr><tr><td>3</td><td>0.59</td><td>0.01</td><td>0.01</td><td>0.03</td><td>0.93</td><td>0.03</td><td>0.43</td><td>0.07</td><td>0.63</td></tr><tr><td>4</td><td>0.84</td><td>0.02</td><td>0.02</td><td>0</td><td>0.95</td><td>0.01</td><td>0.71</td><td>0.03</td><td>0.34</td></tr><tr><td>5</td><td>0.68</td><td>1</td><td>0.99</td><td>0.99</td><td>0</td><td>1</td><td>0.03</td><td>1</td><td>1</td></tr><tr><td>6</td><td>0.73</td><td>0.02</td><td>0.08</td><td>0.02</td><td>1</td><td>0</td><td>0.8</td><td>0.04</td><td>0.33</td></tr><tr><td>7</td><td>0.39</td><td>0.56</td><td>0.62</td><td>0.63</td><td>0.17</td><td>0.78</td><td>0.03</td><td>0.85</td><td>1</td></tr><tr><td>8</td><td>0.89</td><td>0.03</td><td>0.03</td><td>0.01</td><td>1</td><td>0.02</td><td>0.83</td><td>0</td><td>0.16</td></tr><tr><td>9</td><td>1</td><td>0.6</td><td>0.58</td><td>0.27</td><td>1</td><td>0.42</td><td>0.99</td><td>0.13</td><td>0</td></tr></tbody></table>

<table><caption>**Table 3.** Concept drift detection ratio in breast dataset, window size = 20</caption><thead><tr><th rowspan="2">Swap feature</th><th colspan="9">Base feature</th></tr><tr><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th></tr></thead><tbody><tr><td>1</td><td>0.02</td><td>1</td><td>0.98</td><td>1</td><td>1</td><td>1</td><td>0.79</td><td>1</td><td>1</td></tr><tr><td>2</td><td>1</td><td>0</td><td>0</td><td>0.02</td><td>1</td><td>0.07</td><td>1</td><td>0.03</td><td>0.92</td></tr><tr><td>3</td><td>0.99</td><td>0.01</td><td>0</td><td>0.1</td><td>1</td><td>0.08</td><td>0.95</td><td>0.12</td><td>0.99</td></tr><tr><td>4</td><td>1</td><td>0.03</td><td>0.05</td><td>0.01</td><td>1</td><td>0.05</td><td>0.99</td><td>0.03</td><td>0.83</td></tr><tr><td>5</td><td>1</td><td>1</td><td>1</td><td>1</td><td>0</td><td>1</td><td>0.48</td><td>1</td><td>1</td></tr><tr><td>6</td><td>1</td><td>0.1</td><td>0.11</td><td>0.14</td><td>1</td><td>0</td><td>1</td><td>0.11</td><td>0.89</td></tr><tr><td>7</td><td>0.88</td><td>0.98</td><td>0.96</td><td>1</td><td>0.57</td><td>1</td><td>0</td><td>1</td><td>1</td></tr><tr><td>8</td><td>1</td><td>0.07</td><td>0.09</td><td>0.01</td><td>1</td><td>0.03</td><td>1</td><td>0.01</td><td>0.59</td></tr><tr><td>9</td><td>1</td><td>0.99</td><td>1</td><td>0.89</td><td>1</td><td>0.97</td><td>1</td><td>0.58</td><td>0</td></tr></tbody></table>
<table><caption>**Table 4.** Window size influence, *breast* dataset.</caption><thead><tr><th rowspan="2">breast</th><th colspan="9">Feature 1 swapped with feature..</th></tr><tr><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th></tr></thead><tbody><tr><th>win size</th><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><th>5</th><td>0.03</td><td>0.30</td><td>0.24</td><td>0.37</td><td>0.21</td><td>0.34</td><td>0.18</td><td>0.42</td><td>0.86</td></tr><tr><th>10</th><td>0.02</td><td>0.41</td><td>0.37</td><td>0.55</td><td>0.43</td><td>0.47</td><td>0.20</td><td>0.60</td><td>0.97</td></tr><tr><th>20</th><td>0.01</td><td>0.75</td><td>0.62</td><td>0.85</td><td>0.72</td><td>0.79</td><td>0.42</td><td>0.89</td><td>1</td></tr><tr><th>50</th><td>0.01</td><td>0.99</td><td>0.99</td><td>1.00</td><td>0.99</td><td>1.00</td><td>0.85</td><td>1.00</td><td>1</td></tr><tr><th>100</th><td>0.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>0.99</td><td>1.00</td><td>1</td></tr></tbody></table>

<table><caption>**Table 5.** Window size influence, *credit - australian* dataset.</caption><thead><tr><th rowspan="2">credit-aus</th><th colspan="14">Feature 1 swapped with feature..</th></tr><tr><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th><th>10</th><th>11</th><th>12</th><th>13</th><th>14</th></tr></thead><tbody><tr><th>win size</th><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><th>5</th><td>0.00</td><td>1.00</td><td>0.89</td><td>0.91</td><td>1.00</td><td>0.99</td><td>0.46</td><td>0.02</td><td>0.06</td><td>0.26</td><td>0.06</td><td>1.00</td><td>0.96</td><td>0.63</td></tr><tr><th>10</th><td>0.00</td><td>1.00</td><td>0.98</td><td>0.98</td><td>1.00</td><td>1.00</td><td>0.62</td><td>0.04</td><td>0.13</td><td>0.32</td><td>0.09</td><td>1.00</td><td>1.00</td><td>0.78</td></tr><tr><th>20</th><td>0.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>0.09</td><td>0.31</td><td>0.90</td><td>0.24</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><th>50</th><td>0.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>0.31</td><td>0.83</td><td>1.00</td><td>0.70</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><th>100</th><td>0.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.00</td><td>0.63</td><td>0.99</td><td>1.00</td><td>0.97</td><td>1.00</td><td>1.00</td><td>1.00</td></tr></tbody></table>
<table><caption>**Table 6.** Overall sensitivity and specificity scores.</caption><thead><tr><th rowspan="3">Dataset</th><th colspan="12">Sensitivity</th><th colspan="2">Specificity</th></tr><tr><th colspan="2">20</th><th colspan="2">50</th><th colspan="2">100</th><th colspan="2">200</th><th colspan="2">20</th></tr><tr><th>avg</th><th>var</th><th>avg</th><th>var</th><th>avg</th><th>var</th><th>avg</th><th>var</th><th>avg</th><th>var</th></tr></thead><tbody><tr><td>breast</td><td>0.53</td><td>0.15</td><td>0.70</td><td>0.17</td><td>0.77</td><td>0.14</td><td>0.82</td><td>0.11</td><td>0.99</td><td>0.00</td></tr><tr><td>credit-australian</td><td>0.92</td><td>0.06</td><td>0.95</td><td>0.04</td><td>0.96</td><td>0.03</td><td>0.97</td><td>0.02</td><td>0.99</td><td>0.00</td></tr><tr><td>haberman</td><td>0.92</td><td>0.06</td><td>0.95</td><td>0.04</td><td>0.96</td><td>0.03</td><td>0.97</td><td>0.02</td><td>0.99</td><td>0.00</td></tr><tr><td>heart-c</td><td>0.94</td><td>0.04</td><td>0.98</td><td>0.01</td><td>1.00</td><td>0.00</td><td>1.00</td><td>0.00</td><td>0.99</td><td>0.00</td></tr><tr><td>heart-statlog</td><td>0.95</td><td>0.03</td><td>0.98</td><td>0.01</td><td>1.00</td><td>0.00</td><td>1.00</td><td>0.00</td><td>0.99</td><td>0.00</td></tr><tr><td>ionosphere</td><td>0.60</td><td>0.16</td><td>0.70</td><td>0.15</td><td>0.76</td><td>0.14</td><td>0.81</td><td>0.13</td><td>0.97</td><td>0.00</td></tr><tr><td>kr-vs-kp</td><td>0.41</td><td>0.18</td><td>0.56</td><td>0.20</td><td>0.65</td><td>0.19</td><td>0.72</td><td>0.18</td><td>1.00</td><td>0.00</td></tr><tr><td>letter-recognition</td><td>0.48</td><td>0.18</td><td>0.62</td><td>0.19</td><td>0.72</td><td>0.17</td><td>0.78</td><td>0.15</td><td>0.99</td><td>0.00</td></tr><tr><td>mfeat-mor</td><td>0.48</td><td>0.19</td><td>0.62</td><td>0.19</td><td>0.72</td><td>0.17</td><td>0.78</td><td>0.15</td><td>0.99</td><td>0.00</td></tr><tr><td>nursery</td><td>0.47</td><td>0.18</td><td>0.62</td><td>0.19</td><td>0.71</td><td>0.18</td><td>0.77</td><td>0.16</td><td>0.99</td><td>0.00</td></tr><tr><td>optdigits</td><td>0.69</td><td>0.16</td><td>0.78</td><td>0.13</td><td>0.84</td><td>0.11</td><td>0.89</td><td>0.08</td><td>0.98</td><td>0.00</td></tr><tr><td>page-blocks</td><td>0.69</td><td>0.16</td><td>0.79</td><td>0.13</td><td>0.84</td><td>0.10</td><td>0.89</td><td>0.08</td><td>0.98</td><td>0.00</td></tr><tr><td>pendigits</td><td>0.68</td><td>0.16</td><td>0.79</td><td>0.13</td><td>0.84</td><td>0.10</td><td>0.89</td><td>0.08</td><td>0.97</td><td>0.00</td></tr><tr><td>pima-indians-diabetes</td><td>0.69</td><td>0.16</td><td>0.79</td><td>0.13</td><td>0.84</td><td>0.10</td><td>0.89</td><td>0.08</td><td>0.97</td><td>0.00</td></tr><tr><td>segmentation</td><td>0.70</td><td>0.16</td><td>0.79</td><td>0.13</td><td>0.84</td><td>0.10</td><td>0.89</td><td>0.08</td><td>0.98</td><td>0.00</td></tr><tr><td>tic-tac-toe</td><td>0.68</td><td>0.17</td><td>0.78</td><td>0.14</td><td>0.83</td><td>0.11</td><td>0.87</td><td>0.09</td><td>0.98</td><td>0.00</td></tr><tr><td>vehicle</td><td>0.71</td><td>0.16</td><td>0.80</td><td>0.13</td><td>0.85</td><td>0.10</td><td>0.89</td><td>0.08</td><td>0.98</td><td>0.00</td></tr><tr><td>vote</td><td>0.66</td><td>0.17</td><td>0.75</td><td>0.15</td><td>0.81</td><td>0.12</td><td>0.86</td><td>0.10</td><td>0.98</td><td>0.00</td></tr><tr><td>waveform</td><td>0.68</td><td>0.16</td><td>0.78</td><td>0.13</td><td>0.83</td><td>0.11</td><td>0.88</td><td>0.08</td><td>0.97</td><td>0.00</td></tr><tr><td>yeast</td><td>0.68</td><td>0.16</td><td>0.78</td><td>0.13</td><td>0.83</td><td>0.11</td><td>0.88</td><td>0.08</td><td>0.97</td><td>0.00</td></tr></tbody></table>

# 6 Discussion

In this paper we have proposed an unsupervised tool for enhancing the methods coping with concept drift. We have evaluated the efficiency of the Kolmogorov-Smirnov test statistic in detecting the features affected by concept drift in the multidimensional data.

The most apparent conclusion is that the performance of algorithm depends on the data window size. Fig. 1 clearly shows this relation.

Regardless of the window size, algorithm achieves a very high specificity score, proving that the tool performs very well with true negatives, i.e. when
there is no drift. It means, that the tool can be used for detecting features with concept drift without the need to worry about the false positive detections.

On the other hand, sensitivity i.e. the true positive detection rate, leaves a field for improvement. With increasing window size, sensitivity of the tool also increases, what suggests that the tool is more feasible for problems, which do not require a very limited window size.

Overall, the performance of the proposed tool is on a decent level, as e.g. in the optdigits dataset scenario, which has $2^{64}$ possible feature swap combinations, algorithm correctly identifies on average 88% of them with only 8% variance. Pairing it with the fact that the method does not require any supervision, the Kolmogorov-Smirnov test statistic can be considered an efficient tool for detecting the features with concept drift in multidimensional data. This functionality may be used for supporting the adaptation of classifiers as well as improving algorithms designed for detecting concept drift, such as LDCnet [9].

Further research aims on expanding the functionality of the mentioned LD-Cnet algorithm using the presented technique to battle concept drift in the multidimensional data.

## References

1. D.J. Newman A. Asuncion. UCI machine learning repository, 2007.

2. Brian Babcock, Shivnath Babu, Mayur Datar, Rajeev Motwani, and Jennifer Widom. Models and issues in data stream systems. In *Proceedings of the twenty-first ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems*, PODS '02, pages 1–16, New York, NY, USA, 2002. ACM.

3. Anton Dries and Ulrich Rückert. Adaptive concept drift detection. *Stat. Anal. Data Min.*, 2(56):311–327, December 2009.

4. Russell Greiner, Adam J. Grove, and Dan Roth. Learning cost-sensitive active classifiers. *Artif. Intell.*, 139(2):137–174, August 2002.

5. Daniel Revuz and Marc Yor. *Continuous Martingales and Brownian Motion (Grundlehren der mathematischen Wissenschaften)*. Springer, 3rd edition, December 2004.

6. Jeffrey C. Schlimmer and Richard H. Granger, Jr. Incremental learning from noisy data. *Mach. Learn.*, 1(3):317–354, March 1986.

7. N. V. Smirnov. Table for estimating the goodness of fit of empirical distributions. *Ann. Math. Stat.*, 19:279–281, 1948.

8. P. Sobolewski and M. Wozniak. Sequential Tests of Statistical Hypotheses. *The Annals of Mathematical Statistics*, 16(2):117–186, 1945.

9. P. Sobolewski and M. Wozniak. Ldcnet: minimizing the cost of supervision for various types of concept drift. In *Proceedings of the CIDUE 2013 - IEEE Symposium on Computational Intelligence in Dynamic and Uncertain Environments*, CIDUE '13, pages 68–75, 2013.

10. Piotr Sobolewski and Michal Wozniak. Comparable study of statistical tests for virtual concept drift detection. In Robert Burduk, Konrad Jackowski, Marek Kurzynski, Michal Wozniak, and Andrzej Zolnierek, editors, *CORES*, volume 226 of *Advances in Intelligent Systems and Computing*, pages 329–337. Springer, 2013.

11. Frank Wilcoxon. Individual Comparisons by Ranking Methods. *Biometrics Bulletin*, 1(6):80–83, 1945.
12. J. Wolfowitz. On Wald's Proof of the Consistency of the Maximum Likelihood Estimate. *The Annals of Mathematical Statistics*, 20:601–602, 1949.

13. Indre Zliobaite and Ludmila I. Kuncheva. Determining the training window for small sample size classification with concept drift. In *Proceedings of the 2009 IEEE International Conference on Data Mining Workshops*, ICDMW '09, pages 447–452, Washington, DC, USA, 2009. IEEE Computer Society.