A comparative study of automatic techniques for ocular artifact reduction in spontaneous EEG signals based on clinical target variables: A simulation case Eye movement artifacts represent a critical issue for quantitative electroencephalography (EEG) analysis and a number of mathematical approaches have been proposed to reduce their contribution in EEG recordings. The aim of this paper was to objectively and quantitatively evaluate the performance of ocular filtering methods with respect to spectral target variables widely used in clinical and functional EEG studies. In particular the following methods were applied: regression analysis and some blind source separation (BSS) techniques based on second-order statistics (PCA, AMUSE and SOBI) and on higher-order statistics (JADE, INFOMAX and FASTICA). Considering blind source decomposition methods, a completely automatic procedure of BSS based on logical rules related to spectral and topographical information was proposed in order to identify the components related to ocular interference. The automatic procedure was applied in different montages of simulated EEG and electrooculography (EOG) recordings: a full montage with 19 EEG and 2 EOG channels, a reduced one with only 6 EEG leads and a third one where EOG channels were not available. Time and frequency results in all of them indicated that AMUSE and SOBI algorithms preserved and recovered more brain activity than the other methods mainly at anterior regions. In the case of full montage: (i) errors were lower than 5% for all spectral variables at anterior sites; and (ii) the highest improvement in the signal-to-artifact (SAR) ratio was obtained up to 40 dB at these anterior sites. Finally, we concluded that second-order BSS-based algorithms (AMUSE and SOBI) provided an effective technique for eye movement removal even when EOG recordings were not available or when data length was short. ᭧ Quantitative analysis and interpretation of human electroencephalographic (EEG) signals are useful methods to study functional states of the brain, to evaluate drug effects and to diagnose psychiatric and neurological disorders. The most widely used techniques to quantify changes in EEG signals are based on spectral analysis and topographic mapping. In humans, EEG power is traditionally divided into spectral broad frequency bands related to EEG generators and rhythms: delta, theta, alpha and beta.. Electrical scalp recordings do not contain only cerebral activity (EEG) but also non-cortical electrical activities, such as artifacts of physiological and non-physiological origin. Physiologic artifacts are generated from the subject and the ocular activity is the most important. Eye movements and blinking are a major source of EEG contamination because they occur very frequently and the electrooculographic (EOG) signal amplitude can be several times larger than brain scalp potentials. Their vertical ocular projection propagates quite symmetrically in a anterior-posterior direction and it contaminates the recorded EEG signals. Lateral eye movements mainly affect lateral frontal electrodes [1] . The potentials generated by ocular activity interfere with the electric field of neural origin mainly, in the anterior scalp regions. To a certain extent, neural activity especially from the anterior scalp area is also recorded by electrodes placed near the eyes. EEG and EOG recordings are therefore a mixture of brain and ocular electrical activity. This is known as bidirectional contamination.. A critical point in EEG signal processing is the need for careful treatment and reduction of these artifacts which contaminate the EEG signals and can lead to wrong results and conclusions.. The aim of the study was to objectively and quantitatively evaluate the performance of filtering methods based on linear regression and component analysis with respect to spectral target variables and topographic maps, both widely used in clinical and functional EEG studies. For this purpose, two different EEG channel configurations were used and even the absence of EOG recordings was also considered.. The most common reduction method is based on linear regression in the time or frequency domain. This method estimates and removes the influence of EOG on the signals recorded by scalp electrodes from the EEG recordings. However, this technique has an inherent drawback that is the aforementioned bidirectional contamination. In other words, regression procedures consider that the EEG recording is not correlated with the EOG recording, and this assumption is generally erroneous. Therefore, relevant cerebral information contained in the EOG recordings can also be cancelled in the EEG recordings corrected by means of regression techniques [2] . Some studies suggest that most of the high frequency range in the EOG is of neural origin. Thus, in order to reduce the cancellation of interesting cerebral information, low-pass filtering of the EOG signals has been proposed before the application of a time domain regression approach [3] . Furthermore, a recent study suggests that ocular artifact effects in EEG signals are not only present in delta and theta ranges, but also in alpha and beta bands, so the above-mentioned filtering does not seem to be an adequate solution [4] .. Other approaches for ocular artifact reduction are based on a linear decomposition of the EEG and EOG recordings into source components, the so-called blind source separation (BSS) techniques. These component-based methods permit the identification of artifactual sources, and then the reconstruction of EEG recordings without these sources. The first proposed component-based procedure was the principal component analysis (PCA) [5] . By means of a rotation, PCA transforms a multi-variable data set into components which are spatially orthogonal. The main drawback with PCA is the assumption of orthogonality between neural activity and ocular artifact and this is not generally true. Lagerlund et al. reported that PCA could not completely separate eye artifacts from cerebral activity, especially if they have similar amplitudes [6] .. In order to overcome the limitations and constraints presented by regression and PCA procedures newer approaches use blind source extraction based on independent component analysis (ICA). The assumption taken into account in these approaches is that sources must be statistically independent, not just uncorrelated and this is mostly true with regard to the brain and ocular components. ICA is currently considered an important technique for removing artifacts from EEG signals [2, 7, 8] . However, most of the studies detect visually artifactual components following a subjective criterion. In order to overcome this subjectivity, few studies have been carried out using a completely automatic artifact correction process based on statistical properties like kurtosis, entropy or correlation with EOG channels [9] [10] [11] . Many articles use these filtering approaches in real EEG data. However, the quantitative evaluation of the goodness of each correction method is very difficult because it is necessary to know which are the true brain and ocular components. To date, few sets of algorithms based on simulated data have been evaluated in order to solve this problem [10, 12] . Furthermore, opposite results were obtained: in the former study, the best performance was obtained with an ICA algorithm [10] , while in the latter, ocular artifact correction methods based on regression and PCA performed better than ICA-based correction procedures [12] . Besides, some of the ICA algorithms that were evaluated and inadvisable in [10] require much longer data segments than the epoch duration considered in that study in order to work properly [13] . In addition, the approach on which the results of [10] were based, an eye tracker procedure, is quite different from the clinical routine, where spontaneous EEG signals are commonly acquired with closed eyes and with normal (not fast forced) eye movements. Moreover, another recent study defended regression-based methods against BSS when the number of EEG leads was small [14] . Concerning this point, a reduced number of electrode configuration or even without EOG leads (where linear regression approach and automatic procedures based on correlation with EOG channels are not applicable) have not been yet analyzed. Finally, the effects of filtering results on the spectral bands widely used as target variables and on the distribution of the correction effectiveness through the scalp topography have not been analyzed either.. This study takes into account all the questions mentioned above and proposes a completely automatic BSS procedure in simulated EEG and EOG recordings with different durations, where the cerebral and ocular components are previously known, in order to simulate real clinical situations. Ocular contamination of EEG data is an important and a very common problem in the diagnosis of neurobiological events because of the overlap between EEG and EOG in frequencies which can lead to wrong conclusions. Several techniques have been proposed to remove eye movement artifacts from spontaneous EEG signals based on regression analysis and BSS. For component-based methods, a new automatic procedure using logical rules, which are based on spectral and topographical information, is presented in order to identify the components related to eye activity. Results showed that all methods were able to reduce eye activity which contaminated EEG data. Evaluation of the performance of these correction techniques is difficult because of the inability to directly measure the electric activity of eyes and brain separately. To quantify the correction success of a method it is necessary to work with simulated EEG signals reproducing standard clinical situations that are as real as possible. The comparative study is based on spectral variables and topographic maps that are commonly used in clinical routine.. Ocular removal procedures based on regression analysis are considered as the gold standard method in spontaneous EEG signals [16] because they provide a relatively simple mathematical solution by removing the ocular component from EEG channels. However, a partial neural component is also subtracted by the regression process due to the mutual contamination between EEG and EOG channels. In this study, this was shown by the corrected signals from the fronto-polar channels (see Fig. 4 ) and by the high errors obtained for absolute alpha power (29.96%). This neural removal was partly reduced by applying a low-pass filter to the EOG channels before the subtraction. Errors with filtered regression were almost zeros for absolute power in fast bands (alpha and beta) since no ocular contamination was included by the simulation procedure. However, worse results were shown for relative powers of these bands (around 12%, see Table 2 ).. Smaller errors were obtained for PCA using short duration data (see Fig. 5 ), although they were not lower than for regression (see Fig. 6 ). PCA does not use a realistic physical solution since there is no reason to assume that EOG and EEG sources are orthogonal. To overcome this unrealistic constraint, other SOS and HOS techniques have appeared which decomposed the sources more properly.. HOS-based techniques use statistic information as higher order cumulants, entropy or maximum likelihood estimation in order to separate the sources by minimizing an approximation of mutual information. They are composed of time consuming iterations for optimizing certain HOS-based cost functions. HOS-based algorithms required a certain data length. Delorme et al. [13] recommend at least a number of samples corresponding to a few times (k) the square of the number of channels. In our case, k · 21 2 = k · 441 samples were needed, where k is an integer. Results showed that errors decreased with higher data length and tended to similar values to those obtained with PCA when increasing data length. Although the lowest errors were obtained with an epoch duration of 180 s (18 000 samples which corresponded to k = 40), similar values to these minima were already obtained with 90 s (9000 samples which corresponded to k = 20). Stationarity of the spatial structure of the sources has to be assumed when a long data epoch is used, because these approaches would not be appropriated in another case.. Separation by SOS and HOS-based algorithms was not expected to be similar because they have different assumptions about sources modeling. Sources extracted by HOS-based methods are generally noisier than sources extracted by SOS techniques. AMUSE and SOBI algorithms use temporal structure of mixed EEG signals to achieve the blind separation. These spatio-temporal decorrelation procedures are simple, fast to compute and robust. Besides, they were not sensitive to the duration of data in the present study. Similar errors were obtained for all epoch durations and for both approaches (see Fig. 5 ). Time and frequency results indicated that these SOS-based ocular removal techniques preserved and recovered more brain activity than the other methods. Best ocular corrected EEG signals were obtained using them for blind separation: errors lower than 5% for all spectral variables at anterior sites (lower than 2.5% as an average of all channels) and the highest SAR improvements up to 40 dB at these anterior sites.. In clinical routine, electrode configurations change depending on the application. Moreover, in last years, many algorithms using different number of EEG and EOG electrodes have been suggested in order to reduce ocular artifacts from EEG data and opposite conclusions were obtained. Up to date, some studies proposed BSS algorithms [9, 10] , others defended regression analysis, especially when the number of EEG channels is small [12, 14] . Although the main part of this study was performed using 19 EEG and 2 EOG channels according to the international 10-20 system, a comparison between methods was also carried out with only six EEG electrodes and even in the worst situation: without EOG recordings. EOG signals are absolutely necessary for regression-based approaches, but not for BSS procedures. However, the studies in the literature focused the automatic ocular identification on the correlation between an IC source and the EOG channels. Considering only six EEG + two EOG channels, errors in spectral parameters obtained for SOBI technique were much lower than those calculated by regression-based methods. Even, errors for SOBI algorithm considering only 19 EEG channels (without EOG) were slightly lower than those obtained with filtered regression approach but recording additionally two EOG channels in the latter.. Therefore, based on the effects quantified on clinical target variables and applying fully automated BSS procedure in simulated EEG and EOG recordings, we conclude that AMUSE and SOBI are the most suitable algorithms to be used in clinical and functional EEG studies, not only independently of the number of EEG leads, but also when EOG recordings are not available.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Offline recognition of unconstrained handwritten texts using HMMs and statistical language models Abstract-This paper presents a system for the offline recognition of large vocabulary unconstrained handwritten texts. The only assumption made about the data is that it is written in English. This allows the application of Statistical Language Models in order to improve the performance of our system. Several experiments have been performed using both single and multiple writer data. Lexica of variable size (from 10,000 to 50,000 words) have been used. The use of language models is shown to improve the accuracy of the system (when the lexicon contains 50,000 words, the error rate is reduced by $ 50 percent for single writer data and by $ 25 percent for multiple writer data). Our approach is described in detail and compared with other methods presented in the literature to deal with the same problem. An experimental setup to correctly deal with unconstrained text recognition is proposed. O FFLINE cursive handwriting recognition systems presented in the literature deal, almost without exception, with single words [1] , [2] , [3] . This happened, in our opinion, because research focused on application domains (handwritten address recognition and bank check reading) where only single words are involved. Moreover, only recently data sets allowing experiments on the recognition of texts were made available [4] , [5] , [6] . In a few works (see Section 2), postal addresses or legal amounts have been recognized as phrases. The solution of such problems relies on important information coming from the application environment (zip code, courtesy amount, etc.). Moreover, the address lines and the courtesy amounts have a structure that is not very variable. In both cases, the data is made of a few recurring elements that change at most their order in the phrase. The lexica involved are relatively small (20-30 words for bank checks and 10-1,000 for address lines) and, even more important, are closed. This means that all words appearing in the data are represented in the lexicon.. This work presents an offline cursive recognition system dealing with large vocabulary unconstrained handwritten texts. This application is different from the above-mentioned ones under several aspects. The variability in the texts is much higher in terms of phrase structure. The only hypothesis that can be made about an unconstrained text is that it is written in a certain language. No other constraints can be applied to make the problem easier. An unconstrained text can have any kind of content and this makes it very difficult to select a lexicon. The dictionary must be extracted from data independent from the test set and, regardless of its size, it will never contain all the words represented in the data to be recognized. In other words, the lexicon cannot be closed and the data contains Out Of Vocabulary words (OOVs).. The texts we recognize are the transcription of documents belonging to corpora assumed to reproduce the statistics of average English. This allows us to apply Statistical Language Models (SLMs) in order to improve the performance of our system [7] . We used N-gram models (the most successful SLM applied until now [8] ) of order 1, 2, and 3 (called unigrams, bigrams, and trigrams, respectively).. Previous works typically preferred the application of syntax-based postprocessing in order to include linguistic knowledge in the recognition. This approach has the important limit of separating the recognition of handwritten data from the application of language modeling. For this reason, this work applies a different approach where handwriting recognition and language modeling are tightly integrated and performed at the same time. This represents an important element of novelty with respect to previous works in the literature (see Section 2) and has several advantages (see Section 5.3).. When passing from single word to text line recognition, several changes in the experimental setup are necessary. The most important one concerns the selection of the lexicon (see above). In order to deal with unconstrained texts, the lexicon must be large enough to have a high probability of containing the words appearing in the text. On the other hand, if there are too many dictionary entries, the recognition problem becomes difficult. A good trade off between these two effects must be found.. Another important difference is in the way the performance of the system is measured. In single word recognition, a sample is correctly or incorrectly recognized (there is a single kind of error). In text recognition, there are several kinds of error. A word can be not only misclassified, but also deleted. Moreover, words not appearing in the text can be inserted during the decoding. This leads to different performance metrics that must be selected depending on the task the system is used for. Several experiments (changing the size of the lexicon from 10,000 to 50,000 and the order of the language model from 1 to 3) were performed over both single and multiple writer data. Part of the data sets used are publicly available in order to allow a comparison with other systems.. To our knowledge, the recognition of unconstrained texts has been addressed in very few works [9] , [10] . The approach used in [9] is significantly different from ours (see Section 2 for more details). The system described in [10] does not apply trigrams and uses a closed vocabulary which leads to an overestimation of the performance. Part of the results presented in this work have been shown in [11] .. The rest of this paper is organized as follows: Section 2 gives an overview of the works dealing with word sentences, Section 3 provides the statistical foundations of our approach, Section 4 presents SLMs and N-gram models, Section 5 describes the recognition system used in our work, Section 6 reports experiments and results obtained, and Section 7 draws some conclusions. This work presented a system for the offline recognition of handwritten texts. The recognizer is based on continuous density Hidden Markov models and Statistical Language models (unigrams, bigrams, and trigrams). Several experiments were performed using both single and multiple writer data.. The experimental setup reproduces the conditions of unconstrained text recognition: No information about the content of the documents to be transcribed is used, except for the fact that they are written in English. The differences with respect to the case of single word recognition are highlighted. Lexica of different sizes, (from 10,000 to 50,000 words) extracted on a statistical basis from the corpus used to train the language models, were used. The performance of the system was measured in terms of accuracy and recognition.. The effect of the SLMs changes depending on the data. In all our experiments, the N-grams were able to significantly improve the performance of the system independently of the metric used. On the other hand, to pass from unigrams to trigrams results in an improvement of recognition and accuracy only for IAM and Reuters databases. This happens, in our opinion, because the models trained over Cambridge data have already a very high performance without SLMs. The space left for improvement is thus reduced and limited to a part of the text (namely, the functional words) where models of different order have a similar perplexity (see Sections 6.3 and 6.4).. The improvement obtained with SLMs is especially high for the Reuters database. The reason is, in our opinion, that the language model is better aligned with the data to be recognized. The SLMs have been trained over a corpus of documents produced by the same source (the Reuters newswire bulletin) that created the data to recognize. This makes the system more constrained, but more effective. It must be taken into account the fact that SLMs are penalized in our experimental setup because of an important aspect: The lines are decoded separately, so the models of higher order are effective on a smaller part of the data (see the end of Section 6.2). A solution to this problem can be the concatenation of different lines before the recognition, but this requires the system to produce few observations per line in order to reduce the dimension of the search space (see Section 5.3).. In its current configuration, the system appears to be more suitable for the application of content modeling techniques (Indexing, Information Retrieval, Topic Detection, etc.) than for the actual transcription of texts. All symbols other than letters are in fact not modeled and treated as noise. This is due to the fact that only the letters are represented sufficiently for a reliable training. In order to obtain a system more oriented toward the exact transcription of documents, it is necessary to increase the amount of training material (especially for the less represented symbols).. At the present state-of-the-art in language modeling, the possibility of having models performing better than N-grams seems unlikely. The application of other language modeling techniques (e.g., grammars or decision trees) seems to be limited to very constrained problems, like bank check reading, date processing, or postal address recognition. The possibility of adapting generic N-gram models to specific problems can be explored in the same way as writer independent HMMs can be adapted to writer dependent data.. Since very few works have been dedicated to the offline recognition of handwritten texts, the above list of possible future directions is far from being exhaustive. The availability of systems for text recognition can give rise to new applications that were not considered until now. Especially, from this point of view, the recognition of handwritten texts represents an interesting and promising research domain. This section presents a survey of the works dedicated to the recognition of handwritten word sequences or phrases. They can be grouped into three classes depending on the kind of data involved: postal addresses, courtesy amounts, and unconstrained texts.. The recognition of legal amounts on bank checks involves small lexica (20-30 words) and a very low variability. Even if the amounts change, their structure is always the same. Moreover, the courtesy amount (amount written in digits) can be used to support and validate the recognition. In most cases, the recognition systems simply transcribe the different parts of the amount separately (e.g., three hundred fifty-five), but, in a few works, the fact that the amount can be considered as a phrase is used. In [12] , [13] , the amount is first segmented into single words that are recognized separately, then a syntactical postprocessing is applied in order to discard solutions that are not plausible from a linguistic point of view. Such an approach is effective in this problem because the grammars modeling the amounts are relatively simple. The use of the same approach in problems involving more variable data (especially unconstrained texts) is less effective or requires the use of tags grouping the words into few classes (e.g., verbs, adjectives, etc.).. The goal of postal address recognition is the selection of a delivery point, i.e., of a postal plant where a certain mail piece must be routed in order to reach its recipient. An address line contains several elements: a topographic name (e.g., street or road), a street name, and a number. Different versions of the same element (e.g., Street, St., and Str.) and different ordering of the elements are the main sources of variability. On the other hand, only the street name is relevant to the delivery point selection. This is thus the only information that needs to be correctly transcribed. Two important constraints help to solve the problem. The first is that the number of elements in an address line is very limited (three or four, rarely more) and that the same elements are always present. The second is that the dictionaries are closed, i.e., no OOVs are expected.. Several approaches have been proposed to the problem. In [14] , [15] , [16] , a system that segments the lines into words and then recognizes them separately is presented. In this way, the phrase recognition is reduced to a sequence of single word recognition steps. The main limit of the approach is the segmentation of a line into words. This is a difficult process and requires a large effort. Moreover, the segmentation process is not perfect and represents an important source of error: The method described in [15] incorrectly segments around 10 percent of the lines.. An alternative approach is proposed in [17] , [18] , where a keyword spotting technique is used to detect the name of the street. The important advantage of such an approach is that no segmentation of the line into words is required. The keyword spotting step obtains, in fact, the segmentation as a byproduct. The use of keyword spotting is possible because it is not necessary to recognize other words than the street name. The topographic name can be recognized only to apply lexicon reduction mechanisms: Once the topographic name is recognized (e.g., Road), all street names associated with different topographic entities (Street, Avenue, etc.) are discarded.. Few works were dedicated to the recognition of unconstrained texts. Such a problem assumes that no hypothesis can be made about the content of the data except for the fact that they are written in a certain language. This allows the application of language models. Also, in this case, several approaches have been proposed to solve the problem.. An approach based on segmentation and recognition of single words is proposed in [19] , [20] , [21] . After the words of a line have been recognized, a linguistic postprocessing is applied in order to use context information. The words are tagged following gramatical categories and a model of the transition between different categories is used to reweight the recognition scores. Since the segmentation is not a perfect process, several segmentation hypotheses must be considered. The segmentation poses the same problems described above and it is an important source of error. It is not clear in the cited works how the lexica were selected. It is especially important to know whether the lexicon is closed (no OOVs expected) or not. A closed dictionary is a simplifying condition, but is a strong constraint on the texts that can be recognized. Moreover, in order to have a closed dictionary, it is necessary to have information about the test set when building the system. This is not realistic when working on unconstrained texts.. An approach based on HMMs and N-grams has been proposed in [10] . This approach has the important advantage of avoiding the segmentation (which is a byproduct of the recognition). The main limit of this work is the use of a closed vocabulary of small size (less than 10,000 words). The lexicon has been extracted from the data used to test the system and this is not correct because no information coming from such a set should be used when building the system. Moreover, the resulting system is overfitted to the test set, leading to an overestimation of the performance. In such a configuration, the system is also not robust with respect to a change of data. When moving to data having a different topic, most of the words will not be represented in the lexicon and the recognition performance will be dramatically decreased.

O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I I O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Coevolutionary makespan optimisation through different ranking methods for the fuzzy flexible job shop In this paper we tackle a variant of the flexible job shop scheduling problem with uncertain task durations modelled as fuzzy numbers. To minimise the schedule's makespan, we consider different ranking methods for fuzzy numbers. We then propose a cooperative co-evolutionary algorithm with two different populations evolving the two components of a solution: machine assignment and task relative order. Additionally, we incorporate a specific local search method for each population. The resulting hybrid algorithm is then evaluated on existing benchmark instances, comparing favourably with the state-of-theart methods. The experimental results also serve to analyse the influence in the robustness of the resulting schedules of the chosen ranking method. The importance of scheduling as a research topic is undeniable, both as a source of interesting complex combinatorial optimisation problems and as a field with multiple real applications in industry, finance, welfare, etc. In particular, shop problems in their multiple variants-for instance, incorporating flexibility or operators-can model many situations which naturally arise in manufacturing environments [1] .. Fuzzy sets have contributed to enhancing the applicability of scheduling, helping to bridge the gap between classical techniques and real-world user needs. They have been used both for handling flexible constraints and uncertain data [2] , [3] , [4] , [5] . They are also emerging as an interesting tool for improving solution robustness, a much-desired property in real-life applications [6] , [7] , [8] . Incorporating fuzzy sets to scheduling is, however, far from trivial, and they usually require a reformulation of the problem under consideration and the development of new solving techniques. In that sense, one important issue (shared with many other applications of fuzzy sets) is how to rank different solutions when their quality is given as a fuzzy quantity. In deterministic scheduling, the complexity of problems such as job shop means that practical approaches to solving them usually involve metaheuristic strategies [9] . Some attempts have been made to extend such methods, mostly evolutionary algorithms, to the case where uncertain durations are modelled via fuzzy intervals. In particular, the fuzzy flexible job shop problem is receiving an increasing attention, with proposals including a genetic algorithm [10] , a hybrid artificial been colony algorithm [11] , an estimation distribution algorithm [12] , a swarm-based neighbourhood search algorithm [13] and a co-evolutionary algorithm [14] .. Indeed, co-evolutionary algorithms [15] [16] are a special case of evolutionary algorithms which are proving to be very successful in solving complex problems [17] , [18] , [19] . They have been also applied for solving different scheduling problems: for example in [20] are used to solve the integrated problem of process planing and scheduling in job shop problems, in [21] for the stochastic job shop problem or in [14] for the fuzzy flexible job shop.. In the following we tackle the fuzzy flexible job shop problem, where uncertainty in task durations is modelled using fuzzy numbers. After introducing the problem, we consider different ranking methods to minimise the resulting fuzzy makespan and give a definition of schedule robustness based on average behaviour across all possible cases. We shall see how the problem naturally lends itself to coooperative co-evolution and we shall also propose neighbourhood structures for each population, so local search can be embedded in the co-evolutionary algorithm. The experimental results will illustrate the synergy between the co-evolution and the local search, as well as the competitiveness of our approach when compared to the state-of-the-art. The results will also allow for an empirical assessment of different ranking methods in terms of solution robustness. We have tackled the flexible job shop scheduling problem with fuzzy durations and have proposed a new cooperative co-evolutionary algorithm hybridised with local search, named CELS, to solve it. The experimental results have assessed the quality of the initial heuristic seeding and the synergy between coevolution and local search. They have also shown that CELS outperforms values for deterministic and fuzzy predictive schedules the state-of-the-art methods, establishing new best known solutions and, in two cases, even finding the optimal solution. In addition, we have assesed the behaviour of several ranking methods for TFNs in these problems by means of a robustness measure. This measure accounts for the average behaviour of predictive schedules in real situations. We have seen that the use of different ranking methods actually affects the robustness of the algorithm's outcome. Due to the nature of scheduling problems, this outcome is more robust when ranking is based on the expected value of the TFNs, modelling a decision maker who compromises between pessimism and optimism, followed by the approach modelling a pessimistic decision maker, with the optimistic method being definitely worse. These results strongly support the use of the expected value as a ranking method for fuzzy scheduling problems.

O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I I I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Single tap identification for fast BCI control Abstract One of the major aims of BCI research is devoted to achieving faster and more efficient control of external devices. The identification of individual tap events in a motor imagery BCI is therefore a desirable goal. EEG is recorded from subjects performing and imagining finger taps with their left and right hands. A Differential Evolution based feature selection wrapper is used in order to identify optimal features in the spatial and frequency domains for tap identification. Channel-frequency band combinations are found which allow differentiation of tap vs. no-tap control conditions for executed and imagined taps. Left vs. right hand taps may also be differentiated with features found in this manner. A sliding time window is then used to accurately identify individual taps in the executed tap and imagined tap conditions. Highly statistically significant classification accuracies are achieved with time windows of 0.5 s and more allowing taps to be identified on a single trial basis. Brain Computer Interfaces (BCIs) provide a means for controlling a computer that bypasses the peripheral nervous system by extracting signals directly from the brain (Wolpaw 2007) . Control is therefore achieved via thought alone making BCIs well suited for use by disabled individuals who may otherwise by unable to use a computer, communicate or exert control over their environment (Vaughan 2006) .. BCIs may be used to control spelling and communication programs; allowing users to chat, send email etc. They may also be used to control a range of devices such as wheelchairs and prosthetic limbs, allowing their users to achieve a level of communication and control that may not otherwise be possible (Birbaumer 2000) .. The Electroencephalogram (EEG) arises from the currents recorded via electrodes placed across the surface of the scalp which are generated by the summed field potentials of the simultaneous firings of large numbers of cortical neurons. It is a commonly used method for attaining the brain signals used for BCI control (Wolpaw 2007) . This is because EEG is non-invasive, simple to setup and use and relatively portable. It is hence considered to be highly suitable for use by individuals in their own homes (Vaughan 2006) .. A common paradigm for the control of a BCI is motor imagery. The BCI user imagines movement in some part of their body and in response the frequency content of the EEG recorded from electrodes located over, and close to, the motor cortex exhibits increases and/or decreases in power, referred to as Event Related Synchronization/Desynchronization (ERS/D); respectively (Pfurtscheller and Neuper 2001) .. Many different types of motor imagery paradigm are used in BCI control, including the use of imagined hand and foot movement to achieve various types of control. A particular type of motor imagery paradigm is based on finger tapping (Ang 2008) . The user imagines they are tapping their fingers and the task of the BCI is to identify when the subject is tapping for use as a control signal. For example imagined tapping may be used to turn on or off a switch, select from a series of options, move a cursor etc.. Current finger tapping based BCIs make use of the selfpaced tapping condition for control. The subject simply imagines they are repeatedly tapping their fingers at their own pace and the detection of this action is used as a basis for the BCI control (Congedo et al. 2006) .. In Ang (2008) finger tapping over a period of 2 s is identified at accuracies ranging from 53.0 to 96.0% with mean accuracies of 73.3 ± 2.8% achieved by healthy subjects they recorded. An improvement in accuracy is achieved in (Blankertz et al. 2006 ) with a mean classification accuracy of 89.5% achieved when identifying continuous finger tapping within trials of length 2 s.. This work takes a different approach and attempts to investigates whether a more precise and accurate level of BCI control could be achieved via the identification of individual taps-as opposed to continuous tapping-and at what time resolution this may be achieved. First by using Differential Evolution to identify distinguishing features and then identifying individual taps in a manner analogous to online BCI operation. Hence this work comprises a step towards more intuitive, accurate and faster BCI operation.. Individual finger taps are potentially a very intuitive control paradigm, being analogous to key pressing, a computer control paradigm a great many people are familiar with. Hence identification of taps on a single trial basis could allow for much faster and more intuitive BCI control then is currently possible.. It is well known that executed finger movement and imagined finger movement produce very similar responses in the EEG (Ang 2008; Jeannerod 1995) . Therefore EEG is recorded both during executed finger taps and imagined finger taps at different rates and with either the left or right hands.. The problem is approached as one of feature selection. It is necessary to identify first which channels and frequency bands give the most information about the tap events. To find the appropriate features-the channels and frequency bands of interest-Differential Evolution (DE) is used to search the parameter space of possible channel-frequency band combinations; all the channels in the International 10/20 system of electrode placement and all frequencies from 0.1 to 45 Hz are considered.. After identification of appropriate channels and frequency bands online BCI operation is simulated. A sliding window threshold function is trained and applied to classify the power of the selected frequency bands on the selected channels into tap vs. no-tap conditions. The shorter the time window at which taps can be accurately identified the faster and more accurate the BCI control that can potentially be achieved.. Section 2.1 first describes how the data is recorded for this investigation. Section 2.2 then introduces DE as a feature selection method and Sect. 2.3 describes how DE is used to select channels and frequency bands for tap identification. Section 2.4 details the classification method used in this work and Sect. 3 presents the results of first finding appropriate channel-frequency band combinations and then identifying taps in the executed and imagined conditions. Finally Sect. 4 discusses the results and their applications to BCI. Motor imagery is a common BCI paradigm that can make use of finger tapping imagery. This work looks at identification of single taps as a control signal, as opposed to other finger tapping based BCIs which rely on the user making/imagining repeating many taps to achieve control of the BCI.. A DE algorithm is applied to identify appropriate channels and frequency bands for tap identification. A sliding window is then applied, in a manner analogous to online BCI operation, to attempt to identify taps from the mean band power on the selected channels and at the selected frequency bands.. Taps are identified over very short time windows in both the executed movement and imagined movement conditions. Left hand vs. right hand single taps may also be differentiated for approximately half the subjects. This presents the possibility for both faster BCI control and tap pattern based control. Finger tapping is a popular paradigm for motor imagery BCI control in which the user imagines making repeated movements with their fingers in order to control a switch or device. Most current implementations of the finger tapping paradigm require the user to make many repeated taps before the act of tapping is recognized as a control signal (Pfurtscheller and Neuper 2001; Congedo et al. 2006; Balakrishnan and Puthusserypady 2005; Stavrinou et al. 2006) . This work investigates if it is possible to identify the tapping condition with fewer taps and whether individual taps are a feasible control condition for BCI. This may be thought of as an analogous study to much of the work going on in single trial Event Related Potential (ERP) analysis [for example (Williams et al. 2009)] , which aims to detect ERPs on a single trial basis and thus allow for faster and more intuitive BCI control via the ERP based paradigm.. The DE algorithm is first used to select informative EEG channels and frequency bands at which the mean power may be used for classification of tap events into different conditions; imagined taps and executed taps made with either the left hand or the right hand or with the tapping hand discounted. A large variety of different frequency bands on different channels are selected by the DE algorithm for these classifications. These channel-frequency band selections may be used to achieve statistically significant levels of tap identification, as assessed via the method described in Sect. 2.4.. The variety of frequencies and channels selected reflects the well known inter-trial and inter-subject variability in Fig. 5 Theoretical bits per minute achievable by an online BCI using using the sliding window approach as a function of window size the EEG (Thulasidas et al. 2006) . Task relevant components of the EEG are known to change in temporal, spatial and spectral location and morphology from trial to trial and from subject to subject. This variability is a known issue in BCI research and can lead to a lack of robustness and reliability in BCI systems. In this work the DE algorithm selects-for every subject and every condition-a number of different channels and frequency bands to use in the classification of tap events. In doing so the classifier is able to use a number of spectral and spatial locations which could, potentially, lead to more robust classification results. Further to this, because the selection of the channels and frequency locations is made individually for each subject and each condition inter-trial and inter-subject variabilities may be better accommodated for.. AUCs achieved in differentiating tap vs. none tap events using mean frequency powers identified by DE exceed 0.90 for some of the subjects in the executed condition and one subject in the imagined condition. In general the AUCs are in the range of 0.6-0.9 which compares well with analogous 1-dimensional motor imagery BCI paradigms such as (Fabiani et al. 2004 ) which achieves accuracies of between 71.8 and 74.3% or (Ang 2008) which achieves a mean accuracy of 73.3 ± 2.8%.. Differentiating left vs. right hand taps in the executed and imagined tap conditions produces statistically significant discriminations in a number of cases in the executed tap condition. AUCs produced are around 0.70 with a maximum of 0.79 this compares reasonably to (Pfurtscheller and Neuper 2001) which reports an average accuracy of left vs. right hand movement imagery differentiation of 75.0%. Although it should be noted that left-right taps could only be differentiated at a statistically significant rate in approximately half of the subjects.. A sliding window approach is applied to attempt to identify individual tap events in the EEG. The electrode channels chosen by DE are used as the basis for control. The mean band powers of the EEG at the selected frequencies are calculated on these channels. A classification function is then applied within the sliding window to identify tap events. Thus DE is used to first select which channels and frequency bands to concentrate on. Classification is then applied in a manner analogous to online BCI operation.. The subjects are cued to tap at different rates from once every 0.5 s to once every 2 s, interspersed with occasional blocks of rest periods. The length of the sliding window is varied from 0.1 to 1.0 s in steps of 0.1 s. The sliding window is therefore likely to contain just one tap event and, when it's length is 0.5 s or less, it's guaranteed to contain just one tap event.. Classification applied within time windows of 1 s and less produces highly statistically significant identification of taps, both imagined and executed. Shorter time windows also allow classification of taps at a statistically significant rate although the AUC decreases as an inverse function of the window length. The minimum window length at which highly statistically significant (P \ 0.01) rates of tap identification can be achieved is 0.5 s, therefore single taps can be identified via a sliding time window in an analogous manner to online BCI operation.. Importantly using channel and frequency band combinations identified by the DE feature selection algorithm allows identification of the correct channels and frequency bands for tap identification to be performed. Within these channel-frequency band combinations classification may be used to identify executed and imagined movements and hence allow highly statistically significant identification of executed and imagined taps.. The hand in which the subject performs, or imagines performing, a tap may also be used as a control condition. Statistically significant identification is achieved in identifying the tapping hand for approximately half of the subjects, suggesting a possibility for the use of the tapping hand as an additional control condition in a number of cases Additionally EEG recorded during imagined taps is informative for the differentiation of tap vs. none tap events in the imagined tap condition. This indicates that the channels and frequency bands used to differentiate taps from none taps in the executed condition may also be used to differentiate taps from none taps in the imagination condition. This further confirms the results reported in (Jeannerod 1995) that the neurophysiological correlates of motor imagery and motor execution are similar.. An important criterion for ranking BCIs is the bit ratebits per minute (bits/min)-a measure of the speed of information transfer that the BCI may achieve. As shown in Fig. 5 the theoretical bit rate for a BCI based upon single finger tap identifications in the left and right hands varies as a function on the size of the sliding window. The shorter the window the better the bit rate the BCI can achieve but also the lower the accuracy, as the bit rate measure incorporates individual trial accuracies this suggests that the best choice for a practical BCI is to use a short window length which still achieves statistically significant levels of accuracy, e.g. 0.5 s.. Single tap identification is an important result for BCI because it paves the way for much faster, more accurate motor imagery based BCI's. This could potentially include BCIs operated by users imagining patterns or sequences of rapid taps to achieve different control conditions in a fast and intuitive manner. It could also lead to the use of patterns of taps as a control mechanism for BCI, something akin to a tap based morse code for BCI control. Future work will focus on building online BCI systems to differentiate different tap conditions, including tap pattern recognition. The results may be split into a number of distinct sections based upon the conditions; tapping vs. not tapping and left tapping vs. right tapping in both the imagined and real (executed) tap conditions that the subjects perform. Information about the hand with which the tap is to be performed (or imagined) may also be discounted such that DE attempts to identify features for differentiating tap vs. no tap conditions across all the trials regardless of the tapping hand used.. Firstly the results achieved by using the DE feature selection algorithm to find features for differentiating between tap and no tap conditions on each of the datasets for the different subjects and for both imaginary and executed taps are listed. The selected channels and frequency bands to be used in the subsequent sliding window based task are also presented.. Secondly the results achieved by moving the sliding window along the length of the EEG recorded from the subjects under each of the conditions-imagined or executed taps-are listed. This approach is analogous to the operation of an online BCI and provides an indication of the ability to differentiate control conditions, based upon individual taps for use as a BCI control signal. Figure 1 provides an example of typical EEG recorded during imagined finger taps made by a representative subject with their right hand and recorded on electrode C4. Note the period of relaxation followed by a block of cued taps each one second apart. Figure 2 illustrates on which channels and at which frequency bands taps may be identified in the executed tap condition. The channels and frequency bands found to best differentiate taps and none taps for each subject are amalgamated into the figure providing an illustration of the range of different frequencies and channels which may be used to differentiate tap and none tap conditions. Figure 3 illustrates which channels and frequency bands may be used to differentiate imagined taps from imagined none tap events. As can be seen from Figs. 2 and 3 both the executed and imagined tap events may be differentiated using a wide range of channels and frequencies. There is a great deal of inter-subject and inter-condition variability in the channels and frequency bands which may be used to best identify taps. It is difficult to identify a general pattern to the channels and frequency bands that are best for tap identification. Channels over the Motor cortex such as Cz and channels over the Paratial cortex such as Pz are frequently selected although many of the other channels are also selected occasionally. Frequencies in the range 10-30 Hz are most often used to differentiate tap and none tap conditions, although frequencies outside this range are also used. Table 1 lists the AUC's achieved in the classification of features identified by DE on both the training and verification sets recorded in the executed tap conditions across all subjects. The hand used to make the tap is discounted for these results. Subjects with statistically significant AUC's (P \ 0.05) are indicated with an asterisk.. By way of comparison Table 2 lists the accuracies achieved in both the training and verification sets when classifying the data into tap vs. none tap conditions via features selected by DE when the tapping hand is discounted. Again statistically significant accuracies (P \ 0.05) are indicated with an asterisk.. As is seen from the table very high AUCs can be achieved with the majority of subjects in both the executed and imagined tap conditions. In the majority of cases AUCs are statistically significant (P \ 0.05) as assessed against the null hypothesis detailed in Sect. 2.4. Only subject P did not achieve statistically significant AUCs in both the executed and imagined tap conditions. Two other subjects (subject H and subject M) also did not achieve statistically significant AUCs in the imagined tap condition and statistically significant AUCs were also not achieved for subject Q in the executed tap condition.. As the results demonstrate the tap vs. no-tap conditions may be differentiated with a high AUC for both conditions. The AUCs achieved are, in general, lower for the imagined condition then for the executed condition for the majority of subjects. However for some of the subjects the AUCs achieved from the imagined tap condition are marginally higher; although these differences in AUCs between imagined and executed taps are generally not statistically significant (P C 0.05).. The hand-which performs the tap or in which the tap is imagined-is now considered as a control condition. EEG trials are split into left vs. right taps in both the imagined and executed conditions and DE is applied to identify appropriate channel-frequency band combinations to differentiate the conditions. Table 3 lists the channel-frequency band combinations DE identifies for differentiating left vs. right taps in the EEG recorded during both the executed and imagined tap conditions. The accuracies achieved by the classifier on the test set using the selected channels and frequency bands are also listed. As evidenced from the table it is possible to differentiate left vs. right taps in the executed condition at a statistically significant rate (P \ 0.05) for approximately half of the subjects, specifically subjects; B, D, E, I, K, M, O, R and S. This may be contrasted with the AUCs for differentiating the tapping hand in the imagined tap condition. The tapping hand may be identified at a statistically significant rate for the following subjects in the imagined tap condition; B, C, D, I, G, H, I, J, M, N, O and R. This demonstrates that it is possible to use the tapping hand as a control variable for single tap BCI control in approximately half of the investigated cases.. Online BCI operation is simulated by the use of a sliding window. The features (mean power at the channels and frequency bands selected by DE) are extracted from within the sliding window and classified to attempt to identify tap events in the ongoing stream of EEG data. The data is split into training and testing data sets to first train and then evaluate the classification algorithm.. The length of the sliding window is varied from 0.1 to 1.0 s. For each window length the window is slid over the EEG and classification accuracies (as measured via the AUC) are calculated. The shorter the window length the less EEG is required before classification is performed. Shorter window lengths mean more taps may be performed by the subject within a given time period and the BCI may hence be operated faster providing a greater level of responsiveness to the subject. Figure 4 provides an illustration of how varying the windows size effects the classification accuracy for a representative subject with EEG recorded in the imagined condition. As illustrated the longer the window length the greater the classification accuracy achieved as a more accurate estimate of the motor intention of the subject is reflected in the frequency content of the EEG. It is important to note also that statistically significant accuracies (P \ 0.05) are achieved with window lengths of 0.3 s or greater, highly significant accuracies (P \ 0.01) are achieved with window lengths of 0.5 s or greater.. Bits per minute (bits/min) is an important measure of BCI performance (Dennis et al. 2003) . Figure 5 shows how the theoretical (bits/min) achievable with an online BCI using the AUCs achieved with the sliding window based classification technique varies with the size of the windows. Table 4 lists the AUCs achieved on the verification sets for each subject in both the executed and imagined tap conditions when the optimal window size and threshold, found from the training set, are applied. Note that statistically significant AUCs are achieved on both conditions for the majority of the subjects, although as highlighted in Fig. 4 these accuracies fall to insignificance when the sliding window length is shortened towards 0.2 s or less.. As shown in the table the accuracies achievable for differentiating tap vs. none tap conditions are statistically significant in both the imaginary and real conditions for a number of subjects. Coupled with typical window lengths of 0.5 s this demonstrates that it is possible to identify single tap events in the EEG using a sliding window approach with the appropriate channels and frequency bands. Additionally cross-condition training is applied to test the ability of classifiers trained on EEG recorded in the executed tap condition to classify imagined taps. The electrophysiological correlates of real and imagined taps are known to be similar (Jeannerod 1995) and therefore using real taps as the training set to inform about imagined taps in the verification set is feasible. It's also worth noting that in the executed tap condition it's possible to know the exact time of executed taps from the subjects pressing of a button. Thus more accurate estimates of the times of executed taps can be used to inform the training of a threshold to differentiate imagined taps, where the exact tap time is unknown.. When applying cross-condition training a mean AUC of 0.815 ± 0.017 may be achieved in the verification sets across all subjects. This demonstrates that statistically significant differentiation of tap vs. no-tap conditions in EEG recorded in the imagination condition is possible when the threshold value has been trained in the execution condition. Thus it may be inferred that the mean magnitude of the frequency content of the EEG on the DE-selected channels and frequency bands is similar between the imagination and execution conditions.

O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B I O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Mining permission patterns for contrasting clean and malicious android applications • A pattern mining algorithm is proposed to identify contrast permission patterns.• We collected a new dataset with 1227 clean Android applications.• We considered both required and used permission.• Biclustering method has been employed to provide visualization.• The permission patterns show big contrasts between clean apps and malware. a r t i c l e i n f oAn Android application uses a permission system to regulate the access to system resources and users' privacy-relevant information. Existing works have demonstrated several techniques to study the required permissions declared by the developers, but little attention has been paid towards used permissions. Besides, no specific permission combination is identified to be effective for malware detection. To fill these gaps, we have proposed a novel pattern mining algorithm to identify a set of contrast permission patterns that aim to detect the difference between clean and malicious applications. A benchmark malware dataset and a dataset of 1227 clean applications has been collected by us to evaluate the performance of the proposed algorithm. Valuable findings are obtained by analyzing the returned contrast permission patterns. Smartphone is used to describe a mobile device equipped with enhanced computing capability and connectivity [1] , such as Nexus by Google [2] , iPhone by Apple [3], Blackberry by RIM [4] and Windows Phone by Microsoft [5] . In the past few years, the global telephony industry has witnessed an upsurge in the sales of smartphones. A smartphone is usually sold with an in-built mobile operating system (OS) together with a number of pre-installed ''applications'' packaged by the device manufacturer. An application, the software running on smartphones, enhances the smartphone's functionality and supports the interaction with end users to accomplish their tasks. Calendar, address book, alarm clock, media player and web browser are the common applications provided by the device manufacturers, but one important application exists on every smartphone-the ''application store'', which allows end users * Corresponding author. Tel.: +61 411645497.. E-mail addresses: v.moonsamy@research.deakin.edu.au (V. Moonsamy), jiarong@acm.org, jiarong@tulip.org.au (J. Rong), swliu@deakin.edu.au (S. Liu).. to access online application markets to browse and download additional applications of their choice.. Every device manufacturer hosts an application market for its own OS platform, such as Apple's App Store [6] , Blackberry's App World [7] and Google Play [8] . However, far before the first official application markets were introduced in 2008 by Apple, smartphone application distribution was highly dependent on thirdparty sources, where individual application developers were free to upload their products. Due to a huge number of low-price applications being available, there is still a large group of end users who prefer visiting third-party application markets, but not all the applications from markets are ''safe''. The software that is specially designed to harm a device, its OS or other software is called ''Malware'', which stands for malicious software [9] . The increasing sales of smartphones has pushed the rapid growth of mobile malware.. As pointed out by Zhou and Jiang [10] , malware or malicious applications might cause a series of user unexpected operations, for example, stealing user's personal information, making calls or sending an SMS without the user's knowledge. Such malicious behaviors not only cost users extra data usage, but also potentially bring privacy issues. Furthermore, the users may not be aware of running malware on their smartphones because in many cases the malware are downloaded and/or installed without authorization. Accordingly, an efficient and effective malware detection technique is highly demanded to protect smartphone users from the potential prevalence.. To effectively detect malware from millions of applications available on official and third-party markets, many efforts have contributed to studying the nature of smartphone platforms and their applications in the past decade. As the most popular mobile platform, Google's Android overtook others to be the top mobile malware platform. The Android platform employs the permission system to restrict applications' privileges to secure the users' privacy-relevant resources [11] . An application needs to get a user's approval for the requested permissions to access the privacy-relevant resources. Thus, the permission system was designed to protect users from applications with invasive behaviors, but its effectiveness highly depends on the user's comprehension of permission approval. We refer to the permissions that are requested during application installation as required permissions. Unfortunately, not all the users read or understand the warnings of required permissions shown during installation. To improve this situation, many researchers have tried to interpret Android permissions and their combinations [12] [13] [14] [15] . Frank et al. [11] proposed a probability model to identify the common required permission patterns for all Android applications. Zhou and Jiang [10] listed the top required permissions for both clean and malicious applications, but only individual permissions were considered by frequency counting. A problem is still remaining of whether the patterns in a permission combination can provide better performance for malware detection. Furthermore, in the existing literature, only the required permissions are considered in permission pattern mining, no work has incorporated the used permissions that are extracted from static analysis by the Andrubis system (http://anubis.iseclab.org) [16] . Therefore, we are the first group to explore both the required and used permissions. Accordingly, our aim is to propose an efficient pattern mining method to identify a set of contrast permission patterns that effectively distinguish malware from the clean applications.. By using a pattern mining technique to identify the desired permission patterns, we need two datasets: one has only clean Android applications and the other contains all malicious ones. In 2012, Zhou and Jiang [10] published the first benchmark dataset of malicious applications in 49 malware families, which was collected from third-party markets between August 2010 and October 2011. This is an ideal malware dataset for our experiments. On the other hand, due to the lack of a dataset of clean applications published at the same time period as Zhou and Jiang's, we collected our own clean dataset. The clean applications were collected from two popular third-party Android applications markets: SlideME (http://slideme.org) and Pandaapp (http://android.pandaapp.com). We sorted the collected applications based on the times of their download and the ratings given by the users, and only the top ones were picked. Each application was scanned by forty-three antivirus engines on VirusTotal (https://www.virustotal.com) [17] , and only the ones that passed all virus tests were considered as ''clean'' and kept to form the clean dataset. These clean applications do not impede on the smooth execution of the OS. Like Zhou and Jiang, we represent applications in the collected clean dataset using a vector of 130 binary values, each of which is associated with one of the 130 official Android permissions. A value 1 is assigned to a permission only if it is required or used by an application, otherwise, 0 is given instead.. The novelty and contributions of this work can be summarized as follows:. • We collected a new dataset that contains 1227 clean applications that were uploaded to third-party markets from August 2010 to October 2011.. • Beyond the current studies that focused on required permissions only, we also considered the used permissions.. • We utilized a hierarchical Biclustering method to initially analyze both clean and malware datasets. The obtained resulting figures provided a straightforward preview of the data distribution, from which we built up our model of mining a set of permissions rather than using individual permissions as the patterns.. • We proposed a contrast permission pattern mining algorithm to identify the interesting permission sets that can be used to distinguish applications from malicious to clean.. • Our demonstration of the proposed Contrast Permission Pattern Mining proved that both required and used permissions should be considered in late malware detection tasks.. The rest of the paper is organized as follows: Section 2 briefly reviews the concepts of the Android platform, its applications, the permission system and the current research work in malware detection. In Section 3, we present our initial analysis on the collected datasets using a statistical method and biclustering followed by the proposed contrast pattern mining algorithm. The experiments and the obtained results are then reported in Section 4 followed by a further discussion on findings. Finally, Section 5 concludes the entire paper together with our future work. Malware proliferation is rising exponentially and the attack vectors used by malware authors are getting more sophisticated. Current solutions proposed to thwart attacks by malicious applications will struggle to keep up with the increase of malware. The Android platform relies heavily on its permission system to control access to restricted system resources and private information stored on the smartphone. As demonstrated by [11, 33, 34] , permissions that are requested by applications during installation can be helpful in identifying permission patterns.. However, we identify the following problems in the existing literature:. Problem 1 What required permission patterns can be used to detect malicious applications? Problem 2 What used permission patterns can be used to detect malicious applications? Problem 3 Can we extract useful information by incorporating used permissions into the permission patterns? Problem 4 What method can we use to identify these expected permission patterns?. In this paper, we aim to extend the current statistical method used for identifying permission patterns in Android applications by applying pattern mining techniques to a set of clean and malicious applications in order to better understand the similarities and differences between these two datasets. With the help of visualization graphs, we establish possible connections between required permissions and used permissions in order to extrapolate emerging permission patterns that are frequently requested by applications. Then, we apply contrast set mining on the permissions patterns from clean and malicious applications to identify which patterns are most prevalent in each dataset. In this paper, we studied the Android permission system as the smartphone platform makes use of permissions to regulate access to system resources and users' private information. In order to understand and identify permission patterns, the existing work considers only those permissions that are declared in the AndroidManifest.xml files. We refer to those permissions as 'required' permissions. However, there is another permission check that takes place after an application has been installed and is executed by the smartphone OS. We refer to such permissions as 'used permissions'.. In our work, we considered the implications of incorporating used permissions in permission patterns and determined their usefulness in contrasting between clean and malicious applications. We proposed an efficient pattern mining method to generate a set of emerging contrast permission patterns for our clean and malware dataset. Based on our experimental results, we observed that there are several permissions that do not appear in the required permission sets but are present in the used permission sets. We found out that there is a discrepancy in the official documentation that allows for application with API level 3 or level to implicitly inherit certain permissions-although they are not declared in the The Android smartphone has gained in popularity in the past few years. Two main factors that contributed towards this change is the open-source nature of the platform and the flexibility provided to users and developers alike when downloading and developing Android applications, respectively. However, not all applications present on the application markets, both official and third-party, are clean. Previous work showed that malware authors take advantage of the Android permission system to entice users Table 13 Common used permission sets in both clean and malware datasets. into installing unsafe applications. As such, this study aims to understand required and used permissions by Android applications by applying data mining techniques to find emerging permission patterns that can be used to contrast clean and malicious applications.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O