## 2 Information Granulation and Classification Systems
### Proposed Approach
#### 2.2.2 Classification  
In the embedding space, namely the vector space spanned by the symbolic histograms of the form as in Equation (8), any classification system can be used. However, it is worth stressing the importance of feature selection whilst performing classification as per the following two (not mutually exclusive) rationales:  
1. there is no guarantee that all symbols in \(\mathcal{A}\) are indeed useful for the classification problem at hand
2. as introduced in Section 1, it is preferable to have a small, yet informative, alphabet in order to eventually ease an a-posteriori knowledge discovery phase (less symbols to be analysed by field-experts). For a given classification system \(\mathcal{C}\), let us consider its set of hyper-parameters \(\mathcal{H}\) to be tuned. Further, let \(\mathbf{w}\in\{0,1\}^{M}\) be an \(M\)-length binary vector in charge of selecting features (columns) from the instance matrices (i.e., symbols from \(\mathcal{A}\)) corresponding to non-zero elements. The tuplecan be optimised, for example, by means of a genetic algorithm [71] or other metaheuristics. In this work, two different classification systems are investigated. The former relies on non-linear \(\
u\)-Support Vector Machines (\(\
u\)-SVMs) [72], whereas the latter relies on 1-norm Support Vector Machines (\(\ell_{1}\)-SVMs) [73]. The rationale behind using the latter is as follows. \(\ell_{1}\)-SVMs, by minimising the 1-norm instead of the 2-norm of the separating hyperplane as in standard SVMs [72; 74; 75], return a solution (hyperplane coefficient vector) which is sparse: this allows to perform feature selection during training. For the sake of sketching a general framework, let us start our discussion from \(\
u\)-SVMs which do not natively return a sparse solution (i.e., do not natively perform any feature selection). The \(\
u\)-SVM is equipped with the radial basis function kernel:  
\[K(\mathbf{x},\mathbf{y})=\exp\{-\gamma\cdot D(\mathbf{x},\mathbf{y})^{2}\} \tag{10}\]  
where \(\mathbf{x},\mathbf{y}\) are two given patterns from the dataset at hand, \(D(\cdot,\cdot)\) is a suitable (dis)similarity measure and \(\gamma\) is the kernel shape parameter. The adopted dissimilarity measure is the weighted Euclidean distance:  
\[D(\mathbf{x},\mathbf{y},\mathbf{w})=\sqrt{\sum_{i=1}^{M}\mathbf{w}_{i}\cdot( \mathbf{x}_{i}-\mathbf{y}_{i})^{2}} \tag{11}\]  
where \(M\) is the number of features and \(\mathbf{w}_{i}\in\{0,1\}\) is the binary weight for the \(i\)th feature. Hence, it is possible to define \(\mathcal{H}=[\
u,\gamma]\) and the overall genetic code for \(\
u\)-SVM has the form  
\[[\
u,\ \gamma,\ \mathbf{w}] \tag{12}\]  
Each individual from the evolving population exploits \(\mathbf{D}_{\text{TR}}\) to train a \(\
u\)-SVM using the parameters written in its genetic code as follows:  
1. evaluates the kernel matrix using \(\mathbf{w}\) and \(\gamma\) (cf. Equation (10)-(11))
2. trains the \(\
u\)-SVM with regularisation parameter \(\
u\)  
The optimal hyperparameters set is the one that minimises the following objective function on \(\mathbf{D}_{\text{VAL}}\):  
\[F=\alpha\cdot(1-J)+(1-\alpha)\cdot\frac{|\mathbf{w}\
eq 0|}{|\mathbf{w}|}, \tag{13}\]  
where \(J\) is the (normalised (Originally, the informedness is defined as \(J=(\text{Sensitivity}+\text{Specificity}-1)\) and therefore is bounded in \([-1,+1]\). However, since the rightmost term in Equation (13) is bounded in \([0,1]\) and \(\alpha\in[0,1]\), we adopt a normalised version in order to ensure a fair combination.)) informedness (The informedness, by definition, takes into account binary problems. In case of multiclass problems, one can evaluate the informedness for each class by marking it as positive and then consider the average value amongst the problem-related classes.) [76; 77], defined as:  
\[J=\frac{(\text{Sensitivity}+\text{Specificity}-1)+1}{2}\in[0,1], \tag{14}\]  
whereas the rightmost term takes into account the sparsity of the feature selection vector \(\mathbf{w}\).