# Introduction to Conformal Predictors Based on Fuzzy Logic Classifiers

A. Murari, Jesús Vega, D. Mazon, T. Courregelongue

► To cite this version:

A. Murari, Jesús Vega, D. Mazon, T. Courregelongue. Introduction to Conformal Predictors Based on Fuzzy Logic Classifiers. 8th International Conference on Artificial Intelligence Applications and Innovations (AIAI), Sep 2012, Halkidiki, Greece. pp.203-213, 10.1007/978-3-642-33412-2_21. hal-01523072

HAL Id: hal-01523072

https://hal.science/hal-01523072

Submitted on 16 May 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Introduction to Conformal Predictors based on Fuzzy Logic Classifiers

By A.Murari¹, J.Vega², D.Mazon³, T.Courregelongue⁴ and JET-EFDA

Contributors*

JET-EFDA, Culham Science Centre, OX14 3DB, Abingdon, UK

1 Consorzio RFX-Associazione EURATOM ENEA per la Fusione, I-35127 Padova, Italy

2 Asociación EURATOM-CIEMAT para Fusión, CIEMAT, Madrid, Spain

3 Association EURATOM-CEA, CEA Cadarache, 13108 Saint-Paul-lez-Durance, France

4 Arts et Métiers ParisTech Engineering College (ENSAM) 75013 Paris.

* See the Appendix of F. Romanelli et al., Proceedings of the 23rd IAEA Fusion Energy Conference 2010, Daejeon, Korea

**Abstract.** In this paper, an introduction to the main steps required to develop conformal predictors based on fuzzy logic classifiers is provided. The more delicate aspect is the definition of an appropriate nonconformity score, which has to be based on the membership function to preserve the specificities of Fuzzy Logic. Various examples are introduced, to describe the main properties of fuzzy logic based conformal predictors and to compare their performance with alternative approaches. The obtained results are quite promising, since conformal predictors based on fuzzy classifiers show the potential to outperform solutions based on the nearest neighbour in terms of ambiguity, robustness and interpretability

**Keywords:** Fuzzy Logic, Membership Function

## 1 Introduction

Various machine learning techniques, particularly kernel methods, have been recently developed to handle high dimensional data sets more efficiently [1,2,3,4]. Even if they constitute a significant step forward, these approaches have typically the drawback that they do not provide estimates of the reliability of their predictions. This problem has motivated the development of techniques to hedge the estimates of machine learning tools, by providing statistically sound indicators of the reliability of their results. A new family of techniques, called conformal predictors [5], has very recently started to be adopted in the scientific communities such as Nuclear Fusion research [6]. They have been formalised mainly for classification and they “hedge” their prediction by providing two parameters, credibility and confidence, which can be used to determine the level of trust that can be attributed to their estimates.

In this paper, the approach of conformal prediction is applied to the classification based on Fuzzy Logic methods [7,8]. A supervised fuzzy logic classifier is assumed and then the credibility and confidence estimators are calculated, using the
membership functions provided by the classifier. The proposed approach presents several advantages. First of all the classification can exploit the flexibility of the fuzzy logic formalism and therefore the final system can be optimised for the specific problem at hand (by selecting an appropriate membership function or the level of fuzziness for example). Moreover the fuzzy logic approach can be also tuned to improve the interpretability of the results [9,10], an aspect which can be of particular relevance in scientific investigations of complex systems.

## 2 Numerical Example of Conformal Predictors

As a reference, in this section a simple traditional conformal predictor based on the nearest neighbour technique is introduced. When a new example $z_n = (x_n, y_n)$ is available for classification, the nearest-neighbour method finds the $x_i$ closest to $x_n$ and uses its label $y_i$ as the prediction of $y_n$. At this point, it is natural to measure the nonconformity of the new example $z_n$ with respect to the one of the old examples $z_i$ by comparing $x$'s distances to old objects with the same label to its distance to old objects with a different label. For example, the nonconformity scores can be calculated as:

$$ \alpha_i = \frac{\min\{\{x_j - x_i\} : 1 \le j \le n \ \& \ j \ne i \ \& \ y_i = y_j\}}{\min\{\{x_j - x_i\} : 1 \le j \le n \ \& \ j \ne i \ \& \ y_i \ne y_j\}} \quad (1) \\ = \frac{\text{distance to } z_i\text{'s nearest neighbour with the same label}}{\text{distance to } z_i\text{'s nearest neighbour with a different label}} $$

As an intuitive, one dimensional example, the methodology can be applied to a set of points on a straight line, assuming that they belong to three different classes (A, B and C see fig. 1). The aim is to classify the new point Q with confidence and credibility. The points already classified are assumed to be: [0,1,4,6,12,14]. Table I summarizes the results of the required computations. Columns 2-4 give the nonconformity scores calculated according to equation (3). Column 2 (respectively 3 and 4) represents the p-values assuming that the object to classify belongs to class A (respectively B and C).

<figure><img src="image_2.png" /><figcaption>FIG. 1. Point Q must be classified as class</figcaption></figure>

are presented. The nonconformity scores are based on the membership function.

Therefore, the point Q is classified as class C with a credibility of 6/7 = 0.857 and a confidence of 1 - 1/7 = 0.857. In the next sections, various ways to build conformal predictors based on fuzzy logic systems
## 3 Introduction to Conformal Predictors based on Fuzzy Classifiers

The objective of this paper consists of start comparing new conformal predictors based on fuzzy systems with a conformal predictor based on the nearest neighbour [6]. In this section, mainly to exemplify the fundamental aspects of the proposed approach, it is assumed that a fuzzy classifier is available and that new objects have to be classified. For the purpose of this section, this fuzzy classifier can be taken without

further discussion, because the objective is to introduce the formulas for the non conformity scores to develop fuzzy based conformal predictors (and not to describe how the membership functions are determined). The membership functions are therefore assumed to be given and do not change when new elements have to be classified. For the

<table><caption>TABLE I. Example of figure 1</caption><thead><tr><th>Coordinate of object i</th><th>α<sub>i</sub> if y<sub>n</sub>=A</th><th>α<sub>i</sub> if y<sub>n</sub>=B</th><th>α<sub>i</sub> if y<sub>n</sub>=C</th></tr></thead><tbody><tr><td>0</td><td>1/4</td><td>1/4</td><td>1/4</td></tr><tr><td>1</td><td>1/3</td><td>1/3</td><td>1/3</td></tr><tr><td>4</td><td>2/3</td><td>2/3</td><td>2/3</td></tr><tr><td>6</td><td>2/5</td><td>2/5</td><td>2/5</td></tr><tr><td>12</td><td>2/1</td><td>2/1</td><td>1/6</td></tr><tr><td>14</td><td>2/1</td><td>2/1</td><td>1/8</td></tr><tr><td>13</td><td>12/1</td><td>-</td><td>-</td></tr><tr><td>13</td><td>-</td><td>7/1</td><td>-</td></tr><tr><td>13</td><td>-</td><td>-</td><td>1/7</td></tr><tr><td>p-value:</td><td>1/7</td><td>1/7</td><td>6/7</td></tr></tbody></table>

sake of simplicity, the one dimensional example, introduced in the previous section, is also considered as the reference to show how a conformal predictor can be based on a fuzzy classifier.

As mentioned, it is assumed that the fuzzy classifier has already been trained. The membership functions are represented as Gaussian-like functions (see equation 2), whose parameters are chosen so that their sum equals 1. The values of the chosen Gaussian functions are reported in table II and their graphical representation is given in figure 2. The partition matrix U, which is associated to these membership functions, contains the membership value of each object to each class and is organised with the columns representing the samples and the rows the clusters. For the points already considered in the example of the conformal classifier based on the nearest neighbour [0,1,4,6,12,14], the partition matrix is the one given by equation 3.

$$ \mu(x) = e^{\frac{-(x-c_i)^2}{2\sigma_i^2}}, \quad i = \{A, B, C\} \qquad (2) $$
<table><caption>TABLE II. Characteristics of the three Gaussian-like functions used in the supervised fuzzy classifier</caption><thead><tr><th>Cluster</th><th>Peak Centre</th><th>Left part standard deviation</th><th>Right part standard deviation</th></tr></thead><tbody><tr><td>A</td><td>0.5</td><td>1.909596376</td><td>1.909596376</td></tr><tr><td>B</td><td>5</td><td>1.909596376</td><td>4.491478641</td></tr><tr><td>C</td><td>13</td><td>2.757657944</td><td>2.757657944</td></tr></tbody></table>

$$U = \begin{bmatrix} 0.9663 & 0.9663 & 0.1864 & 0.0158 & 1.33e^{-8} & 1.40e^{-11} \\ 0.0325 & 0.1115 & 0.8719 & 0.9755 & 0.2969 & 0.1343 \\ 1.49e^{-5} & 0.0001 & 0.0049 & 0.0399 & 0.9364 & 0.9364 \end{bmatrix} \quad (3)$$

<figure><img src="image_3.png" /><figcaption>FIG. 2. Gaussian-like functions representing the membership functions resulting from the training of a hypothetical supervised fuzzy classifier.</figcaption></figure>

In order to obtain a conformal predictor, the credibility and the confidence of the various classifications must be calculated. Therefore we have to find an expression for the nonconformity score. In order to preserve the specificities of Fuzzy Logic, it has been decided to base the non conformity score on the membership functions. The main point is therefore to express the nonconformity score αik of the new element k

to class i in terms of its membership function μik. Two main approaches are conceivable: 1) the evaluation of the nonconformity score can be based only on the membership function of the new element to be classified or 2) the evaluation of the nonconformity score can be based on the comparison between the level of membership of the new element to its class and the membership of the other elements to the same class. These two different approaches are described in more detail in the next two subsections

## 3.1 Calculation of the nonconformity score based only on the membership function of the new element

The nonconformity score can be based only on the membership values of the new element to be classified and expressed in terms of the difference between the membership of the new elements to one class and its membership to the other classes. Basically the nonconformity score of a new object is low if its membership to a class is much higher than its memberships to all the other classes. In more detail, the nonconformity score of the new object should be low when the difference between the value of the membership to its class and the maximum of the membership values to the other classes is high. On the contrary, the higher the membership values of the new element to classify in the clusters different from its one, the higher its non conformity score should be. The previous considerations can be formalised using the
following equations, which constitute different ways of quantifying the difference between the membership of the new elements to its class and its membership to the others:

$$ \alpha_{ik} = 1 - \mu_{ik} + \max\{\mu_{lk} : 1 \le l \le c \ \& \ l \ne i\} \quad (4) $$

$$ \alpha_{ik} = [1 - \mu_{ik}] * \max\{\mu_{lk} : 1 \le l \le c \ \& \ l \ne i\} \quad (5) $$

$$ \alpha_{ik} = \frac{1 - \mu_{ik}}{1 - \max\{\mu_{lk} : 1 \le l \le c \ \& \ l \ne i\}} \quad (6) $$

where $\mu_{ik}$ is the membership value of the $k$-th element of the new element $i$ to its class $k$, $\alpha_{ik}$ is its nonconformity score, $c$ is the number of classes. The fact that equation (6) can diverge is not a problem for conformal predictors since this does not jeopardise the calculation of the p-values on which the entire approach is based. It is worth mentioning that two nonconformity measures similar to equations (4) (and (7) in the following subsection) have already been proposed for a neural network conformal predictor in [11].

## 3.2 Comparison with membership values of all the elements to the class of the new one

An alternative way to calculate the nonconformity score is based on the comparison of the membership value of the new element to its class and the membership values to the same class of all the other elements, which are not classified in the same class by the fuzzy classifier. Basically the nonconformity score is low if there is a big difference between the membership function of the new point to its assumed class and the membership functions to the same class of the points not in the same class. In more detail, the higher the difference between the membership function of the new element to its class and the highest level of the membership of the elements not classified in the same class, the lower the nonconformity score should be. Again these remarks are quantified by the following equations:

$$ \alpha_{ik} = \frac{\max\{\mu_{im} : 1 \le m \le N \ \text{such as } c_m \ne c_k\}}{\mu_{ik}} \quad (7) $$

$$ \alpha_{ik} = \frac{1 - \mu_{ik}}{1 - \max\{\mu_{im} : 1 \le m \le N \ \text{such as } c_m \ne c_k\}} \quad (8) $$
In these relations m is the index of the examples to classify (up to N) and the inequality $c_m \neq c_k$ indicates that only the examples with a different class as the one of the new element are considered. An alternative way of quantifying the difference between the membership of the new element to its class and the membership values of the other elements not in the same class can be based not on the max value but on the sum of the memberships of all the elements not in the same class. This can be expressed in mathematical terms by the next equations:

$$ \alpha_{ik} = \frac{\text{sum}\{\mu_{im} : 1 \le m \le N \text{ such as } c_m \neq c_k\}}{\mu_{ik}} \quad (9) $$

$$ \alpha_{ik} = [1 - \mu_{ik}] * \text{sum}\{\mu_{im} : 1 \le m \le N \text{ such as } c_m \neq c_k\} \quad (10) $$

Again the fact that equation (8) can in principle diverge is not a problem for conformal predictor since this does not jeopardise the calculation of the p-values on which the entire approach is based.

### 3.3 Comparison between the supervised fuzzy system and the nearest neighbour

The same analysis, as the one for the nearest neighbour case, has been performed to obtain the p-values, the credibility and the confidence on the basis of the fuzzy logic based nonconformity scores introduced in the previous two subsections. In order to compute the non conformity scores for the new points, we need to know the classification of the previous ones, which is given by the following Z matrix

$$ Z = \begin{bmatrix} 0 & 1 & 4 & 6 & 12 & 14 \\ A & A & B & B & C & C \end{bmatrix}^T \quad (11) $$

Using the membership function shown in fig.3, the p-values, the credibility and the confidence are computed for two new points, namely x=0.5 and x=2, to exemplify the differences between the fuzzy based and nearest neighbour techniques. According to the theory of conformal predictors [5], the new points are assumed to belong alternatively to each of the possible classes and their credibility and confidence are calculated. The new points are then classified in the class for which their credibility is the highest.

<figure><img src="image_4.png" /><figcaption>FIG. 3. Membership values for the tested points</figcaption></figure>
<table><caption>TABLE III. Results for x=0.5</caption><thead><tr><th>Expression</th><th>Nearest neighbour</th><th>Equation (6)</th><th>Equation (7)</th><th>Equation (8)</th></tr></thead><tbody><tr><th>P-values</th><td>0.8571 0.1429<br>0.1429</td><td>1 0.1429<br>0.1429</td><td>1 0.1429<br>0.1429</td><td>1 0.1429 0.1429</td></tr><tr><th>Credibility</th><td>0.8571</td><td>1</td><td>1</td><td>1</td></tr><tr><th>Confidence</th><td>0.8571</td><td>0.8571</td><td>0.8571</td><td>0.8571</td></tr></tbody></table>

<table><caption>TABLE IV. Results for x=2</caption><thead><tr><th>Equation (9)</th><th>Equation (10)</th><th>Equation (11)</th><th>Equation (12)</th></tr></thead><tbody><tr><td>1 0.2857 0.2857</td><td>1 0.2857 0.2857</td><td>1 0.2857 0.2857</td><td>1 0.2857 0.2857</td></tr><tr><td>1</td><td>1</td><td>1</td><td>1</td></tr><tr><td>0.7143</td><td>0.7143</td><td>0.7143</td><td>0.7143</td></tr></tbody></table>

<table><thead><tr><th>Expression</th><th>Nearest neighbour</th><th>Equation (6)</th><th>Equation (7)</th></tr></thead><tbody><tr><th>P-values</th><td>0.4286 0.1429 0.1429</td><td>0.7143 0.7143<br>0.7143</td><td>0.7143 0.7143<br>0.7143</td></tr><tr><th>Credibility</th><td>0.4286</td><td>0.7143</td><td>0.7143</td></tr><tr><th>Confidence</th><td>0.8571</td><td>0.2857</td><td>0.2857</td></tr></tbody></table>

<table><thead><tr><th>Equation (8)</th><th>Equation (9)</th><th>Equation (10)</th><th>Equation (11)</th></tr></thead><tbody><tr><td>0.7143 0.7143<br>0.7143</td><td>0.7143 0.5714<br>0.4286</td><td>0.7143 0.5714<br>0.4286</td><td>0.7143 0.5714<br>0.4286</td></tr><tr><td>0.7143</td><td>0.7143</td><td>0.7143</td><td>0.7143</td></tr><tr><td>0.2857</td><td>0.4286</td><td>0.4286</td><td>0.4286</td></tr></tbody></table>

<table><tbody><tr><td>(12)</td></tr><tr><td>0.7143 0.5714 0.4286</td></tr><tr><td>0.7143</td></tr><tr><td>0.4286</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>FIG.4. Initial data points repartition</figcaption></figure>

The results reported in tables III and IV show that the credibility values computed by the supervised fuzzy system are higher than the ones of the nearest neighbour. On the other hand, the confidence of the best formulas is no higher than the one of the nearest neighbour. It is noteworthy that for the example x=2, some of the formulas, used to derive the nonconformity score from the membership functions, give the same value of the maximum p-value for more than one class. These are ambiguities which should be avoided. This aspect and more in general a statistical analysis of the various alternatives to assess their performance are discussed in next section.
3 Statistical Analysis of the proposed classifiers

In this section, the results of an exhaustive statistical analysis of the fuzzy based
conformal predictors are reported. The intention is to compare the results obtained
with the various nonconformity measures reported in subsections 3.1 and 3.2. In this
perspective, we analyse the interval [-5 18.5] with a step of 0.005, namely a total of
4701 points is considered, and compare the different performance between the various
nonconformal predictors. This study relies on the example seen in fig. 2, where
membership functions are represented by Gaussians. To increase the accuracy of the
results, the algorithm has been implemented starting with 30 initial points, supposed
already correctly classified and spread randomly in the three classes. Figure 4 shows
these original points and how they are distributed.

<figure><img src="image_6.png" /><figcaption>FIG. 5. Credibility and confidence levels with 30 initials points using the nearest neighbour technique in red and the Gaussian based conformal predictors in black. The numerical</figcaption></figure>

The main
criteria to
assess the
performance of
the various
conformal
predictors
involve the
values of
credibility and
confidence. For
the objects
correctly
classified, it is

desirable to obtain high values of these two parameters to avoid unnecessary
uncertainties. In figure 5 the results obtained with equation (6) are compared with the
nearest neighbour. Both credibility and confidence vary more smoothly for the system
based on the fuzzy membership and the follow the shape of the classes. They increase
in the regions of the predefined classes as would be expected from an intuitive point
of view. On the other hand the nearest neighbour presents a confidence which can be
zero for points which are at the very centre of the classes. The discriminative power
of the fuzzy based conformal predictor is therefore higher than the one of the nearest
neighbour. Moreover the conformal predictor based on the fuzzy membership is more
conservative, since in general its confidence is lower than the one built on top of the
nearest neighbour. These qualities are shared by various conformal predictors based
on the Gaussian-type fuzzy memberships as shown in figure 6, where credibility and
confidence for the various alternatives are reported. The conformal predictors, based
on the nonconformity score expressed by equations (4) and (5), show an evolution
very similar to the one obtained from equation (6) just discussed. The others present
performance in general less satisfactory. On the other hand, the conformal predictors
based on equations (7) and (9) show a positive characteristic, namely an increase of
the confidence on the extreme left and right ends of the investigated intervals. In some
applications this could be positive since when the new point to classify is located
passed one of the two extreme classes (the leftmost or rightmost), it can be sensible to consider low the ambiguity of its classification.

<figure><img src="image_7.png" /><figcaption>FIG.6. Visual comparison between the computations of the Gaussian based conformal predictors using relations (4) to (10)</figcaption></figure>

A summary of the relative merits of the various fuzzy based conformal predictors is reported in tables V and VI. To make the interpretation of the results easier, the investigation has been particularised for two types of objects, the ones inside and the

ones outside the original classes defined in figure 1. The average confidence and credibility have been then calculated for all the examples falling in each of the category. The fact that equations (5) and (6) provide the best performance is confirmed.

Indeed they present the highest level of credibility and confidence once the results are averaged over the outside and inside class analyses. The choice between the two can

<table><caption>TABLE.V. Inside class analysis</caption><thead><tr><th>Nonconformity expression</th><th>Credibility mean</th><th>Confidence mean</th><th>Nb. of ambiguities</th></tr></thead><tbody><tr><td>Nearest neighbour</td><td>0.4801</td><td>0.9668</td><td>93</td></tr><tr><td>Expression (6)</td><td>0.8529</td><td>0.5562</td><td>0</td></tr><tr><td>Expression (7)</td><td>0.8826</td><td>0.5533</td><td>0</td></tr><tr><td><strong>Expression (8)</strong></td><td><strong>0.8903</strong></td><td><strong>0.7022</strong></td><td><strong>0</strong></td></tr><tr><td>Expression (9)</td><td>0.8930</td><td>0.3493</td><td>0</td></tr><tr><td>Expression (10)</td><td>0.8930</td><td>0.3226</td><td>0</td></tr><tr><td>Expression (11)</td><td>0.8930</td><td>0.3416</td><td>0</td></tr><tr><td>Expression (12)</td><td>0.8930</td><td>0.3226</td><td>0</td></tr></tbody></table>

<table><caption>TABLE.VI. Outside class analysis</caption><thead><tr><th>Nonconformity expression</th><th>Credibility mean</th><th>Confidence mean</th><th>Nb. of ambiguities</th></tr></thead><tbody><tr><td>Nearest neighbour</td><td>0.0625</td><td>0.9677</td><td>3191</td></tr><tr><td>Expression (4)</td><td>0.6948</td><td>0.3368</td><td>2906</td></tr><tr><td><strong>Expression (5)</strong></td><td><strong>0.7487</strong></td><td><strong>0.3367</strong></td><td><strong>1692</strong></td></tr><tr><td>Expression (6)</td><td>0.6846</td><td>0.3334</td><td>3271</td></tr><tr><td>Expression (7)</td><td>0.6809</td><td>0.5031</td><td>1703</td></tr><tr><td>Expression (8)</td><td>0.6838</td><td>0.3226</td><td>3428</td></tr><tr><td>Expression (9)</td><td>0.6838</td><td>0.4944</td><td>1766</td></tr><tr><td>Expression (10)</td><td>0.6838</td><td>0.3226</td><td>3428</td></tr></tbody></table>

depend on the application. On the other hand, as already mentioned, if the maximum
p-value for one sample appears in more than one class, there is an ambiguity and the algorithm is not able to classify the sample. Nonconformity scores which present less ambiguous cases are to be preferred because they have more refined discrimination capability and in this respect the conformal predictor based on equation (5) clearly outperforms the one based on equation (6).

# 5 Interpretation of the results and future developments

The results reported in the last two sections indicate that, provided the appropriate equation for the non conformity score is chosen, fuzzy logic classifiers can be a good basis on which to build conformal predictors. In general the obtained performances are quite good. Compared to a simple conformal predictor based on the nearest neighbour, the conformal predictors based on fuzzy classifiers seem to provide results more in agreement with intuition. Credibility is higher for the new objects which fall within the original classes and equal or lower outside (except for the cases using equations (8) and (9) for the nonconformity score which do no perform any better than a conformal predictor based on the nearest neighbour). The fuzzy logic predictors provide a much more discriminating confidence parameter, which is typically lower than the one of the nearest neighbour in the delicate regions at the borders between the two classes. Moreover, both confidence and credibility vary much more smoothly over the input space and this at least increases the interpretability of the results quite a lot.

# Acknowledgments

This work, supported by the European Communities under the contract of Association between EURATOM/ENEA/CIEMAT and CEA, was carried out under the framework of the European Fusion Development Agreement. The views and opinions expressed herein do not necessarily reflect those of the European Commission.

# References

1. V.N.Vapnik "The nature of statistical learning theory" Springer, New York, 2000 Second edition

2. S.Theodoridis and K.Koutroumbas "Pattern recognition" Elsevier, Academic Press, Canada 2009

3. C.M.Bishop "Pattern recognition and Machine learning" Springer Science+Bussiness Media Singapore 2006

4. R.O.Duda, P.E. Hart, D.G.Stork "Pattern Identification" Wiley-Interscience; 2 edition (October 2000)

5. V.Vovk, A.Gammerman and G.Shafer "Algorithmic learning in a random world" Springer, New York, (2005)

6. J. Vega, A. Murari, A. Pereira, S. González, and I. Pastor, Review Scientific Instruments 81, 10E118 (2010)

7. T.J.Ross "Fuzzy Logic with Engineering Applications" John Wiley and Sons, Ltd, Singapore, 2004

8. Lambrou, H. Papadopoulos and A. Gammerman. Reliable Confidence Measures for Medical Diagnosis with Evolutionary Algorithms. IEEE Transactions on Information Technology in Biomedicine 15(1): 93-99. IEEE, 2011.

9. Riid and E. Ruistern, "Transparent Fuzzy Systems in Modelling and Control" in J. Casillas, O. Cordon, F. Herrera, L. Magdalena (editors) Interpretability Issues in Fuzzy Modeling, Springer, pp. 452-476, 2003.

10. Detlef Nauck and Rudolf Kruse. "How the learning of rule weights affects the interpretability of fuzzy systems" In Proc. IEEE Int. Conf on Fuzzy Systems 1998, pages 1235-1240, Anchorage, May 1998

11. H. Papadopoulos, V. Vovk and A. Gammerman. Conformal Prediction with Neural Networks. In Proceedings of the 19th IEEE International Conference on Tools with Artificial Intelligence (ICTAI'07), Volume II, pp. 388-395. IEEE Computer Society, 2007.