# A Study of Heuristic Evaluation Measures in Fuzzy Rule Induction

Ashraf A. Afifi

► To cite this version:

Ashraf A. Afifi. A Study of Heuristic Evaluation Measures in Fuzzy Rule Induction. 14th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), May 2018, Rhodes, Greece. pp.533-545, 10.1007/978-3-319-92007-8_45. hal-01821074

HAL Id: hal-01821074

https://inria.hal.science/hal-01821074

Submitted on 22 Jun 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Study of Heuristic Evaluation Measures in Fuzzy Rule Induction

Ashraf A. Afifi

Department of Engineering, Design and Mathematics, Faculty of Environment and Technology, University of the West of England, UK

Industrial Engineering Department, Faculty of Engineering, Zagazig University, Egypt

Ashraf.Afifi@uwe.ac.uk

**Abstract.** The rule induction process could be conceived as a search process, and hence an evaluation metric is needed to estimate the quality of rules found in the search space and to direct the search towards the best rule. The evaluation measure is the most influential inductive bias in rule learning. It is therefore important to investigate its influence on the induction process and to compare the behaviour of different evaluation measures. Many different evaluation measures have been used to score crisp rules. For some of these measures, fuzzy variations have been designed and used to score fuzzy rules. This paper examines the most popular crisp evaluation measures and demonstrates how they can be adapted into the fuzzy domain. The paper also studies the performance of these measures on a large number of data sets when used in a recently developed fuzzy rule induction algorithm. Results show that there are no universally applicable evaluation measures and the choice of the best measure depends on the type of the data set and the learning problem.

**Keywords:** fuzzy rule induction, heuristic evaluation measures, fuzzy sets.

## 1 Introduction

Rule induction has proven to be a valuable tool for description, classification and generalisation of data. A variety of methods exist for rule learning using crisp sets [1-4]. Fuzzy sets are a generalisation of crisp sets providing increased expressive power and comprehensibility. A number of fuzzy rule induction algorithms have been developed as an extension of crisp rule learners [5, 6]. Recently, Afifi [7] introduced a novel fuzzy rule induction method called FuzzyRULES (for Fuzzy RULE Extraction System). FuzzyRULES is based on the learning strategy used in the RULES-6 [8] crisp rule induction algorithm, previously developed by the author. FuzzyRULES preserves all the advantages of RULES-6 such as good comprehensibility and high classification accuracy. Moreover, it incorporates approximate reasoning offered by fuzzy representation, which allows better dealing with inconsistent, inexact, subjective, or noisy data.

Rule induction algorithms typically use an evaluation measure to score the performance of rules found during the learning process, and to select the best rules for further exploration. Therefore, the evaluation measure as a search heuristic is very important
as it determines to a large extent the performance of the learning algorithm. The selec-
tion of the most appropriate evaluation measure for any particular learning algorithm is
a critical and difficult task, which is usually carried out empirically. This paper exam-
ines the role of several well-known crisp evaluation measures and how they address the
specific requirements of the rule forming process. The paper also studies the perfor-
mance of the fuzzy variants of these measures when applied to fuzzy rule induction
algorithms. The aim is to identify the most appropriate measures to be applied in fuzzy
rule induction algorithms, especially in FuzzyRULES.

The layout of the paper is as follows. Section 2 reviews the FuzzyRULES algorithm. Section 3 introduces the evaluation measures, and demonstrates the adaptation of these measures to the fuzzy set domain. Section 4 gives an empirical evaluation of the different evaluation measures when used in FuzzyRULES. Section 5 concludes the paper and provides suggestions for future work.

## 2 FuzzyRULES

The recently developed FuzzyRULES fuzzy rule induction algorithm will be used in
this study to assess the performance of several heuristic evaluation measures.
FuzzyRULES is based on the RULES-6 crisp rule induction algorithm. RULES-6
broadly follows the approach of AQ-like [2] learning algorithms. It employs heuristic
search techniques with different learning biases and rule-space pruning strategies that
significantly reduce the proportion of the search space examined during the learning
process, resulting in substantial performance benefits.

FuzzyRULES follows the so-called *separate-and-conquer* or *covering* strategy of
rule induction algorithms. It creates the rule set one rule at a time. Each rule explains
(covers) a part of the training instances. After a rule is generated, the instances covered
by it are removed (separated) from the training data set before subsequent rules are
learned. The remaining instances are conquered using the same induction procedure
until all the instances are covered by at least one rule in the rule set. Figure 1 provides
a simplified description of the FuzzyRULES algorithm. A detailed description of the
algorithms can be found in [7].

## 3 Evaluation Measures

A typical objective of a rule induction algorithm is to find rules that optimise a rule
evaluation measure that takes both training accuracy and rule coverage into account so
that the rules learned are both accurate and general. Many different measures for eval-
uating and assigning a score to crisp rules have been proposed in the literature [9]. For
some of these measures, fuzzy variations have been designed and used to score fuzzy
rules [10]. In this section, the most commonly used rule evaluation measures are dis-
cussed and the adaptation of these measures into the fuzzy domain is demonstrated.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Overview of the FuzzyRULES algorithm.</figcaption></figure>

To derive fuzzy evaluation measures, the cardinality and α-cut operators [11] will be used to describe the instances matched by the rules. Let A => C denote a candidate rule R, where A is the rule antecedent (a conjunction of conditions) and C is the rule consequent (the value predicted for the goal attribute). The set of instances in the training set T that are covered by a rule R can be defined using the cardinality measure M(A), also referred to as the sigma count, as follows:

$$M(A) = \sum_{I \in T} \mu_{R}(I) \qquad (1)$$

where μ<sub>R</sub>(I) represents the degree of match of a particular instance I with the antecedent of a rule R. For each instance, this degree is computed by a fuzzy AND of the degree of matching between each attribute value in the instance and the corresponding fuzzy condition in the rule. The conventional definition of the fuzzy AND has been used as the minimum operator.

In practice, many of the instances can match a rule antecedent to a small degree, and the summation of all these small degrees of membership can give the false impression that the rule is good, which can undermine the reliability of the evaluation measure. To prevent such instances from being covered, a user-defined α-cut threshold can be applied to the instance memberships. The membership of an instance to a rule, μ<sub>R</sub>(I), is
defined to be zero when it lies below the α-cut threshold value. When the value of $\mu_R(I)$ is greater than or equal to the specified value of $\alpha$, rule $R$ is said to α-cover instance $I$. Using the α-cut operator, the set of instances in the training set $T$ that are α-covered by a rule $R$ can be expressed by redefining Equation (1) as follows:

$$M(A) = \sum_{I \in R_{\alpha}(T)} \mu_{R}(I) \qquad (2)$$

where $R_{\alpha}(T) = \{I \in T | \mu_{R}(I) \ge \alpha\}$ contains all instances in the training set $T$ that have a membership grade in $R$ greater than or equal to the specified value of $\alpha$.

The evaluation measures include different parameters that assess the coverage of the rules. For a particular rule $R$, the parameters $P$ and $N$ denote the total number of positive instances (instances belonging to the target class) and negative instances (instances not belonging to the target class) in the training set, $p(n)$ the number of positive (negative) instances covered by rule $R$, and $p'(n')$ the number of positive (negative) instances covered by rule $R'$, the predecessor (parent) of rule $R$.

## 3.1 Purity

This measure is utilised in GREEDY3 [12] and SWAP-1 [13] algorithms. It gives the ratio between the number of positive instances covered by a rule and the total number of covered instances. In the crisp case, the purity measure of a rule $R$ is given by:

$$P(R) = \frac{p}{p+n} \qquad (3)$$

The purity measure attains its optimal value when no negative instances are covered. Also, it does not aim to cover many positive instances. As a result, this metric tends to select very specific rules covering only a small number of instances. This is undesirable since rules covering few instances are unreliable, especially where there is noise in the data. The accuracy of these rules on the training data does not adequately reflect their true predictive accuracy on new test data.

The purity measure can be fuzzified using the cardinality and α-cut operators as follows:

$$P(R) = \frac{M(A \cap C)}{M(A)} = \frac{\sum_{I \in R_{\alpha}(P)} \mu_{R}(I)}{\sum_{I \in R_{\alpha}(T)} \mu_{R}(I)} \qquad (4)$$

where $M(A \cap C)$ is the fuzzification of the number of positive instances covered by a rule $R(p)$, $M(A)$ is the fuzzification of the number of instances covered by the rule $(n+p)$, and $R_{\alpha}(P) = \{I \in P | \mu_{R}(I) \ge \alpha\}$, $R_{\alpha}(T) = \{I \in T | \mu_{R}(I) \ge \alpha\}$ are the sets of positive instances and total instances that are α-covered by rule $R$, respectively.

## 3.2 Information Content

The information content function measures the amount of information contained in the classification of the covered instances. It is originally used in the PRISM inductive learner [14], and its crisp version is given by:
$$ IC(R) = -\log_2 \left( \frac{p}{p+n} \right) \qquad (5) $$

The best rule is the one that minimises Equation (5). The information content function is basically equivalent to the purity measure and its main advantage is that using a logarithmic scale tends to assign higher penalties to rules with low coverage.

The fuzzy version of the information content function can be expressed as:

$$ IC(R) = -\log_2 \left( \frac{M(A \cap C)}{M(A)} \right) = -\log_2 \left( \frac{\sum_{I \in R_a(P)} \mu_R(I)}{\sum_{I \in R_a(T)} \mu_R(I)} \right) \qquad (6) $$

## 3.3 Entropy

The entropy measure is adopted in the original version of the CN2 algorithm [15]. It is the weighted average of the information content of the positive and negative classes. The entropy measure of a rule R is given by (the lower the entropy, the better the rule):

$$ E(R) = - \left( \frac{p}{p+n} \log_2 \frac{p}{p+n} + \frac{n}{p+n} \log_2 \frac{n}{p+n} \right) \qquad (7) $$

The entropy measure suffers from similar deficiencies as purity and information content measures. Also, it does not consider whether the majority of instances are positive or not. For example, a rule that covers 100 positive and 10 negative instances is considered of equal quality to another rule that covers 10 positive and 100 negative instances. To assign higher scores to rules with higher coverage, the following function is adopted [10].

$$ E(R) = - \frac{n}{p} \left( \frac{p}{p+n} \log_2 \frac{p}{p+n} + \frac{n}{p+n} \log_2 \frac{n}{p+n} \right) \qquad (8) $$

Equation (8) can be adapted to the domain of fuzzy sets as follows:

$$ E(R) = - \frac{M(A \setminus C)}{M(A \cap C)} \left( \frac{M(A \cap C)}{M(A)} \log_2 \frac{M(A \cap C)}{M(A)} + \frac{M(A \setminus C)}{M(A)} \log_2 \frac{M(A \setminus C)}{M(A)} \right) \qquad (9) $$ $$ = - \frac{\sum_{I \in R_a(N)} \mu_R(I)}{\sum_{I \in R_a(P)} \mu_R(I)} \left( \frac{\sum_{I \in R_a(P)} \mu_R(I)}{\sum_{I \in R_a(T)} \mu_R(I)} \log_2 \frac{\sum_{I \in R_a(P)} \mu_R(I)}{\sum_{I \in R_a(T)} \mu_R(I)} + \frac{\sum_{I \in R_a(N)} \mu_R(I)}{\sum_{I \in R_a(T)} \mu_R(I)} \log_2 \frac{\sum_{I \in R_a(N)} \mu_R(I)}{\sum_{I \in R_a(T)} \mu_R(I)} \right) $$

where $M(A \setminus C)$ is the fuzzification of the number of negative instances covered by rule R (n), and $R_a(N) = \{I \in N | \mu_R(I) \ge \alpha\}$ is the set of negative instances that are $\alpha$-covered by rule R.

## 3.4 Information Gain

The information gain measure is originally used in the FOIL relational learner [16]. It computes the reduction of the information content in a rule when a new condition is added to it. More precisely, it calculates the difference in the information content of the current rule R and its parent R', multiplied by the number of covered positive instances as a bias for generality. In the crisp case, it is given by:
$$IG(R) = p \cdot (IC(R') - IC(R)) = p \cdot \left( -\log_2\left(\frac{p'}{p'+n'}\right) + \log_2\left(\frac{p}{p+n}\right) \right) \quad (10)$$

The goal is to maximise this measure.

The fuzzy information gain can be computed as follows:

$$IG(R) = p \cdot \left( \log_2\left( \frac{M(A' \cap C)}{M(A')} \right) - \log_2\left( \frac{M(A \cap C)}{M(A)} \right) \right) \quad (11)$$

$$= p \cdot \left( \log_2\left( \frac{\sum_{I \in R_{\alpha}(P)} \mu_R(I)}{\sum_{I \in R_{\alpha}(T)} \mu_R(I)} \right) - \log_2\left( \frac{\sum_{I \in R_{\alpha}(P)} \mu_R(I)}{\sum_{I \in R_{\alpha}(T)} \mu_R(I)} \right) \right)$$

where $M(A' \cap C)$ is the fuzzification of the number of positive instances covered by rule $R'$ ($p'$), $M(A)$ is the fuzzification of the number of instances covered by the rule ($p'+n'$), and $R'_{\alpha}(P) = \{I \in P | \mu_R'(I) \ge \alpha\}$, $R'_{\alpha}(T) = \{I \in T | \mu_R'(I) \ge \alpha\}$ are the sets of positive instances and total instances that are $\alpha$-covered by rule $R'$, respectively.

## 3.5 Accuracy

This metric is employed in I-REP [17]. It is the proportion of positive instances that are covered (p) and negative instances that are not covered (N - n), in all instances (P + N). In the crisp case, the accuracy measure for a rule R is given by:

$$A(R) = \frac{p + (N - n)}{P + N} \quad (12)$$

The accuracy measure favours high coverage and scores rules in the range [-$\infty$, $\infty$], with higher scores given to better evaluations. The problem of the accuracy measure is that this measure sometimes does not lead to a satisfactory behaviour. For example, it favours a rule that covers 2000 positive and 1000 negative instances over another rule that covers 1000 positive and only 1 negative instance.

A fuzzy variant of the accuracy measure can be defined as:

$$A(R) = \frac{M(A \cap C) + (N - M(A \cap C))}{P + N} = \frac{\sum_{I \in R_{\alpha}(P)} \mu_R(I) + (N - \sum_{I \in R_{\alpha}(N)} \mu_R(I))}{P + N} \quad (13)$$

## 3.6 Laplace

The Laplace estimate penalises rules with low coverage and it is used in CN2 [1] and several other algorithms due to its simplicity and efficiency. The Laplace estimate of a rule R is given by:

$$L(R) = \frac{p+1}{p+n+k} \quad (14)$$

where k is the number of classes. Rule induction algorithms learn multi-class concepts by learning one class at a time. Thus, k is always 2.
The Laplace estimate has the desirable property of taking into account both accuracy and coverage when estimating rule accuracy. However, it has a problem when learning rules with less than 50% training accuracy. The Laplace estimate does not satisfy the requirement that the rule quality value should rise with increased coverage. Another problem is that the Laplace accuracy estimate is often unrealistic, especially in multi-class decision problems [18]. This occurred because of the assumption that underlies Laplace accuracy estimate, namely, that the a priori distribution is uniform.

A fuzzy variant of the Laplace estimate can be given by:

$$L(R) = \frac{M(A \cap C) + 1}{M(A) + 2} = \frac{\sum_{I \in R_a(P)} \mu_R(I) + 1}{\sum_{I \in R_a(T)} \mu_R(I) + 2} \quad (15)$$

## 3.7 m-estimate

The m-estimate [18] is a more general version of the Laplace measure and it is defined as follows:

$$M(R) = \frac{p + m * P/(P + N)}{p + n + m} \quad (16)$$

where $P/(P+N)$ is the a priori probability of the target class and $m$ is a domain dependent parameter. The value of $m$ is related to the amount of noise in the domain. $m$ can be small if little noise is expected and should increase if the amount of noise is substantial. $m$ is usually set to $k$, where $k$ is the number of classes.

The m-estimate is a good choice because it is based on strong theoretical foundations and it meets the requirements of a good evaluation function. The Laplace measure can be obtained from the m-estimate when $m$ is set to 2 and the a priori probability is assumed to be uniform. It should be noted that the m-estimate generalises the Laplace measure so that rules that cover no instances will be evaluated with the a priori probability instead of the value 1/2, which is more flexible and convenient.

The fuzzy variant of the m-estimate can be defined as follows:

$$M(R) = \frac{M(A \cap C) + k * P/(P + N)}{M(A) + k} = \frac{\sum_{I \in R_a(P)} \mu_R(I) + k * P/(P + N)}{\sum_{I \in R_a(T)} \mu_R(I) + k} \quad (17)$$

## 3.8 H Measure

This measure is applied in RULES-5 [3]. It is the product of the accuracy (first part) and generality (second part) of a rule.

$$H(R) = \sqrt{\frac{p+n}{P+N}} \left( 2 - 2\sqrt{\frac{p}{p+n} \cdot \frac{P}{P+N}} - 2\sqrt{\left(1 - \frac{p}{p+n}\right)\left(1 - \frac{P}{P+N}\right)} \right) \quad (18)$$

The H measure can be fuzzified as follows:
$$H(R) = \sqrt{\frac{M(A)}{P+N}} \left( 2 - 2\sqrt{\frac{M(A \cap C)}{M(A)} \cdot \frac{P}{P+N}} - 2\sqrt{\left(1 - \frac{M(A \cap C)}{M(A)}\right)\left(1 - \frac{P}{P+N}\right)} \right) \quad (19)$$

$$= \sqrt{\frac{\sum_{I \in R_u(T)} \mu_R(I)}{P+N}} \left( 2 - 2\sqrt{\frac{\sum_{I \in R_u(T)} \mu_R(I) \cdot \frac{P}{P+N}}{\sum_{I \in R_u(T)} \mu_R(I) \cdot P+N}} - 2\sqrt{\left(1 - \frac{\sum_{I \in R_u(T)} \mu_R(I)}{\sum_{I \in R_u(T)} \mu_R(I)}\right)\left(1 - \frac{P}{P+N}\right)} \right)$$

## 3.9 AQ18 Measure

This measure was introduced in AQ18 [2] and it aims to combine completeness, consistency and consistency gain. This is done by allowing changes in the relative importance of completeness (first part) or consistency gain (second part) and also by changing the value of the parameter $w \in [0, 1]$.

$$Q(R) = \left(\frac{p}{P}\right)^w \left(\left(\frac{p}{p+n} - \frac{P}{P+N}\right) \cdot \frac{P+N}{N}\right)^{(1-w)} \quad (20)$$

The fuzzy version of the AQ18 measure can be expressed as:

$$Q(R) = \left( \frac{M(A \cap C)}{P} \right)^w \left( \left( \frac{M(A \cap C)}{M(A)} - \frac{P}{P+N} \right) \cdot \frac{P+N}{N} \right)^{(1-w)} \quad (21)$$

$$= \left( \frac{\sum_{I \in R_u(P)} \mu_R(I)}{P} \right)^w \left( \left( \frac{\sum_{I \in R_u(P)} \mu_R(I)}{\sum_{I \in R_u(T)} \mu_R(I)} - \frac{P}{P+N} \right) \cdot \frac{P+N}{N} \right)^{(1-w)}$$

The aforementioned evaluation measures have been developed alongside specific algorithms and their performances have not been compared to other metrics when applied to the same rule induction algorithm. As part of this work, the performance of these measures when used in the FuzzyRULES algorithm is evaluated experimentally in the following section.

## 4 Experimental Evaluation

This section presents an empirical evaluation of the nine fuzzy heuristic evaluation measures discussed in the previous section. Experimental tests were carried out using the FuzzyRULES algorithm, without applying any pruning procedures because such post processing could mask the real effect of using one or another heuristic. Three criteria were used to evaluate the results, namely, classification accuracy, rule set complexity, and the number of rules examined to obtain the rule set. The complexity of a rule set is measured by the total number of conditions in that rule set. Tests were performed on 30 data sets from the UCI repository of machine learning databases [19]. The selected data sets either have only continuous attributes or a mixture of nominal and continuous attributes. The hold-out approach was used to partition the data into training and test data [20]. FuzzyRULES was run with the default settings, and the default value of 0.5 was used for the parameter $w$ of the AQ18 measure.
Table 1 shows the classification accuracies obtained with each of the nine evaluation measures. Bold numbers indicate the best performance for a specific data set among the tested measures. In the last row, the total performance over all data sets is shown. A number of results are notable. First, the accuracy obtained with the *m*-estimate measure over all the data sets was in total higher than that produced with all the other measures. Also, the *m*-estimate measure produced the best classification results for 14 out of the 30 data sets. The Laplace and information content measures had a very similar total accuracy, but the Laplace measure had the best results overall for a higher number of data sets. The entropy and purity measures yielded the next best total accuracy, followed by the *H* and AQ18 measures. The accuracy and information gain measures obtained the lowest total accuracy. Second, every evaluation measure obtained the best results overall for some data sets, and the best performing measures had a notable worse performance for some other data sets. This suggests that there will not be a universal solution to all learning problems. The best result is obtained by choosing the evaluation measure that best fits the requirements of a particular learning problem.

<table><caption>**Table 1.** Classification accuracies for each evaluation measure when used in FuzzyRULES.</caption><thead><tr><th>Data Set Name</th><th>Purity</th><th>Entropy</th><th>Information Content</th><th>Information Gain</th><th>Accuracy</th><th>Laplace</th><th>m-estimate</th><th>H Measure</th><th>AQ18</th></tr></thead><tbody><tr><td>Abalone</td><td>23.0</td><td>24.2</td><td>23.0</td><td>15.8</td><td>21.6</td><td>22.9</td><td>23.9</td><td><strong>25.0</strong></td><td>18.8</td></tr><tr><td>Adult</td><td><strong>82.0</strong></td><td>79.4</td><td>81.9</td><td>79.8</td><td>77.1</td><td>81.9</td><td>81.8</td><td>79.4</td><td>80.1</td></tr><tr><td>Anneal</td><td>99.0</td><td>87.3</td><td><strong>99.3</strong></td><td>82.3</td><td>84.7</td><td>97.7</td><td>97.7</td><td>96.3</td><td>84.7</td></tr><tr><td>Australian</td><td>80.4</td><td><strong>87.0</strong></td><td>83.0</td><td>86.5</td><td>86.5</td><td>85.7</td><td>85.2</td><td>80.3</td><td>86.5</td></tr><tr><td>Auto</td><td><strong>72.5</strong></td><td>65.2</td><td>68.1</td><td>62.5</td><td>59.4</td><td>59.4</td><td>65.2</td><td>62.9</td><td>44.9</td></tr><tr><td>Balance-scale</td><td>64.6</td><td>64.6</td><td>64.6</td><td>56.9</td><td>66.0</td><td>64.6</td><td><strong>69.6</strong></td><td>65.0</td><td>66.0</td></tr><tr><td>Breast</td><td><strong>95.3</strong></td><td>94.8</td><td>94.0</td><td>92.7</td><td>92.7</td><td>92.3</td><td>94.3</td><td>91.9</td><td>92.7</td></tr><tr><td>Cleve</td><td>75.2</td><td>74.3</td><td>75.2</td><td>74.3</td><td>79.2</td><td><strong>82.2</strong></td><td><strong>82.2</strong></td><td>77.3</td><td>72.3</td></tr><tr><td>Crx</td><td>74.0</td><td>81.0</td><td>75.0</td><td><strong>83.0</strong></td><td><strong>83.0</strong></td><td>74.5</td><td>81.0</td><td>80.0</td><td><strong>83.0</strong></td></tr><tr><td>Diabetes</td><td>64.5</td><td>75.0</td><td>64.5</td><td>72.3</td><td>69.5</td><td>64.5</td><td><strong>75.2</strong></td><td>68.2</td><td>74.6</td></tr><tr><td>German</td><td>70.3</td><td>71.2</td><td>72.1</td><td>69.4</td><td>66.0</td><td>70.0</td><td><strong>75.6</strong></td><td>73.1</td><td>67.9</td></tr><tr><td>German-organisation</td><td>68.5</td><td>71.8</td><td>66.7</td><td>70.9</td><td>66.0</td><td>69.4</td><td><strong>76.4</strong></td><td>67.0</td><td>71.5</td></tr><tr><td>Glass</td><td>52.8</td><td>51.4</td><td>58.3</td><td>49.4</td><td>58.3</td><td>50.0</td><td><strong>64.7</strong></td><td>63.9</td><td>58.3</td></tr><tr><td>Glass2</td><td>78.2</td><td><strong>83.6</strong></td><td>76.4</td><td>76.4</td><td>76.4</td><td><strong>83.6</strong></td><td><strong>83.6</strong></td><td>72.7</td><td>81.5</td></tr><tr><td>Heart-disease</td><td>73.3</td><td>67.8</td><td>76.7</td><td>73.3</td><td>72.2</td><td><strong>80.0</strong></td><td>78.9</td><td>73.3</td><td>75.6</td></tr><tr><td>Heart-Hungarian</td><td>73.5</td><td>76.5</td><td><strong>98.5</strong></td><td>78.6</td><td>74.5</td><td>76.5</td><td>78.6</td><td>72.4</td><td><strong>75.5</strong></td></tr><tr><td>Hepatitis</td><td>80.8</td><td>82.7</td><td>78.8</td><td>73.1</td><td><strong>88.5</strong></td><td>86.5</td><td>84.8</td><td>82.7</td><td><strong>88.5</strong></td></tr><tr><td>Horse-colic</td><td>70.6</td><td>82.4</td><td>70.6</td><td>65.0</td><td>79.4</td><td>76.5</td><td><strong>83.5</strong></td><td>76.2</td><td>80.9</td></tr><tr><td>Hypothyroid</td><td>96.0</td><td>98.1</td><td>96.0</td><td><strong>98.8</strong></td><td>97.0</td><td>98.4</td><td>97.9</td><td>94.8</td><td>98.7</td></tr><tr><td>Ionosphere</td><td>93.2</td><td>88.5</td><td>92.3</td><td>87.8</td><td>95.3</td><td>92.3</td><td>92.3</td><td>88.9</td><td><strong>95.7</strong></td></tr><tr><td>Iris</td><td><strong>96.0</strong></td><td>94.0</td><td><strong>96.0</strong></td><td>94.0</td><td>94.0</td><td><strong>96.0</strong></td><td><strong>96.0</strong></td><td>96.0</td><td><strong>96.0</strong></td></tr><tr><td>Letter</td><td>80.5</td><td>76.4</td><td>82.2</td><td>53.2</td><td>71.1</td><td><strong>82.1</strong></td><td>66.1</td><td>60.7</td><td>58.3</td></tr><tr><td>Lymphography</td><td>74.0</td><td><strong>84.0</strong></td><td>78.0</td><td>56.0</td><td>68.0</td><td>78.0</td><td>76.0</td><td>78.0</td><td>64.0</td></tr><tr><td>Satimage</td><td>81.2</td><td>80.6</td><td>83.2</td><td>60.4</td><td>76.3</td><td><strong>86.0</strong></td><td>84.0</td><td>81.8</td><td>67.1</td></tr><tr><td>Segment</td><td>80.9</td><td>82.1</td><td>87.7</td><td>77.4</td><td>80.8</td><td>84.8</td><td><strong>89.5</strong></td><td>88.4</td><td>79.0</td></tr><tr><td>Shuttle</td><td>97.6</td><td>95.4</td><td>95.9</td><td>94.8</td><td>94.8</td><td><strong>98.4</strong></td><td><strong>98.4</strong></td><td>95.9</td><td>94.8</td></tr><tr><td>Sick-euthyroid</td><td>92.0</td><td>90.4</td><td>91.8</td><td>90.9</td><td>9.6</td><td>91.9</td><td><strong>96.4</strong></td><td>91.6</td><td>90.4</td></tr><tr><td>Sonar</td><td>70.0</td><td>70.0</td><td><strong>80.0</strong></td><td>74.3</td><td>77.1</td><td>80.0</td><td><strong>80.0</strong></td><td>68.6</td><td>61.4</td></tr><tr><td>Tokyo</td><td>91.5</td><td>91.3</td><td>92.3</td><td>91.9</td><td>87.3</td><td><strong>92.5</strong></td><td>92.1</td><td>91.4</td><td>90.8</td></tr><tr><td>Vehicle</td><td>63.1</td><td>48.3</td><td>63.5</td><td>51.7</td><td>61.3</td><td>64.5</td><td><strong>66.0</strong></td><td>61.7</td><td>53.2</td></tr><tr><td><strong>Total</strong></td><td><strong>2314.4</strong></td><td><strong>2319.2</strong></td><td><strong>2365.6</strong></td><td><strong>2193.2</strong></td><td><strong>2213.8</strong></td><td><strong>2363.0</strong></td><td><strong>2416.8</strong></td><td><strong>2305.2</strong></td><td><strong>2252.8</strong></td></tr></tbody></table>
Table 2 shows the total number of conditions generated with each of the evaluation measures. Bold numbers indicate the smallest number of conditions per rule set for a particular data set. The information gain measure produced significantly fewer conditions in total than the other measures. In addition, it obtained the smallest rule sets for 29 out of the 30 data sets. However, the over-general rule sets created resulted in a deteriorated classification accuracy for most data sets. The AQ18 measure also found simple rule sets but again had poor accuracy performance. The accuracy, *m*-estimate, *H*, entropy and Laplace measures created general rules without sacrificing accuracy for most data sets. These measures prefer rules covering more instances, which often helps the learning algorithm to prevent overfitting. The purity and information content measures generated the most complex rule sets, which might be an indication of over-fitting. This is most likely due to that these measures prefer more consistent rules and do not favour rules with high coverage.

Table 3 presents the size of the search space examined for each evaluation measure. The bold numbers indicate the smallest search per data set. The number of rules explored during the search process is often directly linked to the rule set complexity. Therefore, the fewer and more general rules created by the information gain measure

<table><caption>**Table 2.** Total number of conditions for each evaluation measure when used in FuzzyRULES.</caption><thead><tr><th>Data Set Name</th><th>Purity</th><th>Entropy</th><th>Information Content</th><th>Information Gain</th><th>Accuracy</th><th>Laplace</th><th>m-estimate</th><th>H Measure</th><th>AQ18</th></tr></thead><tbody><tr><td>Abalone</td><td>105</td><td>56</td><td>103</td><td><b>12</b></td><td>325</td><td>108</td><td>53</td><td>36</td><td>15</td></tr><tr><td>Adult</td><td>814</td><td>137</td><td>1134</td><td>8</td><td><b>7</b></td><td>286</td><td>128</td><td>179</td><td>30</td></tr><tr><td>Anneal</td><td>60</td><td>24</td><td>47</td><td><b>6</b></td><td>33</td><td>49</td><td>39</td><td>36</td><td>63</td></tr><tr><td>Australian</td><td>222</td><td>6</td><td>207</td><td><b>2</b></td><td><b>2</b></td><td>126</td><td>118</td><td>52</td><td><b>2</b></td></tr><tr><td>Auto</td><td>69</td><td>59</td><td>62</td><td><b>7</b></td><td>32</td><td>62</td><td>48</td><td>67</td><td>39</td></tr><tr><td>Balance-scale</td><td>33</td><td>18</td><td>33</td><td><b>2</b></td><td>11</td><td>33</td><td>29</td><td>30</td><td>14</td></tr><tr><td>Breast</td><td>121</td><td>9</td><td>92</td><td><b>3</b></td><td>12</td><td>30</td><td>27</td><td>22</td><td>12</td></tr><tr><td>Cleve</td><td>103</td><td>30</td><td>78</td><td><b>5</b></td><td>10</td><td>52</td><td>52</td><td>47</td><td>17</td></tr><tr><td>Crx</td><td>173</td><td>17</td><td>159</td><td><b>5</b></td><td>13</td><td>141</td><td>151</td><td>50</td><td>13</td></tr><tr><td>Diabetes</td><td>50</td><td>34</td><td>48</td><td><b>6</b></td><td>9</td><td>39</td><td>32</td><td>26</td><td>9</td></tr><tr><td>German</td><td>402</td><td>125</td><td>352</td><td><b>7</b></td><td>73</td><td>195</td><td>173</td><td>341</td><td>41</td></tr><tr><td>German-organisation</td><td>536</td><td>338</td><td>471</td><td><b>4</b></td><td>175</td><td>222</td><td>219</td><td>312</td><td>45</td></tr><tr><td>Glass</td><td>142</td><td>79</td><td>124</td><td><b>7</b></td><td>34</td><td>104</td><td>60</td><td>96</td><td>14</td></tr><tr><td>Glass2</td><td>59</td><td>11</td><td>48</td><td><b>4</b></td><td>6</td><td>20</td><td>20</td><td>27</td><td>10</td></tr><tr><td>Heart-disease</td><td>136</td><td>16</td><td>63</td><td><b>3</b></td><td>6</td><td>106</td><td>60</td><td>47</td><td>15</td></tr><tr><td>Heart-Hungarian</td><td>82</td><td>4</td><td>72</td><td><b>2</b></td><td>12</td><td>42</td><td>38</td><td>34</td><td>16</td></tr><tr><td>Hepatitis</td><td>65</td><td>13</td><td>46</td><td><b>4</b></td><td>12</td><td>27</td><td>24</td><td>51</td><td>24</td></tr><tr><td>Horse-colic</td><td>135</td><td>16</td><td>104</td><td><b>3</b></td><td>8</td><td>121</td><td>107</td><td>134</td><td>13</td></tr><tr><td>Hypothyroid</td><td>89</td><td>20</td><td>61</td><td><b>5</b></td><td>25</td><td>52</td><td>49</td><td>7</td><td>18</td></tr><tr><td>Ionosphere</td><td>43</td><td>9</td><td>34</td><td><b>2</b></td><td>13</td><td>32</td><td>29</td><td>41</td><td>28</td></tr><tr><td>Iris</td><td>9</td><td><b>3</b></td><td><b>5</b></td><td><b>3</b></td><td><b>3</b></td><td>5</td><td>5</td><td>5</td><td>5</td></tr><tr><td>Letter</td><td>7863</td><td>3248</td><td>5434</td><td><b>31</b></td><td>1997</td><td>2573</td><td>1073</td><td>1307</td><td>517</td></tr><tr><td>Lymphography</td><td>76</td><td>17</td><td>61</td><td><b>7</b></td><td>9</td><td>54</td><td>29</td><td>63</td><td>17</td></tr><tr><td>Satimage</td><td>1844</td><td>395</td><td>1442</td><td><b>24</b></td><td>220</td><td>816</td><td>521</td><td>678</td><td>128</td></tr><tr><td>Segment</td><td>621</td><td>412</td><td>499</td><td><b>13</b></td><td>192</td><td>286</td><td>121</td><td>372</td><td>68</td></tr><tr><td>Shuttle</td><td>419</td><td>61</td><td>846</td><td><b>19</b></td><td>36</td><td>151</td><td>98</td><td>76</td><td>30</td></tr><tr><td>Sick-euthyroid</td><td>163</td><td>18</td><td>162</td><td><b>9</b></td><td>31</td><td>76</td><td>53</td><td>21</td><td>11</td></tr><tr><td>Sonar</td><td>49</td><td>15</td><td>34</td><td><b>2</b></td><td>16</td><td>32</td><td>31</td><td>26</td><td>17</td></tr><tr><td>Tokyo</td><td>89</td><td>13</td><td>74</td><td><b>9</b></td><td>15</td><td>82</td><td>55</td><td>77</td><td>28</td></tr><tr><td>Vehicle</td><td>406</td><td>204</td><td>402</td><td><b>9</b></td><td>169</td><td>360</td><td>158</td><td>150</td><td>61</td></tr></tbody><tfoot><tr><td><b>Total</b></td><td>14978</td><td>5407</td><td>12297</td><td><b>223</b></td><td>3506</td><td>6282</td><td>3600</td><td>4410</td><td>1320</td></tr></tfoot></table>
<table><caption>**Table 3.** Total number of rules for each evaluation measure when used in FuzzyRULES.</caption><thead><tr><th>Data Set Name</th><th>Purity</th><th>Entropy</th><th>Information Content</th><th>Information Gain</th><th>Accuracy</th><th>Laplace</th><th>m-estimate</th><th>H Measure</th><th>AQ18</th></tr></thead><tbody><tr><td>Abalone</td><td>1527</td><td>449</td><td>1500</td><td><strong>231</strong></td><td>3882</td><td>1560</td><td>936</td><td>752</td><td>424</td></tr><tr><td>Adult</td><td>38136</td><td>2920</td><td>30074</td><td><strong>643</strong></td><td>288</td><td>12639</td><td>14091</td><td>10543</td><td>1197</td></tr><tr><td>Anneal</td><td>2669</td><td>955</td><td>2089</td><td><strong>1320</strong></td><td>1898</td><td>2944</td><td>2310</td><td>2715</td><td>3659</td></tr><tr><td>Australian</td><td>4843</td><td>116</td><td>4393</td><td><strong>64</strong></td><td>53</td><td>3198</td><td>2938</td><td>2421</td><td>95</td></tr><tr><td>Auto</td><td>3195</td><td>2819</td><td>2812</td><td><strong>657</strong></td><td>1321</td><td>2802</td><td>2149</td><td>3507</td><td>1798</td></tr><tr><td>Balance-scale</td><td>179</td><td>92</td><td>179</td><td><strong>28</strong></td><td>59</td><td>177</td><td>161</td><td>157</td><td>67</td></tr><tr><td>Breast</td><td>1736</td><td>85</td><td>1216</td><td><strong>49</strong></td><td>127</td><td>400</td><td>364</td><td>239</td><td>135</td></tr><tr><td>Cleve</td><td>2683</td><td>730</td><td>2089</td><td><strong>174</strong></td><td>227</td><td>1539</td><td>1531</td><td>1529</td><td>352</td></tr><tr><td>Crx</td><td>5998</td><td>564</td><td>5619</td><td><strong>326</strong></td><td>333</td><td>4120</td><td>4496</td><td>1092</td><td>450</td></tr><tr><td>Diabetes</td><td>558</td><td>357</td><td>526</td><td><strong>181</strong></td><td>230</td><td>497</td><td>426</td><td>384</td><td>241</td></tr><tr><td>German</td><td>18651</td><td>6245</td><td>13952</td><td><strong>1156</strong></td><td>3836</td><td>8343</td><td>8475</td><td>12666</td><td>3210</td></tr><tr><td>German-organisation</td><td>28988</td><td>9890</td><td>25034</td><td><strong>934</strong></td><td>11778</td><td>14123</td><td>13134</td><td>17819</td><td>4644</td></tr><tr><td>Glass</td><td>2327</td><td>1434</td><td>2053</td><td><strong>198</strong></td><td>568</td><td>1807</td><td>977</td><td>1884</td><td>278</td></tr><tr><td>Glass2</td><td>753</td><td>134</td><td>636</td><td><strong>103</strong></td><td><strong>103</strong></td><td>291</td><td>294</td><td>449</td><td>143</td></tr><tr><td>Heart-disease</td><td>2390</td><td>395</td><td>1659</td><td><strong>101</strong></td><td>138</td><td>1559</td><td>1626</td><td>1595</td><td>329</td></tr><tr><td>Heart-Hungarian</td><td>1787</td><td>48</td><td>1671</td><td><strong>71</strong></td><td>181</td><td>1074</td><td>987</td><td>1084</td><td>406</td></tr><tr><td>Hepatitis</td><td>2186</td><td>406</td><td>1562</td><td><strong>207</strong></td><td>257</td><td>576</td><td>468</td><td>1817</td><td>653</td></tr><tr><td>Horse-colic</td><td>7338</td><td>540</td><td>6353</td><td><strong>216</strong></td><td>285</td><td>5841</td><td>4974</td><td>6370</td><td>560</td></tr><tr><td>Hypothyroid</td><td>2625</td><td>821</td><td>1948</td><td>964</td><td>1059</td><td>1533</td><td>1431</td><td><strong>399</strong></td><td>802</td></tr><tr><td>Ionosphere</td><td>2578</td><td>266</td><td>2029</td><td><strong>185</strong></td><td>687</td><td>1328</td><td>1207</td><td>1399</td><td>1098</td></tr><tr><td>Iris</td><td>50</td><td><strong>12</strong></td><td>30</td><td><strong>12</strong></td><td><strong>12</strong></td><td>23</td><td>23</td><td>23</td><td>18</td></tr><tr><td>Letter</td><td>326494</td><td>162820</td><td>241125</td><td><strong>2445</strong></td><td>114189</td><td>192745</td><td>77979</td><td>101101</td><td>19991</td></tr><tr><td>Lymphography</td><td>2678</td><td>531</td><td>2119</td><td>369</td><td><strong>325</strong></td><td>1947</td><td>687</td><td>2411</td><td>665</td></tr><tr><td>Satimage</td><td>163604</td><td>60538</td><td>128661</td><td><strong>1676</strong></td><td>14699</td><td>50811</td><td>37315</td><td>46844</td><td>9496</td></tr><tr><td>Segment</td><td>14062</td><td>10600</td><td>12389</td><td><strong>507</strong></td><td>5855</td><td>7077</td><td>4247</td><td>9308</td><td>2256</td></tr><tr><td>Shuttle</td><td>5548</td><td>755</td><td>11391</td><td><strong>165</strong></td><td>403</td><td>2412</td><td>1420</td><td>1033</td><td>270</td></tr><tr><td>Sick-euthyroid</td><td>4911</td><td>1666</td><td>4868</td><td><strong>963</strong></td><td>1655</td><td>3405</td><td>2849</td><td>1496</td><td>1046</td></tr><tr><td>Sonar</td><td>2906</td><td>1250</td><td>1292</td><td><strong>181</strong></td><td>735</td><td>1144</td><td>1038</td><td>979</td><td>849</td></tr><tr><td>Tokyo</td><td>7572</td><td>682</td><td>6633</td><td><strong>581</strong></td><td>629</td><td>5764</td><td>3231</td><td>5201</td><td>1387</td></tr><tr><td>Vehicle</td><td>14091</td><td>9633</td><td>14079</td><td><strong>705</strong></td><td>5555</td><td>13854</td><td>7082</td><td>8995</td><td>2209</td></tr><tr><td>Total</td><td>673063</td><td>277753</td><td>529981</td><td><strong>15412</strong></td><td>171367</td><td>345533</td><td>198846</td><td>246212</td><td>58728</td></tr></tbody></table>

made it to examine the least number of rules. It investigated 98% less rules than the purity method, which inspected the most rules. The amount of search required by the other evaluation measures is also related to the rule set complexity of these measures.

# 5 Conclusions and Future Work

This paper has examined the influence of different crisp evaluation metrics and analysed their role during the rule forming process. It has also proposed fuzzy variants of these measures and examined their performance when used in a recently developed fuzzy rule induction algorithm. The paper has demonstrated the strong effect of the evaluation measures on the performance of the learning algorithm. Evaluation measures that prefer more general over more specific rules helped the learning algorithm to determine the correct models for many data sets. However, over-generalisation may also deteriorate performance in problems that require a very specific concept description. The performance of additional evaluation measures, in particular those balancing the complexity and the accuracy of a particular induced model, could be studied on more data sets and other fuzzy rule induction algorithms.
**Acknowledgements:** The author wishes to thank the Department of Engineering, Design and Mathematics, University of the West of England for providing a good environment, facilities and financial means to complete this paper.

## References

1. Clark, P., Boswell, R.: Rule induction with CN2: Some recent improvements. In: 5th European Conference on Artificial Intelligence, Porto, Portugal, pp. 151–163 (1991).

2. Michalski, R.S., Kaufman, K.A.: A measure of description quality for data mining and its implementation in the AQ18 learning system. In: ICSC Symposium on Advances in Intelligent Data Analysis, The Rochester Institute of Technology, USA, pp. 22–25, (1999).

3. Pham, D.T., Bigot, S., Dimov, S.S.: RULES-5: A rule induction algorithm for problems involving continuous attributes. Proceedings of the Institution of Mechanical Engineers, Part C: J. Mech. Eng. Science, **217**(12), 1273–1286 (2003).

4. Pham, D.T., Afify, A.A.: SRI: A scalable rule induction algorithm. Proceedings of the Institution of Mechanical Engineers, Part C: J. Mech. Eng. Science, **220**(5), 537-552 (2006).

5. Hühn, J., Hüllermeier, E.: FURIA: An algorithm for unordered fuzzy rule induction. Data Mining and Knowledge Discovery, **19**, 293–319 (2009).

6. Afify, A.A.: A novel algorithm for fuzzy rule induction in data mining. Proceedings of the Institution of Mechanical Engineers, Part C: J. Mech. Eng. Science, **228**(5), 877–895 (2014).

7. Afify, A.A.: A fuzzy rule induction algorithm for discovering classification rules. J. Intelligent and Fuzzy Systems, **30**(6), 3067-3085 (2016).

8. Pham, D.T., Afify, A.A.: RULES-6: A simple rule induction algorithm for handling large data sets. Proceedings of the Institution of Mechanical Engineers, Part C: J. Mech. Eng. Science, **219**(10), 1119–1137 (2005).

9. Fürnkranz, J., Flach, P.A.: An analysis of rule evaluation metrics. In: 20th International Conference on Machine Learning, Washington, DC, USA, pp. 202–209 (2003).

10. van Zyl, J., Cloete, I.: Heuristic functions for learning fuzzy conjunctive rules. In: IEEE International Conference on Systems, Man and Cybernetics, The Hague, The Netherlands, pp. 2332–2337 (2004).

11. Klir, G.J., Yuan, B.: Fuzzy Sets and Fuzzy Logic: Theory and Applications, Prentice-Hall, Upper Saddle River, New Jersey, USA (1995).

12. Pagallo, G., Haussler, D.: Boolean feature discovery in empirical learning. Machine Learning, **3**, 71–99 (1990).

13. Weiss, S., Indurkhya, N.: Reduced complexity rule induction. In: 12th International Joint Conference on Artificial Intelligence, Sydney, Australia, pp. 678–684 (1991).

14. Cendrowska, J.: PRISM: An algorithm for inducing modular rules. International J. Man-Machine Studies, **27**, 349–370 (1987).

15. Clark, P., Niblett, T.: The CN2 induction algorithm. Machine Learning, **3**, 261–284 (1989).

16. Quinlan, J.R.: Learning logical definitions from relations. Mach. Learn., **5**, 239–266 (1990).

17. Fürnkranz, J., Widmer, G.: Incremental reduced error pruning. In: 11th International Conference on Machine Learning, New Brunswick, NJ, USA, pp. 70–77 (1994).

18. Cestnik, B.: Estimating probabilities: A crucial task in machine learning. In: 3rd European Conference on Artificial Intelligence, Stockholm, Sweden, pp. 147–149 (1990).

19. Blake, C.L., Merz, C.J.: UCI Repository of Machine Learning Databases. http://www.ics.uci.edu/~mlearn/MLRepository.html, last accessed 2017/6/1.

20. Efron B., Tibshirani R.: An Introduction to the Bootstrap. Chapman & Hall, USA (1993).