# A Hierarchical Procedure for the Synthesis of ANFIS Networks

Authors: Massimo Panella
Journal: Advances in Fuzzy Systems
 (2012)

---

## Abstract

Adaptive neurofuzzy inference systems (ANFIS) represent an efficient technique for the solution of function approximation problems. When numerical samples are available in this regard, the synthesis of ANFIS networks can be carried out exploiting clustering algorithms. Starting from a hyperplane clustering synthesis in the joint input-output space, a computationally efficient optimization of ANFIS networks is proposed in this paper. It is based on a hierarchical constructive procedure, by which the number of rules is progressively increased and the optimal one is automatically determined on the basis of learning theory in order to maximize the generalization capability of the resulting ANFIS network. Extensive computer simulations prove the validity of the proposed algorithm and show a favorable comparison with other well-established techniques.


---

## Body

## 1. Introduction

In this paper, a new procedure for the structural optimization of adaptive neurofuzzy inference systems (ANFIS) is proposed, whose synthesis is based on the hyperplane clustering in the joint input-output space of the data set. ANFIS networks are being widely used to many applicative tasks, such as rule-based process controls, pattern recognition, and function approximation. They solve a general regression problem by means of a set ofM rules of Sugeno first-order type [1]. The kth rule,  k=1…M, has the following form:
(1)Ifx1isB1(k)and…andxnisBn(k)theny(k)=∑j=1naj(k)xj+a0(k),
where x_=[x1x2…xn]T is a column vector (or pattern) in the n-dimensional input space and y(k) is the scalar output associated with the rule. The latter is characterized by the MFs μBj(k)(xj) of the fuzzy input variables Bj(k), j=1…n, and by the coefficients aj(k), j=0…n, of the crisp output.Several alternatives are possible for the fuzzification of crisp inputs, the composition of input MFs, and the way rule outputs are combined [2]. Usually, the structure of the fuzzy inference system is the following one:
(2)y~=∑k=1MμB_(k)(x_)y(k)∑k=1MμB_(k)(x_),
where B_(k) is the overall fuzzy input variable, μB_(k)(x_) is the corresponding MF, and y~ the output estimated for an input x_.The numerical parameters of rules are obtained through a learning process, by using a training set ofP input-output pairs (x_i,yi), i=1…P. The crucial problem during the ANFIS learning is to obtain a good generalization capability. This issue has been deeply investigated in the literature; usually, the generalization capability of ANFIS networks is optimized by checking data set for overfitting model validation [3].The generalization capability is maximized only if the ANFIS network consists of a suitable number of rules. However, the determination of the optimal number is a very critical problem to be solved, since the neural network might be easily overfitted in the case of noisy or ill-conditioned data. In this paper, a new hierarchical constructive procedure for the automatic determination of the ANFIS rules is proposed. It aims to a regularization of the network architecture based on learning theory and hyperplane clustering-based techniques [4–7]. The underlying idea of such techniques received in the literature many acknowledgments as, for example, in the case of theoretical models [8–14] as well as applications to specific fields [15–26].The paper is organized as follows. The synthesis technique for determining the parameters of ANFIS rules is described in Section2. Successively, the structural optimization for the automatic determination of the rules number, which is based on the proposed hierarchical constructive procedure, is illustrated in detail in Section 3. Finally, the numerical results obtained by extensive computer simulations on synthetic benchmarks and a real-world application are summarized in Section 4.


## 2. Hyperplane Clustering for the Synthesis of ANFIS Rules

When dealing with numerical data, the rules of ANFIS networks are commonly synthesized by using clustering techniques, which reduce the redundancy of data so that significant rules are determined directly from the clusters modeling the training set. Clustering algorithms can use joint input-output data, input data only, or output data only; this choice heavily affects the way ANFIS rules are built [27–34].When modeling ANFIS networks, the main drawback due to conventional clustering approaches is that induced clusters do not always reflect the real data structure. An innovative approach, dubbed hyperplane clustering synthesis (HCS), was firstly proposed in [4]. By HCS, hyperplane-shaped clusters are determined in correspondence to the consequent part of each Sugeno rule. Namely, the ANFIS architecture can be considered as a smoothed piecewise linear model, where (2) can be rewritten as
(3)y~=f(x_),f:ℜn⟶ℜ.The regression model is identified by the set ofM  hyperplanes, each associated with a cluster. Therefore, the prototype of the kth cluster in the joint input-output space will be represented by the set of coefficients aj(k), j=0…n, evaluated by means of standard least-squares techniques.It is worth summarizing in the following the core steps for clustering in the “hyperplane space,” which is fundamentally an alternating optimization technique aiming to identify the cluster prototypes. LetΓ={Γ1,Γ2,…,ΓM} be a set of M clusters (i.e., hyperplanes) and let every pattern of the training set be assigned to one of these clusters; this is obtained according to a suitable criterion, as well established in the following of the paper. Then, the hyperplane clustering with M prototypes is based on the following iterative steps.Step 1.
The coefficients of each hyperplane are evaluated by using the following procedure. For thekth hyperplane Γk, k=1…M, a set of linear equations has to be solved; the generic equation is
(4)yt=∑j=1naj(k)xtj+a0(k),
where index “t” spans only the pairs of the training set assigned to the kth cluster. Suited least-squares techniques can be used to solve the set of linear equations in (4).Step 2.
The assignment of patterns to clusters is updated. Each pair(x_i,yi), i=1…P, of the training set is now assigned to the cluster Γq, with q such that
(5)di=|yi-(∑j=1naj(q)xij+a0(q))1+∑j=1n(aj(q))2|=mink=1...M|yi-(∑j=1naj(k)xij+a0(k))1+∑j=1n(aj(k))2|.
The previous choice states that a pattern is assigned the hyperplane having the minimum orthogonal distance from it. This assures the best results since other choices, as for example, the distance along the output axis ofyi only, produces ambiguous results especially in the case of ill-conditioned data yielding quite vertical hyperplanes.Step 3.
For every clusterΓk, the local approximation error is evaluated:
(6)Dk=1Pk∑t‍dt,
where index “t” spans only the Pk pairs of the training set assigned lately (i.e., in Step 2) to the kth cluster.Step 4.
The convergence is based on the quantity:(7)Θ=|D-D(old)|D(old),
where D is the global approximation error over the whole data set in the current iteration, which is defined by
(8)D=1P∑i=1Pdi,
and D(old) is the global approximation error calculated in the previous iteration. If Θ is less than a predetermined threshold θ, then the clustering algorithm is stopped. Otherwise, the iteration goes back to Step 1 by using the current updated association of patterns to clusters. The default value θ=0.01 will be used in the following.The clustering algorithm described so far yields the linear consequents of Sugeno rules only. In order to achieve the complete structure of the ANFIS network, it is mandatory to determine the firing strengths corresponding to the antecedents of rules. To this end, once the iterations of hyperplane clustering have stopped, each pattern of the training set can be labeled with an integerq, 1≤q≤M, representing the hyperplane it has been assigned during Step 2 of the last iteration. By using the labeled training set, a classification problem can be solved in the input space only; at the end of the training process, the input space will be tiled by a classification model able to assign a fuzzy label L(x_) to any pattern x_ of the input space
(9)L(x_)=[μB_(1)(x_)μB_(2)(x_)…μB_(M)(x_)],
where the kth element of L(x_) represents the fuzzy membership of the pattern to the kth class and hence, it can be assumed as the firing strength μB_(k)(x_) of the kth rule associated with the hyperplane corresponding to that class.As a reference classification model, we will adopt in the following well-known Simpson's Min-Max model [35], since it has demonstrated the best performance in this regard. The combination of the hyperplane clustering followed by the fuzzy classification in the input space defines the HCS procedure allowing the determination of an ANFIS network for a given number of rules. HCS will be the basic algorithm adopted for the hierarchical optimization successively proposed.It is worth mentioning that the hyperplane clustering inside HCS is a particular instance of the well-known fuzzyc-regression models (FCRM) algorithm [27]; in this case, a set of c=M linear models (hyperplanes) is fit on the training data and each pattern is assigned during clustering in a crisp manner (i.e, using hard labels) to a unique hyperplane, the one at the minimum distance (5). Although FCRM can use nonlinear models as well, they are basically adopted for regression in order to fit the training data only, even if they are not representing a function (i.e, when several output values y correspond to a same input x_). Conversely, the hyperplane clustering of HCS is successively completed by the classification in the input space, under the assumption that the resulting ANFIS model can be used to approximate/estimate an unknown function represented by its samples in the training set.In this regard, when the outputy~ must be estimated for any input x_ during the normal ANFIS operation (i.e., testing), the classifier is used to determine the fuzzy label (9) using only the input value x_; then, the output y~ is calculated using (2) by means of the firing strengths contained in the fuzzy label L(x_) and the linear consequents in (1), early determined by the hyperplane clustering. In this phase, it is not necessary, in general, to assign the test sample x_ a specific hyperplane cluster. However, a “hard approximation” using a single ANFIS rule can be obtained by a defuzzification of the fuzzy label L(x_), that is,
(10)y~(hard)=∑j=1n‍aj(q)xj+a0(q)q=maxk=1…M{μB_(k)(x_)}.


## 3. Hierarchical Optimization of the ANFIS Structure

When training an ANFIS network, the main problems are the local convergence of estimation algorithms and the correct determination of the numberM of rules. As evidenced successively, the former problem mainly depends on a good (usually random) initialization of numerical parameters associated with each rule. The latter one is a well-known problem, which is directly related to the generalization capability of the neurofuzzy network and it can also be referred to as “structural optimization” problem. In fact, the ANFIS performance could be inadequate if the training set is either underfitted or overfitted by a lacking or an excessive number of rules, respectively.


### 3.1. The Plain Optimization Approach

A plain solution to these problems could be based on the use of the HCS algorithm with different values ofM and with different initializations for every value of M. The HCS algorithm should be initialized before Step 1 of its first iteration by setting an association of each pattern with one of the M hyperplane clusters. The simplest way is to associate patterns to clusters randomly. In this paper, a different approach will be followed, by clustering data in the input space only, in order to obtain a preliminary partition of the training set in clusters. Consequently, the random initialization is necessary for the chosen clustering algorithm in the input space as, for instance, the fuzzy c-means [36, 37]. Once the set of different ANFIS networks is generated, the best network can be chosen by relying on the supervised nature of the learning procedure, that is, by using a cost function measuring the overall generalization capability of the network in terms of its complexity and its approximation error. Basic concepts of learning theory can be adopted in this regard [2, 3]. Namely, the ANFIS network achieving the best generalization capability is the one that, under the same performance on the training set, is characterized by the lowest number of rules.As a measure of the network performance on the training set, the mean squared error (MSE) is adopted:(11)E=1P∑i=1P(yi-y~i)2,
where y~i is the output generated by the ANFIS network in correspondence to the ith input pattern of the training set. The optimal network is selected by using the following cost function depending upon the number of ANFIS rules:
(12)F(M)=(1-λ)E(M)-EminEmax-Emin+λMP,
where Emin and Emax are the extreme values of the performance E that are encountered during the performance investigation of the different ANFIS networks; λ is a weight 0≤λ≤1. This weight is not critical, since the results are slightly affected by its variation in a large interval centered in 0.5. Evidently, for a given value M=M-, the ANFIS network showing the best value F(M-) in (12) will be the one whose initialization and the successive HCS iterations yield the best performance E(M-) on the training set.The plain minimization of (12) can be obtained by the constructive technique denoted as optimized HCS (OHCS). In this procedure, the number of rules is progressively increased from 1 to Mmax,  where  Mmax is a fraction of the training set cardinality P and it represents the maximum complexity that is allowed to the network. For each value of M, different initializations are considered and several ANFIS networks are generated through HCS. In fact, as mentioned, the latter is based on a random initialization of the hyperplanes and hence, different initializations might produce different networks for the same value of M. This is mainly due to the hyperplane clustering, which can easily get trapped at local minima of (8), as for any other alternating optimization procedure.If differentT initializations are carried out for each value of M, the OHCS procedure will generate TMmax networks and the optimal one will be selected according to (12). This optimization approach still suffers from serious drawbacks, which basically depend on the number of different initializations performed for each value of M. The lower is the number of initializations, the lower is the probability to obtain a satisfactory ANFIS network after the HCS algorithm. Conversely, the higher is the number of initializations, the higher is the computational cost of the optimization procedure.


### 3.2. The Hierarchical HCS Procedure

In order to overcome the previous problems, in this paper, the use of a splitting hierarchical optimization is proposed, which will be referred to in the following as hierarchical HCS (HHCS). It is based on the constructive procedure shown in Figure1, where M is increased progressively and only one run of the HCS algorithm is needed for every value of M. In fact, HHCS starts with only one hyperplane (i.e., M=1), which is initialized without any ambiguity since every pattern of the training set is assigned to the sole cluster. Then an iteration starts, involving the succession of the following procedures. (i)
Considering the current association of patterns to clusters, the HCS algorithm is executed on the ANFIS network withM hyperplanes.(ii)
An optional fine tuning of the final ANFIS parameters obtained by the HCS algorithm can be performed by using suited techniques, as outlined in Section3.5.(iii)
If the maximum number of rulesMmax  is reached, then the iteration is stopped and the best ANFIS network is chosen among those obtained during the iterations, once again according to (12). Conversely, if  M<Mmax,  a  splitting procedure is carried out.(iv)
By means of the splitting procedure, which is illustrated successively in Section3.3, the cluster associated with the hyperplane showing “in some sense” the worst local approximation of data is selected to be split into two new hyperplanes, with the aim to better approximate the patterns belonging to the split cluster (as determined by the last execution of Step 2 within the HCS algorithm.) After the split is performed, the old cluster is removed from the ANFIS network and the new ones are inserted, increasing evidently by 1 the value of M. So, the iteration starts back with the HCS algorithm applied to an ANFIS network composed by  M+1 rules, where each pattern of the training set is currently assigned to exactly one cluster.Figure 1
Flow chart of the HHCS algorithm. The tuning of ANFIS networks is an optional step, as discussed in Section3.5.It is important to outline that patterns belonging to the split cluster are assigned exclusively to one of the new generated clusters, while the association of patterns to the other clusters is kept unaltered. This evidences the hierarchical nature of the proposed HHCS procedure. Furthermore, an unambiguous initialization of HCS is assured for any value ofM and hence, the necessity of random initializations and to optimize different HCS solutions for a given value of M is eliminated. Consequently, the computational cost of HHCS is heavily reduced with respect to the plain OHCS approach.The algorithm will stop whenM reaches the maximum complexity Mmax. This is not a critical value to be set; in fact, the same solutions obtained when M spans up to a given value of Mmax are still found when a larger Mmax is adopted. So, the quality of the optimal solution can be traded off with the computational cost of the whole optimization routine given that the larger Mmax, the more solutions are explored spending more time. By the way, we have determined as a value generally acceptable for Mmax the 40% of the training set cardinality P (i.e., Mmax=0.4P).


### 3.3. Splitting Procedure of Clusters

The key role in the HHCS algorithm is played by the splitting procedure, which is intended to determine optimally the cluster (or hyperplane) to be split and, successively, to initialize the hyperplanes associated with its offspring. The underlying idea is to prevent the initialization of new hyperplanes in underpopulated zones of the training set, since this is the typical situation where the HCS algorithm will converge to a poor local minimum of (8). Several heuristics are possible in this regard; the one illustrated in the following steps is based on the supervised nature of function approximation problems and on principal component analysis (PCA). In fact, the cluster is split orthogonally with respect to the direction of maximum variance of local data.(1)
Taking into account the local approximation errors (6), which are determined in the last execution of Step 3 within the HCS algorithm, the cluster to be split, denoted as Γs, 1≤s≤M, is the one exhibiting the  largest local approximation error
(13)Ds=maxk=1…MDk.(2)
LetI(s) be the set of indexes of the Ps patterns associated to Γs during the last execution of Step 2 within the HCS algorithm. The mean vector μ_(s) of such patterns is assumed in the following to be
(14)μ_(s)=1Ps∑t∈I(s)‍x_t,and the covariance matrixC(s) to be
(15)C(s)=1Ps∑t∈I(s)‍(x_t-μ_(s))(x_t-μ_(s))T.Using these notations, the principal components (or directions)q_j(s), j=1…n, of Γs are computed. More precisely, the column vector q_j(s) is the jth eigenvector of C(s) and the eigenvectors are sorted in descending order, according to the value of the corresponding eigenvalue.(3)
Letq_1(s) be the principal component corresponding to the highest eigenvalue. Then, the projections of patterns of Γs along the principal direction is calculated
(16)ξt1(s)=(x_t-μ_(s))Tq_1(s),t∈I(s).Finally, the clusterΓs is removed from the ANFIS network together with the corresponding rule; it will be replaced by the new generated clusters Γs(+) and Γs(-). Taking into account the superscript notation of these clusters, each pattern of Γs will be assigned to either Γs(+) or Γs(-), according to the sign of the projection ξt1(s) in (16).


### 3.4. A Toy Example to Clarify the HHCS Procedure

The details of the proposed hierarchical optimization can be further clarified with the following example concerning the approximation of a simple piecewise linear function consisting of 4 different linear models:(17)f(x)={0,-15≤x<02x,0≤x<5-4x+30,5≤x<10-10,10≤x<20.The training set derived from this function is shown in Figure2: 50 points per hyperplane, for a total of 200 points, are randomly sampled using (17) and a small perturbation by white Gaussian noise is also added in order to have a realistic data set.Figure 2
Training set associated with the toy function (17), represented by the continuous line.The HHCS procedure starts with one rule and the hyperplane clustering is uniquely initialized in this case, since all patterns are initially assigned to the unique hyperplane. Its final determination, after the convergence of clustering, is represented in Figure3. Assuming no network tuning is performed in this example, the splitting procedure illustrated in Section 3.3 is applied to the sole hyperplane that will be removed from the ANFIS network. More precisely, the patterns belonging to the hyperplane (i.e., all the training set in the case M=1) are separated according to the sign of projection (16) along their principal direction. The resulting initialization of hyperplane clustering for M=2 is shown in Figure 4, where the patterns associated with the new generated clusters are represented by crosses and circles, respectively. They are used for the first estimation of the hyperplanes, which are also evidenced in Figure 4.Figure 3
Final hyperplane (dashed line) estimated by the hyperplane clustering in the caseM=1.Figure 4
Initialization of hyperplane clustering forM=2. Patterns belonging to the new generated clusters are represented by the symbols “x” and “o”, respectively; the first estimate of hyperplanes based on these subsets is shown by the continuous gray lines.The hyperplane clustering gets trapped in a local convergence in this case. In fact, the final determination of hyperplanes illustrated in Figure5 is obtained after a few iterations and they are really similar to the initial estimate in Figure 4. In spite of this, the HHCS procedure continues and the hyperplane with negative slope will be split because it marks the largest local approximation error (13). After the splitting procedure, the initialization of hyperplane clustering for M=3 is obtained, as illustrated in Figure 6. The patterns belonging to the split cluster are separated into two subsets similarly to the previous case. Obviously, the patterns associated with the other cluster are not involved in the splitting procedure.Figure 5
Final hyperplanes (dashed lines) estimated by the hyperplane clustering in the caseM=2.Figure 6
Initialization of hyperplane clustering forM=3. Patterns belonging to the new generated clusters are represented by the symbols “x” and “o”, respectively; the first estimate of hyperplanes based on these subsets is shown by the continuous gray lines. Patterns represented by simple dots belong to the cluster associated with the hyperplane that has not been split (dashed line).Also in this case, there is a fast but local convergence to the configuration evidenced in Figure7, which is similar to the former initialization. A hyperplane is then selected for splitting in order to obtain the initialization of HCS illustrated in Figure 8 for the next HHCS iteration with M=4. The convergence of hyperplane clustering is slower and accurate in this case and it yields the final configuration shown in Figure 9 where the linear components of (17) are well modeled now.Figure 7
Final hyperplanes (dashed lines) estimated by the hyperplane clustering in the caseM=3.Figure 8
Initialization of hyperplane clustering forM=4. Patterns belonging to the new generated clusters are represented by the symbols “x” and “o”, respectively; the first estimate of hyperplanes based on these subsets is shown by the continuous gray lines. Patterns represented by simple dots belong to the clusters associated with the hyperplanes that have not been split (dashed lines).Figure 9
Final hyperplanes (dashed lines) estimated by the hyperplane clustering in the caseM=4.Finally, the global approximation performed by the ANFIS network obtained by the HHCS procedure with 4 rules is illustrated in Figure10. The plotted function is obtained using a test set of 35000 patterns uniformly sampled in the input space, xi=-15+0.001(i-1),  i=1…35000, -15≤xi<20, and determining the corresponding outputs y~i using (2).Figure 10
Approximation of the toy function (17) using the ANFIS network obtained by the HHCS procedure with 4 rules.


### 3.5. Computational Remarks

When dealing with the hyperplane-based synthesis of ANFIS networks, a number of considerations can be taken mostly pertaining to the inference of rules’ antecedents from the consequents associated with the hyperplanes and to the fine tuning of ANFIS parameters after the HCS procedure. For example, when the nonlinear functions to be approximated are quite complicated, the number of HBs generated by the Min-Max classifier is often larger than the number of hyperplanes determined by the hyperplane clustering. Thus, a same hyperplane induces several HBs in the input space and can approximate patterns belonging to well-separated regions of the input space, each associated with a different HB.Furthermore, a tuning procedure of ANFIS parameters can be applied after every HCS procedure or, alternatively, to the optimal ANFIS network only. A simple tuning step is considered in this paper, it is applied after every execution of the HCS both in OHCS and HHCS procedures. The coefficientsaj(k), j=0…n, k=1…M, calculated during the hyperplane clustering are updated by a fuzzy least-squares solution of a set of (n+1)M linear equations, which are obtained by using the training set and by setting to zero the derivatives of the MSE (11) with respect to every unknown aj(k):
(18)〈yiγij(k)〉i=∑h=1M∑r=0nar(h)〈γir(h)γij(k)〉ij=0…n,k=1…M,
where 〈·〉i denotes the average over subscript  i and
(19)γij(k)={μB_(k)(x_i)∑h=1MμB_(h)(x_i)xij,1≤j≤nμB_(k)(x_i)∑h=1MμB_(h)(x_i),j=0k=1…M,i=1…P.What is more important to remark is that HHCS allows the structural optimization of ANFIS networks and this is obtained by controlling the computational cost. Moreover, the increasing complexity of networks during training makes HHCS structurally constrained, similarly to other deterministic annealing or entropy constrained approaches [38–40]. In other words, overfitting due to the presence of outliers in the training set can be prevented when M is low, ensuring robustness with respect to noise in the function to be approximated.Although the convergence of HHCS is guaranteed by the inner convergence of HCS algorithm and by the maximum complexityMmax allowed to the network, it should be noticed that HHCS does not ensure the determination of the global minimum of (8) for any value of M. As said, achieving a global optimum is inherently prevented by the HCS algorithm, being this an alternating minimization scheme. HHCS algorithm aims to obtain a suboptimal minimum ensuring an acceptable accuracy of the resulting ANFIS network, although the peculiarity of HHCS is to keep limited the computational cost of the whole structural optimization procedure. In fact, the methods commonly adopted in the literature for searching the global optimum are often characterized by an expensive requirement of computational resources. By the way, in the case of HCS algorithm, one can use some particular modifications that have been proposed in this regard [40–42] or more general approaches for searching the global minimum based, for example, on simulated annealing, genetic algorithms, or tabù search.A characteristic behavior that usually occurs, revealing all the HHCS potentiality, is illustrated in the example pertaining to the approximation problemf1(x_) described in Section 4. The final MSE of the data set versus the number of rules is plotted in Figure 11, while the number of HCS iterations needed to reach convergence is plotted in Figure 12. The continuous line is related to the single HCS execution in HHCS that generates just one ANFIS network for each value of M; the dashed line corresponds to the HCS execution yielding the ANFIS network that showed the best performance, among those differently initialized by applying the plain OHCS procedure (10 different initializations have been used for each value of M). It is evident that the performance of ANFIS networks obtained by HHCS is comparable, and even better, with respect to the ones obtained by OHCS. This also means that the faster HCS convergence in HHCS does not implies a poor local convergence. On the contrary, the lower number of HCS iterations in HHCS is due to the change of only one rule in the fuzzy inference system when M is increased.Figure 11
Prediction accuracy of the HCS algorithm: single execution in HHCS (continuous line); best execution in OHCS (dashed line).Figure 12
Convergence of the HCS algorithm: single execution in HHCS (continuous line); best execution in OHCS (dashed line).


## 3.1. The Plain Optimization Approach

A plain solution to these problems could be based on the use of the HCS algorithm with different values ofM and with different initializations for every value of M. The HCS algorithm should be initialized before Step 1 of its first iteration by setting an association of each pattern with one of the M hyperplane clusters. The simplest way is to associate patterns to clusters randomly. In this paper, a different approach will be followed, by clustering data in the input space only, in order to obtain a preliminary partition of the training set in clusters. Consequently, the random initialization is necessary for the chosen clustering algorithm in the input space as, for instance, the fuzzy c-means [36, 37]. Once the set of different ANFIS networks is generated, the best network can be chosen by relying on the supervised nature of the learning procedure, that is, by using a cost function measuring the overall generalization capability of the network in terms of its complexity and its approximation error. Basic concepts of learning theory can be adopted in this regard [2, 3]. Namely, the ANFIS network achieving the best generalization capability is the one that, under the same performance on the training set, is characterized by the lowest number of rules.As a measure of the network performance on the training set, the mean squared error (MSE) is adopted:(11)E=1P∑i=1P(yi-y~i)2,
where y~i is the output generated by the ANFIS network in correspondence to the ith input pattern of the training set. The optimal network is selected by using the following cost function depending upon the number of ANFIS rules:
(12)F(M)=(1-λ)E(M)-EminEmax-Emin+λMP,
where Emin and Emax are the extreme values of the performance E that are encountered during the performance investigation of the different ANFIS networks; λ is a weight 0≤λ≤1. This weight is not critical, since the results are slightly affected by its variation in a large interval centered in 0.5. Evidently, for a given value M=M-, the ANFIS network showing the best value F(M-) in (12) will be the one whose initialization and the successive HCS iterations yield the best performance E(M-) on the training set.The plain minimization of (12) can be obtained by the constructive technique denoted as optimized HCS (OHCS). In this procedure, the number of rules is progressively increased from 1 to Mmax,  where  Mmax is a fraction of the training set cardinality P and it represents the maximum complexity that is allowed to the network. For each value of M, different initializations are considered and several ANFIS networks are generated through HCS. In fact, as mentioned, the latter is based on a random initialization of the hyperplanes and hence, different initializations might produce different networks for the same value of M. This is mainly due to the hyperplane clustering, which can easily get trapped at local minima of (8), as for any other alternating optimization procedure.If differentT initializations are carried out for each value of M, the OHCS procedure will generate TMmax networks and the optimal one will be selected according to (12). This optimization approach still suffers from serious drawbacks, which basically depend on the number of different initializations performed for each value of M. The lower is the number of initializations, the lower is the probability to obtain a satisfactory ANFIS network after the HCS algorithm. Conversely, the higher is the number of initializations, the higher is the computational cost of the optimization procedure.


## 3.2. The Hierarchical HCS Procedure

In order to overcome the previous problems, in this paper, the use of a splitting hierarchical optimization is proposed, which will be referred to in the following as hierarchical HCS (HHCS). It is based on the constructive procedure shown in Figure1, where M is increased progressively and only one run of the HCS algorithm is needed for every value of M. In fact, HHCS starts with only one hyperplane (i.e., M=1), which is initialized without any ambiguity since every pattern of the training set is assigned to the sole cluster. Then an iteration starts, involving the succession of the following procedures. (i)
Considering the current association of patterns to clusters, the HCS algorithm is executed on the ANFIS network withM hyperplanes.(ii)
An optional fine tuning of the final ANFIS parameters obtained by the HCS algorithm can be performed by using suited techniques, as outlined in Section3.5.(iii)
If the maximum number of rulesMmax  is reached, then the iteration is stopped and the best ANFIS network is chosen among those obtained during the iterations, once again according to (12). Conversely, if  M<Mmax,  a  splitting procedure is carried out.(iv)
By means of the splitting procedure, which is illustrated successively in Section3.3, the cluster associated with the hyperplane showing “in some sense” the worst local approximation of data is selected to be split into two new hyperplanes, with the aim to better approximate the patterns belonging to the split cluster (as determined by the last execution of Step 2 within the HCS algorithm.) After the split is performed, the old cluster is removed from the ANFIS network and the new ones are inserted, increasing evidently by 1 the value of M. So, the iteration starts back with the HCS algorithm applied to an ANFIS network composed by  M+1 rules, where each pattern of the training set is currently assigned to exactly one cluster.Figure 1
Flow chart of the HHCS algorithm. The tuning of ANFIS networks is an optional step, as discussed in Section3.5.It is important to outline that patterns belonging to the split cluster are assigned exclusively to one of the new generated clusters, while the association of patterns to the other clusters is kept unaltered. This evidences the hierarchical nature of the proposed HHCS procedure. Furthermore, an unambiguous initialization of HCS is assured for any value ofM and hence, the necessity of random initializations and to optimize different HCS solutions for a given value of M is eliminated. Consequently, the computational cost of HHCS is heavily reduced with respect to the plain OHCS approach.The algorithm will stop whenM reaches the maximum complexity Mmax. This is not a critical value to be set; in fact, the same solutions obtained when M spans up to a given value of Mmax are still found when a larger Mmax is adopted. So, the quality of the optimal solution can be traded off with the computational cost of the whole optimization routine given that the larger Mmax, the more solutions are explored spending more time. By the way, we have determined as a value generally acceptable for Mmax the 40% of the training set cardinality P (i.e., Mmax=0.4P).


## 3.3. Splitting Procedure of Clusters

The key role in the HHCS algorithm is played by the splitting procedure, which is intended to determine optimally the cluster (or hyperplane) to be split and, successively, to initialize the hyperplanes associated with its offspring. The underlying idea is to prevent the initialization of new hyperplanes in underpopulated zones of the training set, since this is the typical situation where the HCS algorithm will converge to a poor local minimum of (8). Several heuristics are possible in this regard; the one illustrated in the following steps is based on the supervised nature of function approximation problems and on principal component analysis (PCA). In fact, the cluster is split orthogonally with respect to the direction of maximum variance of local data.(1)
Taking into account the local approximation errors (6), which are determined in the last execution of Step 3 within the HCS algorithm, the cluster to be split, denoted as Γs, 1≤s≤M, is the one exhibiting the  largest local approximation error
(13)Ds=maxk=1…MDk.(2)
LetI(s) be the set of indexes of the Ps patterns associated to Γs during the last execution of Step 2 within the HCS algorithm. The mean vector μ_(s) of such patterns is assumed in the following to be
(14)μ_(s)=1Ps∑t∈I(s)‍x_t,and the covariance matrixC(s) to be
(15)C(s)=1Ps∑t∈I(s)‍(x_t-μ_(s))(x_t-μ_(s))T.Using these notations, the principal components (or directions)q_j(s), j=1…n, of Γs are computed. More precisely, the column vector q_j(s) is the jth eigenvector of C(s) and the eigenvectors are sorted in descending order, according to the value of the corresponding eigenvalue.(3)
Letq_1(s) be the principal component corresponding to the highest eigenvalue. Then, the projections of patterns of Γs along the principal direction is calculated
(16)ξt1(s)=(x_t-μ_(s))Tq_1(s),t∈I(s).Finally, the clusterΓs is removed from the ANFIS network together with the corresponding rule; it will be replaced by the new generated clusters Γs(+) and Γs(-). Taking into account the superscript notation of these clusters, each pattern of Γs will be assigned to either Γs(+) or Γs(-), according to the sign of the projection ξt1(s) in (16).


## 3.4. A Toy Example to Clarify the HHCS Procedure

The details of the proposed hierarchical optimization can be further clarified with the following example concerning the approximation of a simple piecewise linear function consisting of 4 different linear models:(17)f(x)={0,-15≤x<02x,0≤x<5-4x+30,5≤x<10-10,10≤x<20.The training set derived from this function is shown in Figure2: 50 points per hyperplane, for a total of 200 points, are randomly sampled using (17) and a small perturbation by white Gaussian noise is also added in order to have a realistic data set.Figure 2
Training set associated with the toy function (17), represented by the continuous line.The HHCS procedure starts with one rule and the hyperplane clustering is uniquely initialized in this case, since all patterns are initially assigned to the unique hyperplane. Its final determination, after the convergence of clustering, is represented in Figure3. Assuming no network tuning is performed in this example, the splitting procedure illustrated in Section 3.3 is applied to the sole hyperplane that will be removed from the ANFIS network. More precisely, the patterns belonging to the hyperplane (i.e., all the training set in the case M=1) are separated according to the sign of projection (16) along their principal direction. The resulting initialization of hyperplane clustering for M=2 is shown in Figure 4, where the patterns associated with the new generated clusters are represented by crosses and circles, respectively. They are used for the first estimation of the hyperplanes, which are also evidenced in Figure 4.Figure 3
Final hyperplane (dashed line) estimated by the hyperplane clustering in the caseM=1.Figure 4
Initialization of hyperplane clustering forM=2. Patterns belonging to the new generated clusters are represented by the symbols “x” and “o”, respectively; the first estimate of hyperplanes based on these subsets is shown by the continuous gray lines.The hyperplane clustering gets trapped in a local convergence in this case. In fact, the final determination of hyperplanes illustrated in Figure5 is obtained after a few iterations and they are really similar to the initial estimate in Figure 4. In spite of this, the HHCS procedure continues and the hyperplane with negative slope will be split because it marks the largest local approximation error (13). After the splitting procedure, the initialization of hyperplane clustering for M=3 is obtained, as illustrated in Figure 6. The patterns belonging to the split cluster are separated into two subsets similarly to the previous case. Obviously, the patterns associated with the other cluster are not involved in the splitting procedure.Figure 5
Final hyperplanes (dashed lines) estimated by the hyperplane clustering in the caseM=2.Figure 6
Initialization of hyperplane clustering forM=3. Patterns belonging to the new generated clusters are represented by the symbols “x” and “o”, respectively; the first estimate of hyperplanes based on these subsets is shown by the continuous gray lines. Patterns represented by simple dots belong to the cluster associated with the hyperplane that has not been split (dashed line).Also in this case, there is a fast but local convergence to the configuration evidenced in Figure7, which is similar to the former initialization. A hyperplane is then selected for splitting in order to obtain the initialization of HCS illustrated in Figure 8 for the next HHCS iteration with M=4. The convergence of hyperplane clustering is slower and accurate in this case and it yields the final configuration shown in Figure 9 where the linear components of (17) are well modeled now.Figure 7
Final hyperplanes (dashed lines) estimated by the hyperplane clustering in the caseM=3.Figure 8
Initialization of hyperplane clustering forM=4. Patterns belonging to the new generated clusters are represented by the symbols “x” and “o”, respectively; the first estimate of hyperplanes based on these subsets is shown by the continuous gray lines. Patterns represented by simple dots belong to the clusters associated with the hyperplanes that have not been split (dashed lines).Figure 9
Final hyperplanes (dashed lines) estimated by the hyperplane clustering in the caseM=4.Finally, the global approximation performed by the ANFIS network obtained by the HHCS procedure with 4 rules is illustrated in Figure10. The plotted function is obtained using a test set of 35000 patterns uniformly sampled in the input space, xi=-15+0.001(i-1),  i=1…35000, -15≤xi<20, and determining the corresponding outputs y~i using (2).Figure 10
Approximation of the toy function (17) using the ANFIS network obtained by the HHCS procedure with 4 rules.


## 3.5. Computational Remarks

When dealing with the hyperplane-based synthesis of ANFIS networks, a number of considerations can be taken mostly pertaining to the inference of rules’ antecedents from the consequents associated with the hyperplanes and to the fine tuning of ANFIS parameters after the HCS procedure. For example, when the nonlinear functions to be approximated are quite complicated, the number of HBs generated by the Min-Max classifier is often larger than the number of hyperplanes determined by the hyperplane clustering. Thus, a same hyperplane induces several HBs in the input space and can approximate patterns belonging to well-separated regions of the input space, each associated with a different HB.Furthermore, a tuning procedure of ANFIS parameters can be applied after every HCS procedure or, alternatively, to the optimal ANFIS network only. A simple tuning step is considered in this paper, it is applied after every execution of the HCS both in OHCS and HHCS procedures. The coefficientsaj(k), j=0…n, k=1…M, calculated during the hyperplane clustering are updated by a fuzzy least-squares solution of a set of (n+1)M linear equations, which are obtained by using the training set and by setting to zero the derivatives of the MSE (11) with respect to every unknown aj(k):
(18)〈yiγij(k)〉i=∑h=1M∑r=0nar(h)〈γir(h)γij(k)〉ij=0…n,k=1…M,
where 〈·〉i denotes the average over subscript  i and
(19)γij(k)={μB_(k)(x_i)∑h=1MμB_(h)(x_i)xij,1≤j≤nμB_(k)(x_i)∑h=1MμB_(h)(x_i),j=0k=1…M,i=1…P.What is more important to remark is that HHCS allows the structural optimization of ANFIS networks and this is obtained by controlling the computational cost. Moreover, the increasing complexity of networks during training makes HHCS structurally constrained, similarly to other deterministic annealing or entropy constrained approaches [38–40]. In other words, overfitting due to the presence of outliers in the training set can be prevented when M is low, ensuring robustness with respect to noise in the function to be approximated.Although the convergence of HHCS is guaranteed by the inner convergence of HCS algorithm and by the maximum complexityMmax allowed to the network, it should be noticed that HHCS does not ensure the determination of the global minimum of (8) for any value of M. As said, achieving a global optimum is inherently prevented by the HCS algorithm, being this an alternating minimization scheme. HHCS algorithm aims to obtain a suboptimal minimum ensuring an acceptable accuracy of the resulting ANFIS network, although the peculiarity of HHCS is to keep limited the computational cost of the whole structural optimization procedure. In fact, the methods commonly adopted in the literature for searching the global optimum are often characterized by an expensive requirement of computational resources. By the way, in the case of HCS algorithm, one can use some particular modifications that have been proposed in this regard [40–42] or more general approaches for searching the global minimum based, for example, on simulated annealing, genetic algorithms, or tabù search.A characteristic behavior that usually occurs, revealing all the HHCS potentiality, is illustrated in the example pertaining to the approximation problemf1(x_) described in Section 4. The final MSE of the data set versus the number of rules is plotted in Figure 11, while the number of HCS iterations needed to reach convergence is plotted in Figure 12. The continuous line is related to the single HCS execution in HHCS that generates just one ANFIS network for each value of M; the dashed line corresponds to the HCS execution yielding the ANFIS network that showed the best performance, among those differently initialized by applying the plain OHCS procedure (10 different initializations have been used for each value of M). It is evident that the performance of ANFIS networks obtained by HHCS is comparable, and even better, with respect to the ones obtained by OHCS. This also means that the faster HCS convergence in HHCS does not implies a poor local convergence. On the contrary, the lower number of HCS iterations in HHCS is due to the change of only one rule in the fuzzy inference system when M is increased.Figure 11
Prediction accuracy of the HCS algorithm: single execution in HHCS (continuous line); best execution in OHCS (dashed line).Figure 12
Convergence of the HCS algorithm: single execution in HHCS (continuous line); best execution in OHCS (dashed line).


## 4. Numerical Results

The validity of the HHCS procedure has been proved by extensive computer simulations. Some illustrative examples are summarized in the following, considering the numerical results obtained by well-known neural and neurofuzzy models applied to specific function approximation problems.


### 4.1. Synthetic Benchmarks

First of all, the HHCS procedure is compared with the plain OHCS in terms of accuracy, complexity and the computational cost necessary to determine the optimal network. Such quantities can be represented, respectively, by MSE, number of rules, overall number of HCS iterations (i.e., the cumulative sum of iterations in all the HCS executions required by the whole optimization procedure), and normalized time. We consider for the sake of comparison the well-known function approximation problems proposed in [43]:   (i)
f1(x_)=Ca/(1+e-Cbx2)+Θ(x1);(ii)
f2(x_)=Ccsin2(2π((5-x1)2+(5-x2)2)/10);(iii)
f3(x_)=Cd(5-x2)2/(3·(5-x1)2+(5-x2)2); with x_=[x1,x2],  x1,x2∈[0,1], Θ(x1) is a small perturbation of the output surface of the first example. A uniform spiral distribution was used to produce a training set of 400 examples for each function. As mentioned, the default values also used in the subsequent tests are λ=0.5; Mmax=160 (i.e., 40% of the training set cardinality); T=10 (i.e., 10 different initializations carried out for every value of M in OHCS). The results shown in Table 1 evidence that HHCS has a computational cost considerably smaller than OHCS, while maintaining a good approximation accuracy and controlling the overall complexity in order to avoid overfitting. The normalized time spent in OCHS with respect to HHCS is more than expected by considering the number of iterations only. As said, this is due to the slower convergence of the HCS procedure within OHCS since random initializations are used whenever.Table 1
Comparison between HHCS and OHCS for different approximation problems.
Method
HHCS
OHCS
Rules
MSE
Iterations
Time
Rules
MSE
Iterations
Time
f
1
(
x
_
)
21
3.06
1753
1.0
18
3.26
4187
3.51
f
2
(
x
_
)
48
1.86
2420
1.0
81
1.14
4154
2.66
f
3
(
x
_
)
41
1.07
1819
1.0
37
1.23
4183
3.27
MSE values are scaled by10-2.By using the same data sets, the HHCS procedure is also compared with respect to other neural and neurofuzzy paradigms, in particular the self-organized fuzzy rule generation (SOFRG) method proposed in [43], where also the results of the well-known radial basis function (RBF) [44–46], standard multilayer perceptron (SMLP) [47], and momentum-improved multilayer perceptron (IMLP) [48] are reported. The MSE obtained by these systems on the said data sets are illustrated in Table 2 and once more, they prove that HHCS compare favorably with respect to the other systems and the related learning algorithm. We outline that, in order to obtain comparable results, we have considered the parameters setup of the benchmarks already reported in the literature, where the complexity of RBF, SMLP, and IMLP neural networks was optimized in advance to 25 neurons by using suited cross-validation experiments. Conversely, HHCS, OHCS, and SOFRG determine automatically the complexity of the neurofuzzy network.Table 2
Comparison with other neural and neurofuzzy models.
Method
HHCS
OHCS
SOFRG
RBF
SMLP
IMLP
f
1
(
x
_
)
3.06
3.26
4.90
3.10
2.60
5.20
f
2
(
x
_
)
1.86
1.14
3.80
9.80
17.30
7.80
f
3
(
x
_
)
1.07
1.23
1.20
1.90
2.10
2.70
MSE values are scaled by10-2.The efficacy of the proposed HHCS approach is also evaluated in terms of generalization capability, which is the crucial characteristic of any neural system. Other well-known methods for ANFIS synthesis are considered [49]: the combination of the resilient propagation (RPROP) and the recursive least-squares error (RLSE) technique [50]; the gradient descent (GD) optimization joined to the RLSE method. The following three-input nonlinear function should be modeled:   (iv)
f4(x_)=1+x10.5+x2-1+x3-1.5, where 216 points are used in the training set and other 125 points in the test set. The performance results are reported in Table 3 in terms of MSE. The optimal number of rules reported in the literature in the case of RPROP + RLSE and GD + RLSE algorithms is M=9, while the constructive optimization of OHCS and HHCS yields ANFIS networks having fewer rules and showing a lower error on the test set and hence, a better generalization capability especially using HHCS.Table 3
Comparison with RPROP + RLSE and GD + RLSE methods.
Method
HHCS
OHCS
RPROP + RLSE
GD + RLSE
Rules
3
4
9
9
MSE (training set)
4.75
·
1
0
-
4
1.39
·
1
0
-
4
6.67
·
1
0
-
6
8.60
·
1
0
-
4
MSE (test set)
7.92
·
1
0
-
3
1.27
·
1
0
-
2
4.74
·
1
0
-
2
6.69
·
1
0
-
2Finally, the comparison with some gradient descent optimization techniques for the learning of fuzzy rule bases, in particular the Sugeno rules (TS-A) and centered-Sugeno rules (C-TS-A), is considered. Both training and test sets consist of 250 patterns, obtained by random sampling of the following two-input nonlinear function [51]:   (v)
f5(x_)=(3e2x1+2e-4x2)/170,x1,x2∈[-1,1].The numerical results are shown in Table4; the optimal number of rules obtained in [51] by the TS-A and C-TS-A approaches is M=16; also in this case HHCS attains a good generalization capability on the test set even if a reduced number of rules is used with respect to OHCS, TS-A, and C-TS-A methods.Table 4
Comparison with TS-A and C-TS-A methods.
Method
HHCS
OHCS
TS-A
C-TS-A
Rules
13
19
16
16
MSE (training set)
12.13
3.32
10.40
7.20
MSE (test set)
5.41
5.73
34.40
37.60
MSE values are scaled by10-6.


### 4.2.  Time Series Prediction

The synthesis procedure proposed in this paper is also validated by considering the use of the resulting ANFIS network in a typical real-world application. More precisely, we focus on the prediction of biological time series, which are relevant to health monitoring and risk prevention in many daily activities as in clinical applications, telemedicine, road safety, and so on [52].The general approach to solve a prediction problem is based on the solution of a suitable function approximation problem. In fact, a prediction problem can be solved by synthesizing the function that links the current sample to be predicted to a suitable set of past ones [53]. Let S(t) be the sample of the time series to be predicted at time t and let the past samples be known up to time t-1, the estimated value S~(t) can be obtained as
(20)S~(t)=f(x_),f:ℜn→ℜ,x_=[S(t-1)S(t-2)…S(t-n)].Hence, the implementation of a predictor coincides with the estimation of an approximation model as in (3), by using any data driven function approximation technique. Neurofuzzy networks are useful to solve such problems, because of the intrinsic nonlinearity and complexity of the underlying functions to be estimated [54]. The ANFIS network obtained by the HHCS procedure is compared with several data driven modeling techniques useful for approximation: a linear model determined by the well-known least-squares approximation (LSE); the RBF neural network; the Mixture of Gaussian (MoG) neural network, which is particularly suited to the solution of multivalued and nonconvex function approximation problems [40, 55].The glucose time series is sampled every 20 minutes; the time series of heart rate is sampled every 5 seconds; the skin conductivity is sampled every one second. On the basis of a preliminary analysis of such time series, performed by experts in the context of clinical trials, three past samples are sufficient to predict the future value, that is,n=3 is used in (20). Each predictor is trained on the first 500 samples of every time series (training set); the related performance is evaluated as the signal-to-noise ratio (SNR) in predicting the successive 150 samples (test set). The SNR is defined as the ratio, measured in decibels (dB), between the mean squared error of prediction and the variance of the time series to be predicted.The performances obtained on both training and test set of the considered time series are illustrated in Table5. Except for the linear model, RBF and MoG neural networks have been optimized by cross-validation and constructive procedures similarly to HHCS, in order to find the optimal number of parameters that maximizes the generalization capability. Looking at the performance on the test set, the ANFIS networks generated by the HHCS procedure obtain a higher SNR with respect to the other models for all the considered time series.Table 5
Prediction of biological time series (dB).
Predictor
Glucose
Heart rate
Conductivity
LSE (training set)
4.541
27.082
12.879
LSE (test set)
2.845
21.258
12.614
RBF (training set)
22.127
29.435
14.707
RBF (test set)
21.817
23.582
11.787
MoG (training set)
23.996
28.133
16.269
MoG (test set)
22.195
22.329
12.136
HHCS (training set)
26.745
27.992
17.032
HHCS (test set)
23.576
24.855
12.863


## 4.1. Synthetic Benchmarks

First of all, the HHCS procedure is compared with the plain OHCS in terms of accuracy, complexity and the computational cost necessary to determine the optimal network. Such quantities can be represented, respectively, by MSE, number of rules, overall number of HCS iterations (i.e., the cumulative sum of iterations in all the HCS executions required by the whole optimization procedure), and normalized time. We consider for the sake of comparison the well-known function approximation problems proposed in [43]:   (i)
f1(x_)=Ca/(1+e-Cbx2)+Θ(x1);(ii)
f2(x_)=Ccsin2(2π((5-x1)2+(5-x2)2)/10);(iii)
f3(x_)=Cd(5-x2)2/(3·(5-x1)2+(5-x2)2); with x_=[x1,x2],  x1,x2∈[0,1], Θ(x1) is a small perturbation of the output surface of the first example. A uniform spiral distribution was used to produce a training set of 400 examples for each function. As mentioned, the default values also used in the subsequent tests are λ=0.5; Mmax=160 (i.e., 40% of the training set cardinality); T=10 (i.e., 10 different initializations carried out for every value of M in OHCS). The results shown in Table 1 evidence that HHCS has a computational cost considerably smaller than OHCS, while maintaining a good approximation accuracy and controlling the overall complexity in order to avoid overfitting. The normalized time spent in OCHS with respect to HHCS is more than expected by considering the number of iterations only. As said, this is due to the slower convergence of the HCS procedure within OHCS since random initializations are used whenever.Table 1
Comparison between HHCS and OHCS for different approximation problems.
Method
HHCS
OHCS
Rules
MSE
Iterations
Time
Rules
MSE
Iterations
Time
f
1
(
x
_
)
21
3.06
1753
1.0
18
3.26
4187
3.51
f
2
(
x
_
)
48
1.86
2420
1.0
81
1.14
4154
2.66
f
3
(
x
_
)
41
1.07
1819
1.0
37
1.23
4183
3.27
MSE values are scaled by10-2.By using the same data sets, the HHCS procedure is also compared with respect to other neural and neurofuzzy paradigms, in particular the self-organized fuzzy rule generation (SOFRG) method proposed in [43], where also the results of the well-known radial basis function (RBF) [44–46], standard multilayer perceptron (SMLP) [47], and momentum-improved multilayer perceptron (IMLP) [48] are reported. The MSE obtained by these systems on the said data sets are illustrated in Table 2 and once more, they prove that HHCS compare favorably with respect to the other systems and the related learning algorithm. We outline that, in order to obtain comparable results, we have considered the parameters setup of the benchmarks already reported in the literature, where the complexity of RBF, SMLP, and IMLP neural networks was optimized in advance to 25 neurons by using suited cross-validation experiments. Conversely, HHCS, OHCS, and SOFRG determine automatically the complexity of the neurofuzzy network.Table 2
Comparison with other neural and neurofuzzy models.
Method
HHCS
OHCS
SOFRG
RBF
SMLP
IMLP
f
1
(
x
_
)
3.06
3.26
4.90
3.10
2.60
5.20
f
2
(
x
_
)
1.86
1.14
3.80
9.80
17.30
7.80
f
3
(
x
_
)
1.07
1.23
1.20
1.90
2.10
2.70
MSE values are scaled by10-2.The efficacy of the proposed HHCS approach is also evaluated in terms of generalization capability, which is the crucial characteristic of any neural system. Other well-known methods for ANFIS synthesis are considered [49]: the combination of the resilient propagation (RPROP) and the recursive least-squares error (RLSE) technique [50]; the gradient descent (GD) optimization joined to the RLSE method. The following three-input nonlinear function should be modeled:   (iv)
f4(x_)=1+x10.5+x2-1+x3-1.5, where 216 points are used in the training set and other 125 points in the test set. The performance results are reported in Table 3 in terms of MSE. The optimal number of rules reported in the literature in the case of RPROP + RLSE and GD + RLSE algorithms is M=9, while the constructive optimization of OHCS and HHCS yields ANFIS networks having fewer rules and showing a lower error on the test set and hence, a better generalization capability especially using HHCS.Table 3
Comparison with RPROP + RLSE and GD + RLSE methods.
Method
HHCS
OHCS
RPROP + RLSE
GD + RLSE
Rules
3
4
9
9
MSE (training set)
4.75
·
1
0
-
4
1.39
·
1
0
-
4
6.67
·
1
0
-
6
8.60
·
1
0
-
4
MSE (test set)
7.92
·
1
0
-
3
1.27
·
1
0
-
2
4.74
·
1
0
-
2
6.69
·
1
0
-
2Finally, the comparison with some gradient descent optimization techniques for the learning of fuzzy rule bases, in particular the Sugeno rules (TS-A) and centered-Sugeno rules (C-TS-A), is considered. Both training and test sets consist of 250 patterns, obtained by random sampling of the following two-input nonlinear function [51]:   (v)
f5(x_)=(3e2x1+2e-4x2)/170,x1,x2∈[-1,1].The numerical results are shown in Table4; the optimal number of rules obtained in [51] by the TS-A and C-TS-A approaches is M=16; also in this case HHCS attains a good generalization capability on the test set even if a reduced number of rules is used with respect to OHCS, TS-A, and C-TS-A methods.Table 4
Comparison with TS-A and C-TS-A methods.
Method
HHCS
OHCS
TS-A
C-TS-A
Rules
13
19
16
16
MSE (training set)
12.13
3.32
10.40
7.20
MSE (test set)
5.41
5.73
34.40
37.60
MSE values are scaled by10-6.


## 4.2.  Time Series Prediction

The synthesis procedure proposed in this paper is also validated by considering the use of the resulting ANFIS network in a typical real-world application. More precisely, we focus on the prediction of biological time series, which are relevant to health monitoring and risk prevention in many daily activities as in clinical applications, telemedicine, road safety, and so on [52].The general approach to solve a prediction problem is based on the solution of a suitable function approximation problem. In fact, a prediction problem can be solved by synthesizing the function that links the current sample to be predicted to a suitable set of past ones [53]. Let S(t) be the sample of the time series to be predicted at time t and let the past samples be known up to time t-1, the estimated value S~(t) can be obtained as
(20)S~(t)=f(x_),f:ℜn→ℜ,x_=[S(t-1)S(t-2)…S(t-n)].Hence, the implementation of a predictor coincides with the estimation of an approximation model as in (3), by using any data driven function approximation technique. Neurofuzzy networks are useful to solve such problems, because of the intrinsic nonlinearity and complexity of the underlying functions to be estimated [54]. The ANFIS network obtained by the HHCS procedure is compared with several data driven modeling techniques useful for approximation: a linear model determined by the well-known least-squares approximation (LSE); the RBF neural network; the Mixture of Gaussian (MoG) neural network, which is particularly suited to the solution of multivalued and nonconvex function approximation problems [40, 55].The glucose time series is sampled every 20 minutes; the time series of heart rate is sampled every 5 seconds; the skin conductivity is sampled every one second. On the basis of a preliminary analysis of such time series, performed by experts in the context of clinical trials, three past samples are sufficient to predict the future value, that is,n=3 is used in (20). Each predictor is trained on the first 500 samples of every time series (training set); the related performance is evaluated as the signal-to-noise ratio (SNR) in predicting the successive 150 samples (test set). The SNR is defined as the ratio, measured in decibels (dB), between the mean squared error of prediction and the variance of the time series to be predicted.The performances obtained on both training and test set of the considered time series are illustrated in Table5. Except for the linear model, RBF and MoG neural networks have been optimized by cross-validation and constructive procedures similarly to HHCS, in order to find the optimal number of parameters that maximizes the generalization capability. Looking at the performance on the test set, the ANFIS networks generated by the HHCS procedure obtain a higher SNR with respect to the other models for all the considered time series.Table 5
Prediction of biological time series (dB).
Predictor
Glucose
Heart rate
Conductivity
LSE (training set)
4.541
27.082
12.879
LSE (test set)
2.845
21.258
12.614
RBF (training set)
22.127
29.435
14.707
RBF (test set)
21.817
23.582
11.787
MoG (training set)
23.996
28.133
16.269
MoG (test set)
22.195
22.329
12.136
HHCS (training set)
26.745
27.992
17.032
HHCS (test set)
23.576
24.855
12.863


## 5. Conclusion

This paper focuses on the training of ANFIS networks with the aim to find effective solutions for improving the accuracy in actual regression problems. The novel HHCS procedure is proposed in the paper within the framework of hyperplane clustering synthesis of ANFIS networks; it involves a constructive approach that progressively increases the number of rules. This procedure ensures a structural robustness of the resulting network with respect to the overfitting phenomenon, while improving the convergence and reducing the computational cost in correspondence to any given number of rules. Thus, it is characterized by a high robustness and it is well suited to play the core role in more complex modeling systems.The numerical results illustrated in the paper encourage to a further development of the HHCS approach, since the resulting ANFIS networks perform better than several neurofuzzy models and the related learning algorithms applied to well-known benchmarks and real-world problems reported in the literature.



---

*Source: 491237-2012-10-04.xml*