growing radial basis neural networks merging supervised and unsupervised learning with network growth techniques this paper proposes a framework for constructing and training radial basis function rbf neural networks the proposed growing radial basis function grbf network begins with a small number of prototypes which determine the locations of radial basis functions in the process of training the grbf network gross by splitting one of the prototypes at each growing cycle two splitting criteria are proposed to determine which prototype to split in each growing cycle the proposed hybrid learning scheme provides a framework for incorporating existing algorithms in the training of grbf networks these include unsupervised algorithms for clustering and learning vector quantization as well as learning algorithms for training single layer linear neural networks a supervised learning scheme based on the minimization of the localized class conditional variance is also proposed and tested grbf neural networks are evaluated and tested on a variety of data sets with very satisfactory results

asymptotic behaviour of some bootstrap estimators the aim of this paper is to present a theoretical analysis of three bootstrap estimators commonly used in system identification the main results concern the local convergence and accuracy properties of the estimators both iterative and on line versions are examined

two stage orthogonal least squares methods for neural network construction a number of neural networks can be formulated as the linear in the parameters models training such networks can be transformed to a model selection problem where a compact model is selected from all the candidates using subset selection algorithms forward selection methods are popular fast subset selection approaches however they may only produce suboptimal models and can be trapped into a local minimum more recently a two stage fast recursive algorithm tsfra combining forward selection and backward model refinement has been proposed to improve the compactness and generalization performance of the model this paper proposes unified two stage orthogonal least squares methods instead of the fast recursive based methods in contrast to the tsfra this paper derives a new simplified relationship between the forward and the backward stages to avoid repetitive computations using the inherent orthogonal properties of the least squares methods furthermore a new term exchanging scheme for backward model refinement is introduced to reduce computational demand finally given the error reduction ratio criterion effective and efficient forward and backward subset selection procedures are proposed extensive examples are presented to demonstrate the improved model compactness constructed by the proposed technique in comparison with some popular methods

universal approximation using radial basis function networks there have been several recent studies concerning feedforward networks and the problem of approximating arbitrary functionals of a finite number of real variables some of these studies deal with cases in which the hidden layer nonlinearity is not a sigmoid this was motivated by successful applications of feedforward networks with nonsigmoidal hidden layer units this paper reports on a related study of radial basis function rbf networks and it is proved that rbf networks having one hidden layer are capable of universal approximation here the emphasis is on the case of typical rbf networks and the results show that a certain class of rbf networks with the same smoothing factor in each kernel node is broad enough for universal approximation

a new discrete continuous algorithm for radial basis function networks construction the construction of a radial basis function rbf network involves the determination of the model size hidden nodes and output weights least squares based subset selection methods can determine a rbf model size and its parameters simultaneously although these methods are robust they may not achieve optimal results alternatively gradient methods are widely used to optimize all the parameters the drawback is that most algorithms may converge slowly as they treat hidden nodes and output weights separately and ignore their correlations in this paper a new discrete continuous algorithm is proposed for the construction of a rbf model first the orthogonal least squares ols based forward stepwise selection constructs an initial model by selecting model terms one by one from a candidate term pool then a new levenberg marquardt lm based parameter optimization is proposed to further optimize the hidden nodes and output weights in the continuous space to speed up the convergence the proposed parameter optimization method considers the correlation between the hidden nodes and output weights which is achieved by translating the output weights to dependent parameters using the ols method the correlation is also used by the previously proposed continuous forward algorithm cfa however unlike the cfa the new method optimizes all the parameters simultaneously in addition an equivalent recursive sum of squared error is derived to reduce the computation demanding for the first derivatives used in the lm method computational complexity is given to confirm the new method is much more computationally efficient than the cfa different numerical examples are presented to illustrate the effectiveness of the proposed method further friedman statistical tests on 13 classification problems are performed and the results demonstrate that rbf networks built by the new method are very competitive in comparison with some popular classifiers

an axiomatic approach to soft learning vector quantization and clustering this paper presents an axiomatic approach to soft learning vector quantization lvq and clustering based on reformulation the reformulation of the fuzzy c means fcm algorithm provides the basis for reformulating entropy constrained fuzzy clustering ecfc algorithms according to the proposed approach the development of specific algorithms reduces to the selection of a generator function linear generator functions lead to the fcm and fuzzy learning vector quantization algorithms while exponential generator functions lead to ecfc and entropy constrained learning vector quantization algorithms the reformulation of lvq and clustering algorithms also provides the basis for developing uncertainty measures that can identify feature vectors equidistant from all prototypes these measures are employed by a procedure developed to make soft lvq and clustering algorithms capable of identifying outliers in the data set this procedure is evaluated by testing the algorithms generated by linear and exponential generator functions on speech data

technical communique analysis of an output error identification algorithm an output error based algorithm proposed by landau and recently analysed by dugard and landau is shown to have a certain robustness property specifically it is proved that the parameters of the system transfer function can be estimated consistently with that algorithm even if the noise is not correctly modelled this fact was noticed empirically by dugard and landau but a theoretical analysis was not provided

quantum neural networks qnns inherently fuzzy feedforward neural networks this paper introduces quantum neural networks qnns a class of feedforward neural networks ffnns inherently capable of estimating the structure of a feature space in the form of fuzzy sets the hidden units of these networks develop quantized representations of the sample information provided by the training data set in various graded levels of certainty unlike other approaches attempting to merge fuzzy logic and neural networks qnns can be used in pattern classification problems without any restricting assumptions such as the availability of a priori knowledge or desired membership profile convexity of classes a limited number of classes etc experimental results presented here show that qnns are capable of recognizing structures in data a property that conventional ffnns with sigmoidal hidden units lack

classification methodologies of multilayer perceptrons with sigmoid activation functions this paper studies the classification mechanisms of multilayer perceptrons mlps with sigmoid activation functions safs the viewpoint is presented that in the input space the hyperplanes determined by the hidden basis functions with values 0 s do not play the role of decision boundaries and such hyperplanes do not certainly go through the marginal regions between different classes for solving an n class problem a single hidden layer perceptron with at least log2 n 1 2 hidden nodes is needed the final number of hidden neurons is still related to the sample distribution shapes and regions but not to the number of samples and input dimensions as a result an empirical formula for optimally selecting the initial number of hidden nodes is proposed the ranks of response matrixes of hidden layers should be taken as a main basis for pruning or growing the existing hidden neurons a structure fixed perceptron ought to learn more than one round from different starting weight points for one classification task and only the group of weights and biases that has the best generalization performance should be reserved finally three examples are given to verify the above viewpoints

an integrated approach to fuzzy learning vector quantization and fuzzy c means clustering derives an interpretation for a family of competitive learning algorithms and investigates their relationship to fuzzy c means and fuzzy learning vector quantization these algorithms map a set of feature vectors into a set of prototypes associated with a competitive network that performs unsupervised learning derivation of the new algorithms is accomplished by minimizing an average generalized distance between the feature vectors and prototypes using gradient descent a close relationship between the resulting algorithms and fuzzy c means is revealed by investigating the functionals involved it is also shown that the fuzzy c means and fuzzy learning vector quantization algorithms are related to the proposed algorithms if the learning rate at each iteration is selected to satisfy a certain condition

fast and efficient second order method for training radial basis function networks this paper proposes an improved second order iso algorithm for training radial basis function rbf networks besides the traditional parameters including centers widths and output weights the input weights on the connections between input layer and hidden layer are also adjusted during the training process more accurate results can be obtained by increasing variable dimensions initial centers are chosen from training patterns and other parameters are generated randomly in limited range taking the advantages of fast convergence and powerful search ability of second order algorithms the proposed iso algorithm can normally reach smaller training testing error with much less number of rbf units during the computation process quasi hessian matrix and gradient vector are accumulated as the sum of related sub matrices and vectors respectively only one jacobian row is stored and used for multiplication instead of the entire jacobian matrix storage and multiplication memory reduction benefits the computation speed and allows the training of problems with basically unlimited number of patterns several practical discrete and continuous classification problems are applied to test the properties of the proposed iso training algorithm

technical communique eigenvalue location of certain matrices arising in convergence analysis problems we give simple formulas for the eigenvalues of a certain type of matrices occurring in some convergence analysis problems application of the result to a pseudo linear regression algorithm gives explicit necessary and sufficient conditions for local convergence

analysis of augmented input layer rbfnn in this paper we present and analyze a new structure for designing a radial basis function neural network rbfnn in the training phase input layer of rbfnn is augmented with desired output vector generalization phase involves the following steps 1 identify the cluster to which a previously unseen input vector belongs 2 augment the input layer with an average of the targets of the input vectors in the identified cluster and 3 use the augmented network to estimate the unknown target it is shown that under some reasonable assumptions the generalization error function admits an upper bound in terms of the quantization errors minimized when determining the centers of the proposed method over the training set and the difference between training samples and generalization samples in a deterministic setting when the difference between the training and generalization samples goes to zero the upper bound can be made arbitrarily small by increasing the number of hidden neurons computer simulations verified the effectiveness of the proposed method

auxiliary model based least squares identification methods for hammerstein output error systems abstract the difficulty in identification of a hammerstein a linear dynamical block following a memoryless nonlinear block nonlinear output error model is that the information vector in the identification model contains unknown variables the noise free true outputs of the system in this paper an auxiliary model based least squares identification algorithm is developed the basic idea is to replace the unknown variables by the output of an auxiliary model convergence analysis of the algorithm indicates that the parameter estimation error consistently converges to zero under a generalized persistent excitation condition the simulation results show the effectiveness of the proposed algorithms

combined parameter and output estimation of dual rate systems using an auxiliary model for a dual rate sampled data system an auxiliary model based identification algorithm for combined parameter and output estimation is proposed the basic idea is to use an auxiliary model to estimate the unknown noise free output true output of the system and directly to identify the parameters of the underlying fast single rate model from the dual rate input output data it is shown that the parameter estimation error consistently converges to zero under generalized or weak persistent excitation conditions and unbounded noise variance and that the output estimates uniformly converge to the true outputs an example is included

parameterization of point cloud freeform surfaces using adaptive sequential learning rbfnetworks we propose a self organizing radial basis function rbf neural network method for parameterization of freeform surfaces from larger noisy and unoriented point clouds in particular an adaptive sequential learning algorithm is presented for network construction from a single instance of point set the adaptive learning allows neurons to be dynamically inserted and fully adjusted e g their locations widths and weights according to mapping residuals and data point novelty associated to underlying geometry pseudo neurons exhibiting very limited contributions can be removed through a pruning procedure additionally a neighborhood extended kalman filter nekf was developed to significantly accelerate parameterization experimental results show that this adaptive learning enables effective capture of global low frequency variations while preserving sharp local details ultimately leading to accurate and compact parameterization as characterized by a small number of neurons parameterization using the proposed rbf network provides simple low cost and low storage solutions to many problems such as surface construction re sampling hole filling multiple level of detail meshing and data compression from unstructured and incomplete range data performance results are also presented for comparison

optimization of clustering criteria by reformulation various hard fuzzy and possibilistic clustering criteria objective functions are useful as bases for a variety of pattern recognition problems at present many of these criteria have customized individual optimization algorithms because of the specialized nature of these algorithms experimentation with new and existing criteria can be very inconvenient and costly in terms of development and implementation time this paper shows how to reformulate some clustering criteria so that specialized algorithms can be replaced by general optimization routines found in commercially available software we prove that the original and reformulated versions of each criterion are fully equivalent finally two numerical examples are given to illustrate reformulation

solving a system of nonlinear integral equations by an rbf network in this paper a novel learning strategy for radial basis function networks rbfn is proposed by adjusting the parameters of the hidden layer including the rbf centers and widths the weights of the output layer are adapted by local optimization methods a new local optimization algorithm based on a combination of the gradient and newton methods is introduced the efficiency of some local optimization methods to update the weights of rbfn is studied in solving systems of nonlinear integral equations

rbf based neurodynamic nearest neighbor classification in real pattern space superposition of radial basis functions centered at given prototype patterns constitutes one of the most suitable energy forms for gradient systems that perform nearest neighbor classification with real valued static prototypes it is shown in this paper that a continuous time dynamical neural network model employing a radial basis function and a sigmoid multi layer perceptron sub networks is capable of maximizing such an energy form locally thus performing almost perfectly nearest neighbor classification when initiated by a distorted pattern the proposed design scheme allows for explicit representation of prototype patterns as network parameters as well as augmenting additional or forgetting existing memory patterns the dynamical classification scheme implemented by the network eliminates all comparisons which are the vital steps of the conventional nearest neighbor classification process the performance of the proposed network model is demonstrated on binary and gray scale image reconstruction applications

reformulated radial basis neural networks trained by gradient descent this paper presents an axiomatic approach for constructing radial basis function rbf neural networks this approach results in a broad variety of admissible rbf models including those employing gaussian rbfs the form of the rbfs is determined by a generator function new rbf models can be developed according to the proposed approach by selecting generator functions other than exponential ones which lead to gaussian rbfs this paper also proposes a supervised learning algorithm based on gradient descent for training reformulated rbf neural networks constructed using the proposed approach a sensitivity analysis of the proposed algorithm relates the properties of rbfs with the convergence of gradient descent learning experiments involving a variety of reformulated rbf networks generated by linear and exponential generator functions indicate that gradient descent learning is simple easily implementable and produces rbf networks that perform considerably better than conventional rbf models trained by existing algorithms

multiobjective evolutionary optimization of the size shape and position parameters of radial basis function networks for function approximation this paper presents a multiobjective evolutionary algorithm to optimize radial basis function neural networks rbfnns in order to approach target functions from a set of input output pairs the procedure allows the application of heuristics to improve the solution of the problem at hand by including some new genetic operators in the evolutionary process these new operators are based on two well known matrix transformations singular value decomposition svd and orthogonal least squares ols which have been used to define new mutation operators that produce local or global modifications in the radial basis functions rbfs of the networks the individuals in the population in the evolutionary procedure after analyzing the efficiency of the different operators we have shown that the global mutation operators yield an improved procedure to adjust the parameters of the rbfnns

multivariate modified fourier series and application to boundary value problems in this paper we analyse the approximation theoretic properties of modified fourier series in cartesian product domains with coefficients from both full and hyperbolic cross index sets we show that the number of expansion coefficients may be reduced significantly whilst retaining comparable error estimates in doing so we extend the univariate results of iserles norsett and s olver we then demonstrate that these series can be used in the spectral galerkin approximation of second order neumann boundary value problems which offers some advantages over standard chebyshev or legendre polynomial discretizations

approximation and radial basis function networks this paper concerns conditions for the approximation of functions in certain general spaces using radial basis function networks it has been shown in recent papers that certain classes of radial basis function networks are broad enough for universal approximation in this paper these results are considerably extended and sharpened

on the construction and training of reformulated radial basis function neural networks presents a systematic approach for constructing reformulated radial basis function rbf neural networks which was developed to facilitate their training by supervised learning algorithms based on gradient descent this approach reduces the construction of radial basis function models to the selection of admissible generator functions the selection of generator functions relies on the concept of the blind spot which is introduced in the paper the paper also introduces a new family of reformulated radial basis function neural networks which are referred to as cosine radial basis functions cosine radial basis functions are constructed by linear generator functions of a special form and their use as similarity measures in radial basis function models is justified by their geometric interpretation a set of experiments on a variety of datasets indicate that cosine radial basis functions outperform considerably conventional radial basis function neural networks with gaussian radial basis functions cosine radial basis functions are also strong competitors to existing reformulated radial basis function models trained by gradient descent and feedforward neural networks with sigmoid hidden units

sparse multivariate gaussian mixture regression fitting a multivariate gaussian mixture to data represents an attractive as well as challenging problem in especial when sparsity in the solution is demanded achieving this objective requires the concurrent update of all parameters weight centers and precisions of all multivariate gaussian functions during the learning process such is the focus of this paper which presents a novel method founded on the minimization of the error of the generalized logarithmic utility function gluf this choice which allows us to move smoothly from the mean square error mse criterion to the one based on the logarithmic error yields an optimization problem that resembles a locally convex problem and can be solved with a quasi newton method the gluf framework also facilitates the comparative study between both extremes concluding that the classical mse optimization is not the most adequate for the task the performance of the proposed novel technique is demonstrated on simulated as well as realistic scenarios