growing radial basis neural networks merging supervised and unsupervised learning with network growth techniques this paper proposes a framework for constructing and training radial basis function rbf neural networks the proposed growing radial basis function grbf network begins with a small number of prototypes which determine the locations of radial basis functions in the process of training the grbf network gross by splitting one of the prototypes at each growing cycle two splitting criteria are proposed to determine which prototype to split in each growing cycle the proposed hybrid learning scheme provides a framework for incorporating existing algorithms in the training of grbf networks these include unsupervised algorithms for clustering and learning vector quantization as well as learning algorithms for training single layer linear neural networks a supervised learning scheme based on the minimization of the localized class conditional variance is also proposed and tested grbf neural networks are evaluated and tested on a variety of data sets with very satisfactory results

matrix momentum for practical natural gradient learning an on line learning rule based on the introduction of a matrix momentum term is presented aimed at alleviating the computational costs of standard natural gradient learning the new rule natural gradient matrix momentum is analysed in the case of two layer feed forward neural network learning via methods of statistical physics it appears to provide a practical algorithm that performs as well as standard natural gradient descent in both the transient and asymptotic regimes but with a hugely reduced complexity

two level linear programming decentralized planning has long been recognized as an important decision making problem many approaches based on the concepts of large scale system decomposition have generally lacked the ability to model the type of truly independent subsystems which often exist in practice multilevel programming models partition control over decision variables among ordered levels within a hierarchical planning structure a planner at one level of the hierarchy may have his objective function and set of feasible decisions determined in part by other levels however his control instruments may allow him to influence the policies at other levels and thereby improve his own objective function this paper examines the special case of the two level linear programming problem geometric characterizations and algorithms are presented with some examples the goal is to demonstrate the tractability of such problems and motivate a wider interest in their study

statistical theory of learning curves under entropic loss criterion the present paper elucidates a universal property of learning curves which shows how the generalization error training error and the complexity of the underlying stochastic machine are related and how the behavior of a stochastic machine is improved as the number of training examples increases the error is measured by the entropic loss it is proved that the generalization error converges to h0 the entropy of the conditional distribution of the true machine as h0 m 2t while the training error converges as h0 m 2t where t is the number of examples and m shows the complexity of the network when the model is faithful implying that the true machine is in the model m is reduced to m the number of modifiable parameters this is a universal law because it holds for any regular machine irrespective of its structure under the maximum likelihood estimator similar relations are obtained for the bayes and gibbs learning algorithms these learning curves show the relation among the accuracy of learning the complexity of a model and the number of training examples

optimal state reductions of automata with partially specified behaviors nondeterministic finite automata with don t care states namely states which neither accept nor reject are considered a characterization of deterministic automata compatible with such a device is obtained furthermore an optimal state bound for the smallest compatible deterministic automata is provided it is proved that the problem of minimizing deterministic don t care automata is np complete and pspace hard in the nondeterministic case the restriction to the unary case is also considered

three level incentive strategies in differential games the three level incentive strategy in a non linear differential game ia defined the separability in obtaining incentive strategies is shown the sufficient condition for a three level incentive strategy in a linear quadratic differential game is derived it is shown that a gradient method is applicable for finding a three level incentive strategy in a linear quadratic differential game

universal approximation using radial basis function networks there have been several recent studies concerning feedforward networks and the problem of approximating arbitrary functionals of a finite number of real variables some of these studies deal with cases in which the hidden layer nonlinearity is not a sigmoid this was motivated by successful applications of feedforward networks with nonsigmoidal hidden layer units this paper reports on a related study of radial basis function rbf networks and it is proved that rbf networks having one hidden layer are capable of universal approximation here the emphasis is on the case of typical rbf networks and the results show that a certain class of rbf networks with the same smoothing factor in each kernel node is broad enough for universal approximation

how initial conditions affect generalization performance in large networks generalization is one of the most important problems in neural network research it is influenced by several factors in the network design such as network size weight decay factor and others we show here that the initial weight distribution for gradient decent training algorithms is one other factor that influences generalization the initial conditions guide the training algorithm to search particular places of the weight space for instance small initial weights tend to result in low complexity networks and therefore can effectively act as a regularization factor we propose a novel network complexity measure which is helpful in shedding insight into the phenomenon as well as in studying other aspects of generalization

asymptotic statistical theory of overtraining and cross validation a statistical theory for overtraining is proposed the analysis treats general realizable stochastic neural networks trained with kullback leibler divergence in the asymptotic case of a large number of training examples it is shown that the asymptotic gain in the generalization error is small if we perform early stopping even if we have access to the optimal stopping time based on the cross validation stopping we consider the ratio the examples should be divided into training and cross validation sets in order to obtain the optimum performance although cross validated early stopping is useless in the asymptotic region it surely decreases the generalization error in the nonasymptotic region our large scale simulations done on a cm5 are in good agreement with our analytical findings

generalization errors of the simple perceptron to find an exact form for the generalization error of a learning machine is an open problem even in the simplest case simple perceptron learning we introduce a new approach to tackle the problem the generalization error of the simple perceptron is expressed as a linear combination of extreme values of inputs with the help of extreme value theory in statistics we then obtain an exact form of the generalization error of the simple perceptron in the case of the worst learning generalization errors of the higher order perceptron taking the form of an inverse power law in the number of examples are also considered

optimal incentive strategy for leader follower games an optimal incentive strategy by which the leader suffers the least losses in punishing the follower s deviation from the decision desired by the leader is defined sufficient conditions for the existence of optimal incentive strategy are given static and dynamic leader follower games with quadratic cost functionals are investigated it is shown that leader follower games with quadratic cost functionals admit optimal incentive strategies under the follower s decision variable being scalar there exists a unique linear optimal incentive strategy such an incentive strategy can be explicitly determined

singularities affect dynamics of learning in neuromanifolds the parameter spaces of hierarchical systems such as multilayer perceptrons include singularities due to the symmetry and degeneration of hidden units a parameter space forms a geometrical manifold called the neuromanifold in the case of neural networks such a model is identified with a statistical model and a riemannian metric is given by the fisher information matrix however the matrix degenerates at singularities such a singular structure is ubiquitous not only in multilayer perceptrons but also in the gaussian mixture probability densities arma time series model and many other cases the standard statistical paradigm of the cramer rao theorem does not hold and the singularity gives rise to strange behaviors in parameter estimation hypothesis testing bayesian inference model selection and in particular the dynamics of learning from examples prevailing theories so far have not paid much attention to the problem caused by singularity relying only on ordinary statistical theories developed for regular nonsingular models only recently have researchers remarked on the effects of singularity and theories are now being developed this article gives an overview of the phenomena caused by the singularities of statistical manifolds related to multilayer perceptrons and gaussian mixtures we demonstrate our recent results on these problems simple toy models are also used to show explicit solutions we explain that the maximum likelihood estimator is no longer subject to the gaussian distribution even asymptotically because the fisher information matrix degenerates that the model selection criteria such as aic bic and mdl fail to hold in these models that a smooth bayesian prior becomes singular in such models and that the trajectories of dynamics of learning are strongly affected by the singularity causing plateaus or slow manifolds in the parameter space the natural gradient method is shown to perform well because it takes the singular geometrical structure into account the generalization error and the training error are studied in some examples

four types of learning curves if machines are learning to make decisions given a number of examples the generalization error e t is defined as the average probability that an incorrect decision is made for a new example by a machine when trained with t examples the generalization error decreases as t increases and the curve e t is called a learning curve the present paper uses the bayesian approach to show that given the annealed approximation learning curves can be classified into four asymptotic types if the machine is deterministic with noiseless teacher signals then 1 e at 1 when the correct machine parameter is unique and 2 e at 2 when the set of the correct parameters has a finite measure if the teacher signals are noisy then 3 e at 1 2 for a deterministic machine and 4 e c at 1 for a stochastic machine

advantageous strategies in differential games for a class of suboptimal control laws abstract this paper deals with suboptimal control policies in differential games sufficient conditions for advantageous strategies of either player are examined for a linear quadratic game in which the players are constrained to use suboptimal control laws of specified form the concept of a bargaining matrix is employed specific scalar and vector examples are included to illustrate the theory

a numerical study on learning curves in stochastic multilayer feedforward networks the universal asymptotic scaling laws proposed by amari et al are studied in large scale simulations using a cm5 small stochastic multilayer feedforward networks trained with backpropagation are investigated in the range of a large number of training patterns t the asymptotic generalization error scales as 1 t as predicted for a medium range t a faster 1 t2 scaling is observed this effect is explained by using higher order corrections of the likelihood expansion it is shown for small t that the scaling law changes drastically when the network undergoes a transition from strong overfitting to effective learning

a neural network learning theory and a polynomial time rbf algorithm this paper presents a new learning theory a set of principles for brain like learning and a corresponding algorithm for the neural network field the learning theory defines computational characteristics that are much more brain like than that of classical connectionist learning robust and reliable learning algorithms would result if these learning principles are followed rigorously when developing neural network algorithms this paper also presents a new algorithm for generating radial basis function rbf nets for function approximation the design of the algorithm is based on the proposed set of learning principles the net generated by this algorithm is not a typical rbf net but a combination of truncated rbf and other types of hidden units the algorithm uses random clustering and linear programming lp to design and train this mixed rbf net polynomial time complexity of the algorithm is proven and computational results are provided for the well known mackey glass chaotic time series problem the logistic map prediction problem various neuro control problems and several time series forecasting problems the algorithm can also be implemented as an online adaptive algorithm

use of bias term in projection pursuit learning improves approximation and convergence properties in a regression problem one is given a multidimensional random vector x the components of which are called predictor variables and a random variable y called response a regression surface describes a general relationship between x and y a nonparametric regression technique that has been successfully applied to high dimensional data is projection pursuit regression ppr the regression surface is approximated by a sum of empirically determined univariate functions of linear combinations of the predictors projection pursuit learning ppl formulates ppr using a 2 layer feedforward neural network the smoothers in ppr are nonparametric whereas those in ppl are based on hermite functions of some predefined highest order r we demonstrate that ppl networks in the original form do not have the universal approximation property for any finite r and thus cannot converge to the desired function even with an arbitrarily large number of hidden units but by including a bias term in each linear projection of the predictor variables ppl networks can regain these capabilities independent of the exact choice of r experimentally it is shown in this paper that this modification increases the rate of convergence with respect to the number of hidden units improves the generalization performance and makes it less sensitive to the setting of r finally we apply ppl to chaotic time series prediction and obtain superior results compared with the cascade correlation architecture

construction of minimal n 2 n encoders for any n the encoding problem rumelhart and mcclelland 1986 is an important canonical problem it has been widely used as a benchmark here we have analytically derived minimal sized nets necessary and sufficient to solve encoding problems of arbitrary size the proofs are constructive we construct n 2 n encoders and show that two hidden units are also necessary for n 2 moreover the geometric approach employed is general and has much wider applications for example this method has also helped us derive lower bounds on the redundancy necessary for achieving complete fault tolerance phatak and koren 1992a b

stochastic complexity and modeling on demontre un theoreme fondamental qui donne une borne inferieure pour la longueur de code et donc pour les erreurs de prediction on definit les notions d information a priori et d information utile dans les donnees

neural representation of space using sinusoidal arrays o keefe 1991 has proposed that spatial information in rats might be represented as phasors phase and amplitude of a sine wave encoding angle and distance to a landmark we describe computer simulations showing that operations on phasors can be efficiently realized by arrays of spiking neurons that re code the temporal dimension of the sine wave spatially some cells in motor and parietal cortex exhibit response properties compatible with this proposal

on an unsupervised learning rule for scalar quantization following the maximum entropy principle in this paper we observe that a particular class of rational function rf approximations may be viewed as feedforward networks like the radial basis function rbf network the training of the rf network may be performed using a linear adaptive filtering algorithm we illustrate the application of the rf network by considering two nonlinear signal processing problems the first problem concerns the one step prediction of a time series consisting of a pair of complex sinusoid in the presence of colored non gaussian noise simulated data were used for this problem in the second problem we use the rf network to build a nonlinear dynamic model of sea clutter radar backscattering from a sea surface here real life data were used for the study

parameter convergence and learning curves for neural networks we revisit the oft studied asymptotic in sample size behavior of the parameter or weight estimate returned by any member of a large family of neural network training algorithms by properly accounting for the characteristic property of neural networks that their empirical and generalization errors possess multiple minima we rigorously establish conditions under which the parameter estimate converges strongly into the set of minima of the generalization error convergence of the parameter estimate to a particular value cannot be guaranteed under our assumptions we then evaluate the asymptotic distribution of the distance between the parameter estimate and its nearest neighbor among the set of minima of the generalization error results on this question have appeared numerous times and generally assert asymptotic normality the conclusion expected from familiar statistical arguments concerned with maximum likelihood estimators these conclusions are usually reached on the basis of somewhat informal calculations although we shall see that the situation is somewhat delicate the preceding results then provide a derivation of learning curves for generalization and empirical errors that leads to bounds on rates of convergence

error bounds for functional approximation and estimation using mixtures of experts we examine some mathematical aspects of learning unknown mappings with the mixture of experts model mem specifically we observe that the mem is at least as powerful as a class of neural networks in a sense that will be made precise upper bounds on the approximation error are established for a wide class of target functions the general theorem states that spl par f f sub n spl par sub p spl les c n sup r d for f spl isin w sub p sup r l a sobolev class over 1 1 sup d and f sub n belongs to an n dimensional manifold of normalized ridge functions the same bound holds for the mem as a special case of the above the stochastic error in the context of learning from independent and identically distributed i i d examples is also examined an asymptotic analysis establishes the limiting behavior of this error in terms of certain pseudo information matrices these results substantiate the intuition behind the mem and motivate applications

attraction radii in binary hopfield nets are hard to compute we prove that it is an np hard problem to determine the attraction radius of a stable vector in a binary hopfield memory network and even that the attraction radius is hard to approximate under synchronous updating the problems are already np hard for two step attraction radii direct one step attraction radii can be computed in polynomial time

derivation of necessary and sufficient conditions for single stage stackelberg games via the inducible region concept the inducible region is defined as the collection of all the possible outcomes it is typically a subset of the entire decision space the best the leader can obtain is then the optimal outcome in this inducible region necessary and sufficient conditions are derived by first delineating the inducible region and then obtaining a leader s optimal strategy if it exists if not an e strategy always exists provided that the leader s stackelberg cost is bounded

nondeterministic moore automata and brzozowski s minimization algorithm moore automata represent a model that has many applications in this paper we define a notion of coherent nondeterministic moore automaton nma and show that such a model has the same computational power of the classical deterministic moore automaton we consider also the problem of constructing the minimal deterministic moore automaton equivalent to a given nma we propose an algorithm that is a variant of brzozowski s minimization algorithm in the sense that it is essentially structured as reverse operation and subset construction performed twice moreover we explore more general classes of nma and analyze the applicability of the algorithm for some of such classes the algorithm does not return the minimal equivalent deterministic automaton

prediction intervals for artificial neural networks abstract the artificial neural network ann is becoming a very popular model for engineering and scientific applications inspired by brain architecture artificial neural networks represent a class of nonlinear models capable of learning from data neural networks have been applied in many areas including pattern matching classification prediction and process control this article focuses on the construction of prediction intervals previous statistical theory for constructing confidence intervals for the parameters or the weights in an ann is inappropriate because the parameters are unidentifiable we show in this article that the problem disappears in prediction we then construct asymptotically valid prediction intervals and also show how to use the prediction intervals to choose the number of nodes in the network we then apply the theory to an example for predicting the electrical load

on dimensionality sample size and classification error of nonparametric linear classification algorithms this paper compares two nonparametric linear classification algorithms the zero empirical error classifier and the maximum margin classifier with parametric linear classifiers designed to classify multivariate gaussian populations formulae and a table for the mean expected probability of misclassification mep sub n are presented they show that the classification error is mainly determined by n p a learning set size dimensionality ratio however the influences of learning set size on the generalization error of parametric and nonparametric linear classifiers are quite different under certain conditions the nonparametric approach allows us to obtain reliable rules even in cases where the number of features is larger than the number of training vectors

finknn a fuzzy interval number k nearest neighbor classifier for prediction of sugar production from populations of samples this work introduces finknn a k nearest neighbor classifier operating over the metric lattice of conventional interval supported convex fuzzy sets we show that for problems involving populations of measurements data can be represented by fuzzy interval numbers fins and we present an algorithm for constructing fins from such populations we then present a lattice theoretic metric distance between fins with arbitrary shaped membership functions which forms the basis for finknn s similarity measurements we apply finknn to the task of predicting annual sugar production based on populations of measurements supplied by hellenic sugar industry we show that finknn improves prediction accuracy on this task and discuss the broader scope and potential utility of these techniques

layered neural networks with gaussian hidden units as universal approximations a neural network with a single layer of hidden units of gaussian type is proved to be a universal approximator for real valued maps defined on convex compact sets of rn

redundancy reduction as a strategy for unsupervised learning a redundancy reduction strategy which can be applied in stages is proposed as a way to learn as efficiently as possible the statistical properties of an ensemble of sensory messages the method works best for inputs consisting of strongly correlated groups that is features with weaker statistical dependence between different features this is the case for localized objects in an image or for words in a text a local feature measure determining how much a single feature reduces the total redundancy is derived which turns out to depend only on the probability of the feature and of its components but not on the statistical properties of any other features the locality of this measure makes it ideal as the basis for a neural implementation of redundancy reduction and an example of a very simple non hebbian algorithm is given the effect of noise on learning redundancy is also discussed

algebraic analysis for nonidentifiable learning machines this article clarifies the relation between the learning curve and the algebraic geometrical structure of a nonidentifiable learning machine such as a multilayer neural network whose true parameter set is an analytic set with singular points by using a concept in algebraic analysis we rigorously prove that the bayesian stochastic complexity or the free energy is asymptotically equal to 1 log n m1 1 log log n constant where n is the number of training samples and 1 and m1 are the rational number and the natural number which are determined as the birational invariant values of the singularities in the parameter space also we show an algorithm to calculate 1 and m1 based on the resolution of singularities in algebraic geometry in regular statistical models 2 1 is equal to the number of parameters and m1 1 whereas in nonregular models such as multilayer networks 2 1 is not larger than the number of parameters and m1 1 since the increase of the stochastic complexity is equal to the learning curve or the generalization error the nonidentifiable learning machines are better models than the regular ones if bayesian ensemble learning is applied

natural gradient works efficiently in learning when a parameter space has a certain underlying structure the ordinary gradient of a function does not represent its steepest direction but the natural gradient does information geometry is used for calculating the natural gradients in the parameter space of perceptrons the space of matrices for blind source separation and the space of linear dynamical systems for blind source deconvolution the dynamical behavior of natural gradient online learning is analyzed and is proved to be fisher efficient implying that it has asymptotically the same performance as the optimal batch estimation of parameters this suggests that the plateau phenomenon which appears in the backpropagation learning algorithm of multilayer perceptrons might disappear or might not be so serious when the natural gradient is used an adaptive method of updating the learning rate is proposed and analyzed

a polynomial time algorithm for generating neural networks for pattern classification its stability properties and some test results polynomial time training and network design are two major issues for the neural network community a new algorithm has been developed that can learn in polynomial time and also design an appropriate network the algorithm is for classification problems and uses linear programing models to design and train the network this paper summarizes the new algorithm proves its stability properties and provides some computational results to demonstrate its potential

stochastic complexity and generalization error of a restricted boltzmann machine in bayesian estimation in this paper we consider the asymptotic form of the generalization error for the restricted boltzmann machine in bayesian estimation it has been shown that obtaining the maximum pole of zeta functions is related to the asymptotic form of the generalization error for hierarchical learning models watanabe 2001a b the zeta function is defined by using a kullback function we use two methods to obtain the maximum pole a new eigenvalue analysis method and a recursive blowing up process we show that these methods are effective for obtaining the asymptotic form of the generalization error of hierarchical learning models

learning in feedforward layered networks the tiling algorithm the authors propose a new algorithm which builds a feedforward layered network in order to learn any boolean function of n boolean units the number of layers and the number of hidden units in each layer are not prescribed in advance they are outputs of the algorithm it is an algorithm for growth of the network which adds layers and units inside a layer at will until convergence the convergence is guaranteed and numerical tests of this strategy look promising

three level incentive schemes using follower s strategies in differential games we deal with three level incentive differential games in which first and second leaders have access not only to slate information but also to information on follower s strategies we derive sufficient conditions for three level incentive schemes using information on follower s strategies in both linear and non linear differential games and show that three level incentive schemes using information on follower s strategies depend on an initial state value

a control theoretic view on incentives the idea of declaring a reward punishment for a decision maker according to his particular choice of action in order to induce certain desired behavior on the part of the decision maker is known as incentive threat this practice is age old however only in recent years have the notions been formalized in the development of a control theoretic view on incentives we first investigate the deterministic version of the incentive problem this reveals the basic simple idea behind the problem it also illustrates the different possibilities introduced by the presence of dynamics and multi follower nature of the problem this is followed by two variants of the stochastic version of the problem where we concentrate on the role of uncertainties relationship to economic literature is also discussed

asymptotic properties of the fisher kernel this letter analyzes the fisher kernel from a statistical point of view the fisher kernel is a particularly interesting method for constructing a model of the posterior probability that makes intelligent use of unlabeled data i e of the underlying data density it is important to analyze and ultimately understand the statistical properties of the fisher kernel to this end we first establish sufficient conditions that the constructed posterior model is realizable i e it contains the true distribution realizability immediately leads to consistency results subsequently we focus on an asymptotic analysis of the generalization error which elucidates the learning curves of the fisher kernel and how unlabeled data contribute to learning we also point out that the squared or log loss is theoretically preferable because both yield consistent estimators to other losses such as the exponential loss when a linear classifier is used together with the fisher kernel therefore this letter underlines that the fisher kernel should be viewed not as a heuristics but as a powerful statistical tool with well controlled statistical properties

on the geometry of feedforward neural network error surfaces many feedforward neural network architectures have the property that their overall input output function is unchanged by certain weight permutations and sign flips in this paper the geometric structure of these equioutput weight space transformations is explored for the case of multilayer perceptron networks with tanh activation functions similar results hold for many other types of neural networks it is shown that these transformations form an algebraic group isomorphic to a direct product of weyl groups results concerning the root spaces of the lie algebras associated with these weyl groups are then used to derive sets of simple equations for minimal sufficient search sets in weight space these sets which take the geometric forms of a wedge and a cone occupy only a minute fraction of the volume of weight space a separate analysis shows that large numbers of copies of a network performance function optimum weight vector are created by the action of the equioutput transformation group and that these copies all lie on the same sphere some implications of these results for learning are discussed

mutual information metric entropy and cumulative relative entropy risk assume p theta theta epsilon theta is a set of probability distributions with a common dominating measure on a complete separable metric space y a state theta epsilon theta is chosen by nature a statistician obtains n independent observations y 1 dots y n from y distributed according to p theta for each time t between 1 and n based on the observations y 1 dots y t 1 the statistician produces an estimated distribution hat p t for p theta and suffers a loss l p theta hat p t the cumulative risk for the statistician is the average total loss up to time n of special interest in information theory data compression mathematical finance computational learning theory and statistical mechanics is the special case when the loss l p theta hat p t is the relative entropy between the true distribution p theta and the estimated distribution hat p t here the cumulative bayes risk from time 1 to n is the mutual information between the random parameter theta and the observations y 1 dots y n new bounds on this mutual information are given in terms of the laplace transform of the hellinger distance between pairs of distributions indexed by parameters in theta from these bounds on the cumulative minimax risk are given in terms of the metric entropy of theta with respect to the hellinger distance the assumptions required for these bounds are very general and do not depend on the choice of the dominating measure they apply to both finite and infinite dimensional theta they apply in some cases where y is infinite dimensional in some cases where y is not compact in some cases where the distributions are not smooth and in some parametric cases where asymptotic normality of the posterior distribution fails

on the asymptotic normality of hierarchical mixtures of experts for generalized linear models in the class of hierarchical mixtures of experts hme models experts in the exponential family with generalized linear mean functions of the form spl psi spl alpha x sup t spl beta are mixed according to a set of local weights called the gating functions depending on the predictor x here spl psi spl middot is the inverse link function we provide regularity conditions on the experts and on the gating functions under which the maximum likelihood method in the large sample limit produces a consistent and asymptotically normal estimator of the mean response the regularity conditions are validated for poisson gamma normal and binomial experts

finite automata and unary languages abstract we prove that o e n log n states are sufficient to simulate an n state 1nfa recognizing a unary language by a 1dfa the lower bound is the same similar tight bounds are shown for the simulation of a 2dfa by a 1dfa and a 1nfa we also show that o n 2 states are sufficient and necessary to simulate an n state 1nfa recognizing a unary language by a 2dfa

optimal simulation of self verifying automata by deterministic automata self verifying automata are a special variant of finite automata with a symmetric kind of nondeterminism we study the conversion of self verifying automata to deterministic automata from a descriptional complexity point of view the main result is the exact cost in terms of the number of states of such a simulation

constructive learning of recurrent neural networks limitations of recurrent cascade correlation and a simple solution it is often difficult to predict the optimal neural network size for a particular application constructive or destructive methods that add or subtract neurons layers connections etc might offer a solution to this problem we prove that one method recurrent cascade correlation due to its topology has fundamental limitations in representation and thus in its learning capabilities it cannot represent with monotone i e sigmoid and hard threshold activation functions certain finite state automata we give a preliminary approach on how to get around these limitations by devising a simple constructive training method that adds neurons during training while still preserving the powerful fully recurrent structure we illustrate this approach by simulations which learn many examples of regular grammars that the recurrent cascade correlation method is unable to learn

constraints on synchronizing oscillator networks this paper investigates the constraints placed on some synchronized oscillator models by their underlying dynamics phase response graphs are used to determine the phase locking behaviors of three oscillator models these results are compared with idealized phase response graphs for single phase and multiple phase systems we find that all three oscillators studied are best suited to operate in a single phase system and that the requirements placed on oscillatory models for operation in a multiple phase system are not compatible with the underlying dynamics of oscillatory behavior for these types of oscillator models

recurrent and feedforward polynomial modeling of coupled time series we present two methods for the prediction of coupled time series the first one is based on modeling the series by a dynamic system with a polynomial format this method can be formulated in terms of learning in a recurrent network for which we give a computationally effective algorithm the second method is a purely feedforward network procedure whose architecture derives from the recurrence relations for the derivatives of the trajectories of a ricatti format dynamic system it can also be used for the modeling of discrete series in terms of nonlinear mappings both methods have been tested successfully against chaotic series

an asymptotic statistical theory of polynomial kernel methods the generalization properties of learning classifiers with a polynomial kernel function are examined in kernel methods input vectors are mapped into a high dimensional feature space where the mapped vectors are linearly separated it is well known that a linear dichotomy has an average generalization error or a learning curve proportional to the dimension of the input space and inversely proportional to the number of given examples in the asymptotic limit however it does not hold in the case of kernel methods since the feature vectors lie on a submanifold in the feature space called the input surface in this letter we discuss how the asymptotic average generalization error depends on the relationship between the input surface and the true separating hyperplane in the feature space where the essential dimension of the true separating polynomial named the class is important we show its upper bounds in several cases and confirm these using computer simulations

three level stackelberg decision problems three level stackelberg decision problems are studied by using the inducible region concept through a systematic derivation it is identified that the leader s control has dual purposes which in general are not separable a special class of problems is then considered where explicit results are obtained

learning coefficient of generalization error in bayesian estimation and vandermonde matrix type singularity the term algebraic statistics arises from the study of probabilistic models and techniques for statistical inference using methods from algebra and geometry sturmfels 2009 the purpose of our study is to consider the generalization error and stochastic complexity in learning theory by using the log canonical threshold in algebraic geometry such thresholds correspond to the main term of the generalization error in bayesian estimation which is called a learning coefficient watanabe 2001a 2001b the learning coefficient serves to measure the learning efficiencies in hierarchical learning models in this letter we consider learning coefficients for vandermonde matrix type singularities by using a new approach focusing on the generators of the ideal which defines singularities we give tight new bound values of learning coefficients for the vandermonde matrix type singularities and the explicit values with certain conditions by applying our results we can show the learning coefficients of three layered neural networks and normal mixture models

approximation and radial basis function networks this paper concerns conditions for the approximation of functions in certain general spaces using radial basis function networks it has been shown in recent papers that certain classes of radial basis function networks are broad enough for universal approximation in this paper these results are considerably extended and sharpened

convergent algorithm for sensory receptive field development an unsupervised developmental algorithm for linear maps is derived which reduces the pixel entropy using the measure introduced in previous work at every update and thus removes pairwise correlations between pixels since the measure of pixel entropy has only a global minimum the algorithm is guaranteed to converge to the minimum entropy map such optimal maps have recently been shown to possess cognitively desirable properties and are likely to be used by the nervous system to organize sensory information the algorithm derived here turns out to be one proposed by goodall for pairwise decorrelation it is biologically plausible since in a neural network implementation it requires only data available locally to a neuron in training over ensembles of two dimensional input signals with the same spatial power spectrum as natural scenes networks develop output neurons with center surround receptive fields similar to those of ganglion cells in the retina some technical issues pertinent to developmental algorithms of this sort such as symmetry fixing are also discussed

function approximation fast convergence neural approach based on spectral analysis we propose a constructive approach to building single hidden layer neural networks for nonlinear function approximation using frequency domain analysis we introduce a spectrum based learning procedure that minimizes the difference between the spectrum of the training data and the spectrum of the network s estimates the network is built up incrementally during training and automatically determines the appropriate number of hidden units this technique achieves similar or better approximation with faster convergence times than traditional techniques such as backpropagation

learning from examples in large neural networks a statistical mechanical theory of learning from examples in layered networks at finite temperature is studied when the training error is a smooth function of continuously varying weights the generalization error falls off asymptotically as the inverse number of examples by analytical and numerical studies of single layer perceptrons we show that when the weights are discrete the generalization error can exhibit a discontinuous transition to perfect generalization for intermediate sizes of the example set the state of perfect generalization coexists with a metastable spin glass state