measuring the strangeness of strange attractors we study the correlation exponent v introduced recently as a characteristic measure of strange attractors which allows one to distinguish between deterministic chaos and random noise the exponent v is closely related to the fractal dimension and the information dimension but its computation is considerably easier its usefulness in characterizing experimental data which stem from very high dimensional systems is stressed algorithms for extracting v from the time series of a single variable are proposed the relations between the various measures of strange attractors and between them and the lyapunov exponents are discussed it is shown that the conjecture of kaplan and yorke for the dimension gives an upper bound for v various examples of finite and infinite dimensional systems are treated both numerically and analytically

maximum entropy density estimation with generalized regularization and an application to species distribution modeling we present a unified and complete account of maximum entropy density estimation subject to constraints represented by convex potential functions or alternatively by convex regularization we provide fully general performance guarantees and an algorithm with a complete convergence proof as special cases we easily derive performance guarantees for many known regularization types including l1 l2 l22 and l2 l22 style regularization we propose an algorithm solving a large and general subclass of generalized maximum entropy problems including all discussed in the paper and prove its convergence our approach generalizes and unifies techniques based on information geometry and bregman divergences as well as those based more directly on compactness our work is motivated by a novel application of maximum entropy to species distribution modeling an important problem in conservation biology and ecology in a set of experiments on real world data we demonstrate the utility of maximum entropy in this setting we explore effects of different feature types sample sizes and regularization levels on the performance of maxent and discuss interpretability of the resulting models

fast exact multiplication by the hessian just storing the hessian h the matrix of second derivatives 2e wi wj of the error e with respect to each pair of weights of a large neural network is difficult since a common use of a large matrix like h is to compute its product with various vectors we derive a technique that directly calculates hv where v is an arbitrary vector to calculate hv we first define a differential operator rv f w r f w rv r 0 note that rv w hv and rv w v and then apply rv to the equations used to compute w the result is an exact and numerically stable procedure for computing hv which takes about as much computation and is about as local as a gradient evaluation we then apply the technique to a one pass gradient calculation algorithm backpropagation a relaxation gradient calculation algorithm recurrent backpropagation and two stochastic gradient calculation algorithms boltzmann machines and weight perturbation finally we show that this technique can be used at the heart of many iterative techniques for computing various properties of h obviating any need to calculate the full hessian

the subspace information criterion for infinite dimensional hypothesis spaces a central problem in learning is selection of an appropriate model this is typically done by estimating the unknown generalization errors of a set of models to be selected from and then choosing the model with minimal generalization error estimate in this article we discuss the problem of model selection and generalization error estimation in the context of kernel regression models e g kernel ridge regression kernel subset regression or gaussian process regression previously a non asymptotic generalization error estimator called the subspace information criterion sic was proposed that could be successfully applied to finite dimensional subspace models sic is an unbiased estimator of the generalization error for the finite sample case under the conditions that the learning target function belongs to a specified reproducing kernel hilbert space rkhs h and the reproducing kernels centered on training sample points span the whole space h these conditions hold only if dim h l where l is the number of training examples therefore sic could be applied only to finite dimensional rkhss in this paper we extend the range of applicability of sic and show that even if the reproducing kernels centered on training sample points do not span the whole space h sic is an unbiased estimator of an essential part of the generalization error our extension allows the use of any rkhss including infinite dimensional ones i e richer function classes commonly used in gaussian processes support vector machines or boosting we further show that when the kernel matrix is invertible sic can be expressed in a much simpler form making its computation highly efficient in computer simulations on ridge parameter selection with real and artificial data sets sic is compared favorably with other standard model selection techniques for instance leave one out cross validation or an empirical bayesian method

hierarchical bayesian models for regularization in sequential learning we show that a hierarchical bayesian modeling approach allows us to perform regularization in sequential learning we identify three inference levels within this hierarchy model selection parameter estimation and noise estimation in environments where data arrive sequentially techniques such as cross validation to achieve regularization or model selection are not possible the bayesian approach with extended kalman filtering at the parameter estimation level allows for regularization within a minimum variance framework a multilayer perceptron is used to generate the extended kalman filter nonlinear measurements mapping we describe several algorithms at the noise estimation level that allow us to implement on line regularization we also show the theoretical links between adaptive noise estimation in extended kalman filtering multiple adaptive learning rates and multiple smoothing regularization coefficients

bayesian nonlinear model selection and neural networks a conjugate prior approach in order to select the best predictive neural network architecture in a set of several candidate networks we propose a general bayesian nonlinear regression model comparison procedure based on the maximization of an expected utility criterion this criterion selects the model under which the training set achieves the highest level of internal consistency through the predictive probability distribution of each model the density of this distribution is computed as the model posterior predictive density and is asymptotically approximated from the assumed gaussian likelihood of the data set and the related conjugate prior density of the parameters the use of such a conjugate prior allows the analytic calculation of the parameter posterior and predictive posterior densities in an empirical bayes like approach this bayesian selection procedure allows us to compare general nonlinear regression models and in particular feedforward neural networks in addition to embedded models as usual with asymptotic comparison tests

bayesian regularization and pruning using a laplace prior standard techniques for improved generalization from neural networks include weight decay and pruning weight decay has a bayesian interpretation with the decay function corresponding to a prior over weights the method of transformation groups and maximum entropy suggests a laplace rather than a gaussian prior after training the weights then arrange themselves into two classes 1 those with a common sensitivity to the data error and 2 those failing to achieve this sensitivity and that therefore vanish since the critical value is determined adaptively during training pruning in the sense of setting weights to exact zeros becomes an automatic consequence of regularization alone the count of free parameters is also reduced automatically as weights are pruned a comparison is made with results of mackay using the evidence framework and a gaussian regularizer

feedforward neural network design with tridiagonal symmetry constraints this paper introduces a pruning algorithm with tridiagonal symmetry constraints for feedforward neural network fann design the algorithm uses a reflection transform applied to the input hidden weight matrix in order to reduce it to its tridiagonal form the designed fann structures obtained by applying the proposed algorithm are compact and symmetrical therefore they are well suited for efficient hardware and software implementations moreover the number of the fann parameters is reduced without a significant loss in performance we illustrate the complexity and performance of the proposed algorithm by applying it as a solution to a nonlinear regression problem we also compare the results of our proposed algorithm with those of the optimal brain damage algorithm

model predictive control for stochastic max plus linear systems with chance constraints the topic of this paper is model predictive control mpc for max plus linear systems with stochastic uncertainties the distribution of which is supposed to be known we consider linear constraints on the inputs and the outputs due to the uncertainties these linear constraints are formulated as probabilistic or chance constraints i e the constraints are required to be satisfied with a predefined probability level the proposed chance constraints can be equivalently rewritten into a max affine i e the maximum of affine terms form if the linear constraints are monotonically nondecreasing as a function of the outputs based on the resulting max affine form two methods are developed for solving the chance constrained mpc problem for stochastic max plus linear systems method 1 uses boole s inequality to convert the multivariate chance constraint into univariate chance constraints for which the probability can be computed more efficiently method 2 employs chebyshev s inequality and transforms the chance constraint into linear constraints on the inputs the simulation results for a production system example show that the two proposed methods are faster than the monte carlo simulation method and yield lower closed loop costs than the nominal mpc method

accelerated solution of multivariate polynomial systems of equations we propose new las vegas randomized algorithms for the solution of a square nondegenerate system of equations with well separated roots the algorithms use oc delta csttn d 2 log d log b arithmetic operations in addition to the operations required to compute the normal form of the boundary monomials modulo the ideal to approximate all real roots of the system as well as all roots lying in a fixed n dimensional box or disc here d is an upper bound on the number of all complex roots of the system e g bezout or bernshtein bound delta is the number of real roots or the roots lying in the box or disc and epsilon 2 b is the required upper bound on the output errors for computing the normal form modulo the ideal the efficient practical algorithms of b mourrain and p trebuchet in proceedings of the international symposium on symbolic and algebraic computation acm new york 2000 pp 231 238 or j c faugere j pure appl algebra 139 1999 pp 61 88 can be applied we also yield the bound oc csttn d 2 log d on the complexity of counting the numbers of all roots in a fixed box disc and all real roots for a large class of inputs and typically in practical computations the factor delta is much smaller than d delta o d this improves by the order of magnitude the known complexity estimates of the order of at least 3n d4 d3 log b or d4 which so far are the record estimates even for the approximation of a single root of a system and for each of the cited counting problems respectively our progress relies on proposing several novel techniques in particular we exploit the structure of matrices associated to a given polynomial system and relate it to the associated linear operators dual space of linear forms and normal forms of polynomials in the quotient algebra furthermore our techniques support the new nontrivial extension of the matrix sign and quadratic inverse power iterations to the case of multivariate polynomial systems where we emulate the recursive splitting of a univariate polynomial into factors of smaller degree

the statistical mechanics of constructive algorithms the storage capacity of multilayer networks with overlapping receptive fields is studied for constructive algorithms using boolean perceptrons as their basic building block which have been investigated within a replica framework the assumption of weak coupling between subsequently constructed perceptrons is verified within a replica symmetric rs ansatz and shown to be negligible in most cases in comparison with correction due to replica symmetry breaking rsb in individual perceptrons the capacities of a tiling like and variants of the upstart algorithm are then calculated within rs and one step rsb with the quenched average taken over the individual units separately for networks with up to k 4000 and k 600 units respectively within this treatment the storage capacity seems to exhibit a power law behaviour in with an exponent n that may depend on the algorithm and the stability however due to finite size effects in k reliable estimates of n could not be extracted nevertheless the results strongly indicate that n should be strictly smaller than 1 within one step rsb whereas within rs the mitchison durbin bound is violated for finite k and n 1 may hold asymptotically

iterative learning algorithms for linear gaussian observation models in this paper we consider a signal parameter estimation problem that is based on a linear model structure and a given setting of statistical models with unknown hyperparameters we consider several combinations of gaussian and laplacian models we develop iterative algorithms based on two typical machine learning methods the evidence based method and the integration based method to deal with the hyperparameters we have applied the proposed algorithms to adaptive prediction and wavelet denoising in linear prediction we show that the proposed algorithms are efficient tools for tackling a difficult problem of adapting simultaneously the order and the coefficients of the predictor in wavelet denoising we show that by using the proposed algorithms the noisy wavelet coefficients are subject to shrinkage and thresholding

brief paper stochastic tube mpc with state estimation an output feedback model predictive control mpc strategy for linear systems with additive stochastic disturbances and probabilistic constraints is proposed given the probability distributions of the disturbance input the measurement noise and the initial state estimation error the distributions of future realizations of the constrained variables are predicted using the dynamics of the plant and a linear state estimator from these distributions a set of deterministic constraints is computed for the predictions of a nominal model the constraints are incorporated in a receding horizon optimization of an expected quadratic cost which is formulated as a quadratic program the constraints are constructed so as to provide a guarantee of recursive feasibility and the closed loop system is stable in a mean square sense all uncertainties in this paper are taken to be bounded in most control applications this gives a more realistic representation of process and measurement noise than the more traditional gaussian assumption

a neural root finder of polynomials based on root moments this letter proposes a novel neural root finder based on the root moment method rmm to find the arbitrary roots including complex ones of arbitrary polynomials this neural root finder nrf was designed based on feedforward neural networks fnn and trained with a constrained learning algorithm cla specifically we have incorporated the a priori information about the root moments of polynomials into the conventional backpropagation algorithm bpa to construct a new cla the resulting nrf is shown to be able to rapidly estimate the distributions of roots of polynomials we study and compare the advantage of the rmm based nrf over the previous root coefficient method based nrf and the traditional muller and laguerre methods as well as the mathematica roots function and the behaviors the accuracies of the resulting root finders and their training speeds of two specific structures corresponding to this fnn root finder the log and the fnn we also analyze the effects of the three controlling parameters p0 p with the cla on the two nrfs theoretically and experimentally finally we present computer simulation results to support our claims

model predictive control of linear systems with multiplicative unbounded uncertainty and chance constraints this paper presents a novel stochastic model predictive control algorithm for linear systems characterized by multiplicative and possibly unbounded model uncertainty probabilistic constraints on the states and inputs are considered and a quadratic cost function is minimized the stochastic control problem and in particular the probabilistic constraints are reformulated in deterministic terms by means of the cantelli inequality so that the on line computational burden of the algorithm is similar to the one of a standard mpc method the properties of the algorithm namely the recursive feasibility and the pointwise convergence of the state are proven by suitably selecting the terminal cost and the constraints on the mean and the variance of the state at the end of the prediction horizon and by considering as additional optimization variables also the mean and the covariance of the state at the beginning of the prediction horizon an extension to deal with the case of expectation rather than probabilistic constraints is reported the numerical issues related to the off line selection of the algorithm s parameters and its on line implementation are discussed simulation results referred to a system with unbounded uncertainty are shown to compare the performances achievable with probabilistic and expectation constraints

constrained least absolute deviation neural networks it is well known that least absolute deviation lad criterion or norm used for estimation of parameters is characterized by robustness i e the estimated parameters are totally resistant insensitive to large changes in the sampled data this is an extremely useful feature especially when the sampled data are known to be contaminated by occasionally occurring outliers or by spiky noise in our previous works we have proposed the least absolute deviation neural network ladnn to solve unconstrained lad problems the theoretical proofs and numerical simulations have shown that the ladnn is lyapunov stable and it can globally converge to the exact solution to a given unconstrained lad problem we have also demonstrated its excellent application value in time delay estimation more generally a practical lad application problem may contain some linear constraints such as a set of equalities and or inequalities which is called constrained lad problem whereas the unconstrained lad can be considered as a special form of the constrained lad in this paper we present a new neural network called constrained least absolute deviation neural network cladnn to solve general constrained lad problems theoretical proofs and numerical simulations demonstrate that the proposed cladnn is lyapunov stable and globally converges to the exact solution to a given constrained lad problem independent of initial values the numerical simulations have also illustrated that the proposed cladnn can be used to robustly estimate parameters for nonlinear curve fitting which is extensively used in signal and image processing

survey constrained model predictive control stability and optimality model predictive control is a form of control in which the current control action is obtained by solving at each sampling instant a finite horizon open loop optimal control problem using the current state of the plant as the initial state the optimization yields an optimal control sequence and the first control in this sequence is applied to the plant an important advantage of this type of control is its ability to cope with hard constraints on controls and states it has therefore been widely applied in petro chemical and related industries where satisfaction of constraints is particularly important because efficiency demands operating points on or close to the boundary of the set of admissible states and controls in this review we focus on model predictive control of constrained systems both linear and nonlinear and discuss only briefly model predictive control of unconstrained nonlinear and or time varying systems we concentrate our attention on research dealing with stability and optimality in these areas the subject has developed in our opinion to a stage where it has achieved sufficient maturity to warrant the active interest of researchers in nonlinear control we distill from an extensive literature essential principles that ensure stability and use these to present a concise characterization of most of the model predictive controllers that have been proposed in the literature in some cases the finite horizon optimal control problem solved on line is exactly equivalent to the same problem with an infinite horizon in other cases it is equivalent to a modified infinite horizon optimal control problem in both situations known advantages of infinite horizon optimal control accrue

characterization of chaotic dynamic behavior in the gas liquid slug flow using directed weighted complex network analysis abstract understanding the nonlinear and complex dynamics underlying the gas liquid slug flow is a significant but challenging problem we systematically carried out gas liquid two phase flow experiments for measuring the time series of flow signals which is studied in terms of the mapping from time series to complex networks in particular we construct directed weighted complex networks dwcn from time series and then associate different aspects of chaotic dynamics with the topological indices of the dwcn and further demonstrate that the dwcn can be exploited to detect unstable periodic orbits of low periods examples using time series from classical chaotic systems are provided to demonstrate the effectiveness of our approach we construct and analyze numbers of dwcns for different gas liquid flow patterns and find that our approach can quantitatively distinguish different experimental gas liquid flow patterns furthermore the dwcn analysis indicates that slug flow shows obvious chaotic behavior and its unstable periodic orbits reflect the intermittent quasi periodic oscillation behavior between liquid slug and large gas slug these interesting and significant findings suggest that the directed weighted complex network can potentially be a powerful tool for uncovering the underlying dynamics leading to the formation of the gas liquid slug flow

the effects of adding noise during backpropagation training on a generalization performance we study the effects of adding noise to the inputs outputs weight connections and weight changes of multilayer feedforward neural networks during backpropagation training we rigorously derive and analyze the objective functions that are minimized by the noise affected training processes we show that input noise and weight noise encourage the neural network output to be a smooth function of the input or its weights respectively in the weak noise limit noise added to the output of the neural networks only changes the objective function by a constant hence it cannot improve generalization input noise introduces penalty terms in the objective function that are related to but distinct from those found in the regularization approaches simulations have been performed on a regression and a classification problem to further substantiate our analysis input noise is found to be effective in improving the generalization performance for both problems however weight noise is found to be effective in improving the generalization performance only for the classification problem other forms of noise have practically no effect on generalization

nonlinear adaptive wavelet control using constructive wavelet networks in this paper an adaptive wavelet network based control approach is proposed for highly nonlinear uncertain dynamical systems wavelet network as a kind of universal approximator has two novel properties orthonormality and multiresolution the orthonormal property ensures that adding a new resolution new wavelets does not affect the existing wavelet network that may have been well tuned in the sequel the online adjustment of the structure of the nonlinear adaptive wavelet controller awc can be done in a constructive manner by gradually increasing the network resolution the multiresolution property on the other hand assures a guaranteed improvement of the approximation precision when a new resolution is added in real life problems we are unable to know the adequate size of a network either a neural network nn or a wavelet network to produce the required approximation precision by virtue of the novel wavelet network properties a coarse or very simple structure can be selected first if the system fails to converge after the elapse of a dwell time a new wavelet resolution is considered to be necessary and added directly in this manner the awc can be easily constructed and tuned from the coarse to finer levels until the performance requirement is satisfied the trial and error way of selecting the network structure which may lead to either an inadequate or a highly redundant structure can be avoided in this paper the proposed adaptive wavelet network is applied first to a class of nonlinear dynamical systems with a partially known model and an affine in input structure then the adaptive wavelet network is applied to a class of nonlinear nonaffine dynamical systems

on the hausdorff dimension of fractal attractors we consider such mappingsx n 1 f xn of an interval into itself for which the attractor is a cantor set for the same class of mappings for which the feigenbaum scaling laws hold we show that the hausdorff dimension of the attractor is universally equal tod 0 538

robust model predictive control via scenario optimization this paper discusses a novel probabilistic approach for the design of robust model predictive control mpc laws for discrete time linear systems affected by parametric uncertainty and additive disturbances the proposed technique is based on the iterated solution at each step of a finite horizon optimal control problem fhocp that takes into account a suitable number of randomly extracted scenarios of uncertainty and disturbances followed by a specific command selection rule implemented in a receding horizon fashion the scenario fhocp is always convex also when the uncertain parameters and disturbance belong to nonconvex sets and irrespective of how the model uncertainty influences the system s matrices moreover the computational complexity of the proposed approach does not depend on the uncertainty disturbance dimensions and scales quadratically with the control horizon the main result in this work is related to the analysis of the closed loop system under receding horizon implementation of the scenario fhocp and essentially states that the devised control law guarantees constraint satisfaction at each step with some a priori assigned probability p while the system s state reaches the target set either asymptotically or in finite time with probability at least p the proposed method may be a valid alternative when other existing techniques either deterministic or stochastic are not directly usable due to excessive conservatism or to numerical intractability caused by lack of convexity of the robust or chance constrained optimization problem

simple exponential family pca principal component analysis pca is a widely used model for dimensionality reduction in this paper we address the problem of determining the intrinsic dimensionality of a general type data population by selecting the number of principal components for a generalized pca model in particular we propose a generalized bayesian pca model which deals with general type data by employing exponential family distributions model selection is realized by empirical bayesian inference of the model we name the model as simple exponential family pca sepca since it embraces both the principal of using a simple model for data representation and the practice of using a simplified computational procedure for the inference our analysis shows that the empirical bayesian inference in sepca formally realizes an intuitive criterion for pca model selection a preserved principal component must sufficiently correlate to data variance that is uncorrelated to the other principal components experiments on synthetic and real data sets demonstrate effectiveness of sepca and exemplify its characteristics for model selection

stochastic receding horizon control of constrained linear systems with state and control multiplicative noise we develop a receding horizon control approach to stochastic linear systems with control and state multiplicative noise that also contain constraints our receding horizon formulation is based upon an on line optimization that utilizes open loop plus linear feedback and is solved as a semi definite programming problem we also provide a characterization of stability performance and constraint satisfaction properties of the receding horizon controlled system under a specific choice of terminal weight and terminal constraint a simple numerical example is used to illustrate the approach

a hybrid learning algorithm combined with generalized rls approach for radial basis function neural networks in this paper a new hybrid learning method for radial basis function neural networks based on generalized recursive least square algorithm is proposed firstly the generalized recursive least square grls model including a general quadratic weight decay term in the energy function for the training of rbf neural networks is described then combined with the grls approach a new hybrid learning method is proposed to meet the design goals improving the generalization ability of the trained network finally experimental results demonstrate that our approach can achieve a significantly improved generalization performance of the rbf networks

improved constrained learning algorithms by incorporating additional functional constraints into neural networks in this paper two improved constrained learning algorithms that are able to guarantee to obtain better generalization performance are proposed these two algorithms are substantially on line learning ones the cost term for the additional functionality of the first improved algorithm is selected based on the first order derivatives of the neural activation at hidden layers while the one of the second improved algorithm is selected based on second order derivatives of the neural activation at hidden layers and output layer in the course of training the cost terms selected from these additional cost functions can penalize the input to output mapping sensitivity or high frequency components included in training data finally theoretical justifications and simulation results are given to verify the efficiency and effectiveness of the two proposed learning algorithms

a practical bayesian framework for backpropagation networks a quantitative and practical bayesian framework is described for learning of mappings in feedforward networks the framework makes possible 1 objective comparisons between solutions using alternative network architectures 2 objective stopping rules for network pruning or growing procedures 3 objective choice of magnitude and type of weight decay terms or additive regularizers for penalizing large weights etc 4 a measure of the effective number of well determined parameters in a model 5 quantified estimates of the error bars on network parameters and on network output and 6 objective comparisons with alternative learning and interpolation models such as splines and radial basis functions the bayesian evidence automatically embodies occam s razor penalizing overflexible and overcomplex models the bayesian approach helps detect poor underlying assumptions in learning models for learning models well matched to a problem a good correlation between generalization ability and the bayesian evidence is obtained

stochastic mpc with inequality stability constraints stochastic uncertainty is a common feature of many control engineering problems and is also present in a wider class of applications e g finance and sustainable development recent work proposed a constrained mpc approach that took explicit account of the distributions of uncertain model parameters but used terminal equality constraints to ensure stability the present paper reformulates the problem in order to relax the stability constraints by invoking appropriate terminal inequalities the application of the proposed strategy and its advantages over earlier work are illustrated by means of a numerical example

adaptive polynomial factorization by coefficient matching a polynomial factorization algorithm is presented which updates all roots simultaneously and efficiently in response to coefficient perturbations the algorithm requires approximately 2n sup 2 complex floating point operations to update all roots of nth order polynomial close to the true root vector the algorithm s convergence rate is quadratic the root update requires only the solution of two sets of structured linear equations and a convolution the algorithm can be used to track the roots of time varying polynomials which is useful for application in adaptive signal processing

subspace information criterion for nonquadratic regularizers model selection for sparse regressors nonquadratic regularizers in particular the l sub 1 norm regularizer can yield sparse solutions that generalize well in this work we propose the generalized subspace information criterion gsic that allows to predict the generalization error for this useful family of regularizers we show that under some technical assumptions gsic is an asymptotically unbiased estimator of the generalization error gsic is demonstrated to have a good performance in experiments with the l sub 1 norm regularizer as we compare with the network information criterion nic and cross validation in relatively large sample cases however in the small sample case gsic tends to fail to capture the optimal model due to its large variance therefore also a biased version of gsic is introduced which achieves reliable model selection in the relevant and challenging scenario of high dimensional data and few samples

identification of non linear systems using empirical data and prior knowledge an optimization approach abstract the choice of a parametric model structure in empirical and semi empirical non linear modeling is usually viewed as an important and critical step however it is known that by augmenting the least squares identification criterion with a term that imposes a penalty on the non smoothness of the model an optimal non parametric model can be found explicitly the optimal non parametric model will depend on the particular form of the penalty which can be looked upon as a priori knowledge or the desired properties of the model in this paper these results are extended in several directions i we show how useful types of prior knowledge other than smoothness can be included as a term in the criterion or as a constraint and how this influences the optimal model ii dynamic models and a general prediction error penalty are considered iii we present a practical numerical procedure for the identification of a close to optimal semi parametric model the numerical approach is motivated by the difficulty of deriving the optimal non parametric model if there are complicated constraints or penalty terms in the criterion and finally iv we discuss determination of the appropriate model complexity through selecting weights on the different penalties on the basis of empirical data since the optimal non parametric model is identified using an augmented optimization criterion and all prior knowledge and desired model properties may be specified through the augmented optimization criterion it is the choice of this criterion that is critical in this approach this elevates the empirical and semi empirical modeling problems to a more transparent and engineering friendly level than is achieved by directly specifying a parametric model structure hence our results shed some more light on modern non linear empirical modeling approaches like radial basis functions splines and neural networks in addition to providing a fundamental insight into the role of different kinds of prior knowledge this high level formulation is also attractive from a practical point of view as it lies closer to engineering thinking and less guesswork is required the flexibility and power of this approach are illustrated with a semi realistic simulation example

an approach to output feedback mpc of stochastic linear discrete time systems in this paper we propose an output feedback model predictive control mpc algorithm for linear discrete time systems affected by a possibly unbounded additive noise and subject to probabilistic constraints in case the noise distribution is unknown the probabilistic constraints on the input and state variables are reformulated by means of the chebyshev cantelli inequality the recursive feasibility is guaranteed the convergence of the state to a suitable neighbor of the origin is proved under mild assumptions and the implementation issues are thoroughly addressed two examples are discussed in detail with the aim of providing an insight into the performance achievable by the proposed control scheme

combinations of weak classifiers to obtain classification systems with both good generalization performance and efficiency in space and time we propose a learning method based on combinations of weak classifiers where weak classifiers are linear classifiers perceptrons which can do a little better than making random guesses a randomized algorithm is proposed to find the weak classifiers they are then combined through a majority vote as demonstrated through systematic experiments the method developed is able to obtain combinations of weak classifiers with good generalization performance and a fast training time on a variety of test problems and real applications theoretical analysis on one of the test problems investigated in our experiments provides insights on when and why the proposed method works in particular when the strength of weak classifiers is properly chosen combinations of weak classifiers can achieve a good generalization performance with polynomial space and time complexity

pattern classification with class probability output network the output of a classifier is usually determined by the value of a discriminant function and a decision is made based on this output which does not necessarily represent the posterior probability for the soft decision of classification in this context it is desirable that the output of a classifier be calibrated in such a way to give the meaning of the posterior probability of class membership this paper presents a new method of postprocessing for the probabilistic scaling of classifier s output for this purpose the output of a classifier is analyzed and the distribution of the output is described by the beta distribution parameters for more accurate approximation of class output distribution the beta distribution parameters as well as the kernel parameters describing the discriminant function are adjusted in such a way to improve the uniformity of beta cumulative distribution function cdf values for the given class output samples as a result the classifier with the proposed scaling method referred to as the class probability output network cpon can provide accurate posterior probabilities for the soft decision of classification to show the effectiveness of the proposed method the simulation for pattern classification using the support vector machine svm classifiers is performed for the university of california at irvine uci data sets the simulation results using the svm classifiers with the proposed cpon demonstrated a statistically meaningful performance improvement over the svm and svm related classifiers and also other probabilistic scaling methods

on the combination of supervised and unsupervised learning abstract the bias variance dilemma is addressed in the context of neural networks a bias constraint based on prior knowledge about the underlying distribution of the data is discussed as a means for reducing the overall error measure of a classifier

discriminative random fields in this research we address the problem of classification and labeling of regions given a single static natural image natural images exhibit strong spatial dependencies and modeling these dependencies in a principled manner is crucial to achieve good classification accuracy in this work we present discriminative random fields drfs to model spatial interactions in images in a discriminative framework based on the concept of conditional random fields proposed by lafferty et al 2001 the drfs classify image regions by incorporating neighborhood spatial interactions in the labels as well as the observed data the drf framework offers several advantages over the conventional markov random field mrf framework first the drfs allow to relax the strong assumption of conditional independence of the observed data generally used in the mrf framework for tractability this assumption is too restrictive for a large number of applications in computer vision second the drfs derive their classification power by exploiting the probabilistic discriminative models instead of the generative models used for modeling observations in the mrf framework third the interaction in labels in drfs is based on the idea of pairwise discrimination of the observed data making it data adaptive instead of being fixed a priori as in mrfs finally all the parameters in the drf model are estimated simultaneously from the training data unlike the mrf framework where the likelihood parameters are usually learned separately from the field parameters we present preliminary experiments with man made structure detection and binary image restoration tasks and compare the drf results with the mrf results

a generalized constraint neural network model associating partially known relationships for nonlinear regressions in an attempt to enhance the neural network technique so that it can evolve from a black box tool into a semi analytical one we propose a novel modeling approach of imposing generalized constraints on a standard neural network we redefine approximation problems by use of a new formalization with the aim of embedding prior knowledge explicitly into the model to the maximum extent a generalized constraint neural network gcnn model has therefore been developed which basically consists of two submodels one is constructed by the standard neural network technique to approximate the unknown part of the target function the other is formed from partially known relationships to impose generalized constraints on the whole model three issues arising after combination of the two submodels are discussed a the better approximation provided by the gcnn model compared with a standard neural network b the identifiability of parameters in the partially known relationships and c the discrepancy in the approximation due to removable singularities in the target function numerical studies of three benchmark problems show important findings that have not previously been reported in the literature significant benefits were observed from using the gcnn model in comparison with a standard neural network

a delayed neural network for solving linear projection equations and its analysis in this paper we present a delayed neural network approach to solve linear projection equations the lyapunov krasovskii theory for functional differential equations and the linear matrix inequality lmi approach are employed to analyze the global asymptotic stability and global exponential stability of the delayed neural network compared with the existing linear projection neural network theoretical results and illustrative examples show that the delayed neural network can effectively solve a class of linear projection equations and some quadratic programming problems

model predictive control for systems with stochastic multiplicative uncertainty and probabilistic constraints abstract robust predictive control handles constrained systems that are subject to stochastic uncertainty but propagating the effects of uncertainty over a prediction horizon can be computationally expensive and conservative this paper overcomes these issues through an augmented autonomous prediction formulation and provides a method of handling probabilistic constraints and ensuring closed loop stability through the use of an extension of the concept of invariance namely invariance with probability p

efficient bayesian multivariate fmri analysis using a sparsifying spatio temporal prior bayesian logistic regression with a multivariate laplace prior is introduced as a multivariate approach to the analysis of neuroimaging data it is shown that by rewriting the multivariate laplace distribution as a scale mixture we can incorporate spatio temporal constraints which lead to smooth importance maps that facilitate subsequent interpretation the posterior of interest is computed using an approximate inference method called expectation propagation and becomes feasible due to fast inversion of a sparse precision matrix we illustrate the performance of the method on an fmri dataset acquired while subjects were shown handwritten digits the obtained models perform competitively in terms of predictive performance and give rise to interpretable importance maps estimation of the posterior of interest is shown to be feasible even for very large models with thousands of variables

on different facets of regularization theory this review provides a comprehensive understanding of regularization theory from different perspectives emphasizing smoothness and simplicity principles using the tools of operator theory and fourier analysis it is shown that the solution of the classical tikhonov regularization problem can be derived from the regularized functional defined by a linear differential integral operator in the spatial fourier domain state of the art research relevant to the regularization theory is reviewed covering occam s razor minimum length description bayesian theory pruning algorithms informational entropy theory statistical learning theory and equivalent regularization the universal principle of regularization in terms of kolmogorov complexity is discussed finally some prospective studies on regularization theory and beyond are suggested

preventing over fitting during model selection via bayesian regularisation of the hyper parameters while the model parameters of a kernel machine are typically given by the solution of a convex optimisation problem with a single global optimum the selection of good values for the regularisation and kernel parameters is much less straightforward fortunately the leave one out cross validation procedure can be performed or a least approximated very efficiently in closed form for a wide variety of kernel learning methods providing a convenient means for model selection leave one out cross validation based estimates of performance however generally exhibit a relatively high variance and are therefore prone to over fitting in this paper we investigate the novel use of bayesian regularisation at the second level of inference adding a regularisation term to the model selection criterion corresponding to a prior over the hyper parameter values where the additional regularisation parameters are integrated out analytically results obtained on a suite of thirteen real world and synthetic benchmark data sets clearly demonstrate the benefit of this approach

income prediction in the agrarian sector using product unit neural networks abstract european union financial subsidies in the agrarian sector are directly related to maintaining a sustainable farm income so its determination using for example the farm gross margin is a basic element in agrarian programs for sustainable development using this tool it is possible the identification of the agrarian structures that need financial support and to what extent it is needed however the process of farm gross margin determination is complicated and expensive because it is necessary to find the value of all the inputs consumed and outputs produced considering the circumstances mentioned the objectives of this research were to 1 select a representative and reduced set of easy to collect descriptive variables to estimate the gross margin of a group of olive tree farms in andalusia 2 investigate if artificial neural network models ann with two different types of basis functions sigmoidal and product units could effectively predict the gross margin of olive tree farms 3 compare the effectiveness of multiple linear quadratic and robust regression models versus ann and 4 validate the best mathematical model obtained for gross margin prediction by analysing realistic farm and farmer scenarios results from ann models specially the product unit ones have provided the most accurate gross margin predictions

determining the number of real roots of polynomials through neural networks the ability of feedforward neural networks to identify the number of real roots of univariate polynomials is investigated furthermore their ability to determine whether a system of multivariate polynomial equations has real solutions is examined on a problem of determining the structure of a molecule the obtained experimental results indicate that neural networks are capable of performing this task with high accuracy even when the training set is very small compared to the test set

convex approximations of chance constrained programs we consider a chance constrained problem where one seeks to minimize a convex objective over solutions satisfying with a given close to one probability a system of randomly perturbed convex constraints this problem may happen to be computationally intractable our goal is to build its computationally tractable approximation i e an efficiently solvable deterministic optimization program with the feasible set contained in the chance constrained problem we construct a general class of such convex conservative approximations of the corresponding chance constrained problem moreover under the assumptions that the constraints are affine in the perturbations and the entries in the perturbation vector are independent of each other random variables we build a large deviation type approximation referred to as bernstein approximation of the chance constrained problem this approximation is convex and efficiently solvable we propose a simulation based scheme for bounding the optimal value in the chance constrained problem and report numerical experiments aimed at comparing the bernstein and well known scenario approximation approaches finally we extend our construction to the case of ambiguous chance constrained problems where the random perturbations are independent with the collection of distributions known to belong to a given convex compact set rather than to be known exactly while the chance constraint should be satisfied for every distribution given by this set

modified constrained learning algorithms incorporating additional functional constraints into neural networks in this paper two modified constrained learning algorithms are proposed to obtain better generalization performance and faster convergence rate the additional cost terms of the first algorithm are selected based on the first order derivatives of the activation functions of the hidden neurons and the second order derivatives of the activation functions of the output neurons while the additional cost terms of the second one are selected based on the first order derivatives of the activation functions of the output neurons and the second order derivatives of the activation functions of the hidden neurons in the course of training the additional cost terms of the proposed algorithms can penalize the input to output mapping sensitivity and the high frequency components simultaneously so that the better generalization performance can be obtained finally theoretical justifications and simulation results are given to verify the efficiency and effectiveness of our proposed learning algorithms

evolutionary algorithm based learning of fuzzy neural networks part 2 recurrent fuzzy neural networks fuzzy neural networks fnn as opposed to neuro fuzzy systems whose main task is to process numerical relationships can process both numerical measurement based information and perception based information in spite of great importance of fuzzy feed forward and recurrent neural networks for solving wide range of real world problems today there are no effective training algorithm for them currently there are two approaches for training of fnn first approach is based on application of the level sets of fuzzy numbers and the back propagation bp algorithm the second approach involves using evolutionary algorithms to minimize error function and determine the fuzzy connection weights and biases the method based on the second approach was proposed by the authors and published in part 1 of this paper r a aliev b fazlollahi r vahidov genetic algorithm based learning of fuzzy neural networks part 1 feed forward fuzzy neural networks fuzzy sets and systems 118 2001 351 358 in contrast to the bp and other supervised learning algorithms evolutionary algorithms do not require nor use information about differentials and hence they are most effective in case where the derivative is very difficult to obtain or even unavailable however the main deficiency of the existing fnn based on the feed forward architecture is its adherence to static problems in case of dynamic or temporal problems there is a need for recurrent fuzzy neural networks rfnn designing efficient training algorithms for rfnn has recently become an active research direction in this paper we propose an effective differential evolution optimization deo based learning algorithm for rfnn with fuzzy inputs fuzzy weights and biases and fuzzy outputs the effectiveness of the proposed method is illustrated through simulation of benchmark forecasting and identification problems and comparisons with the existing methods the suggested approach has also been used for real applications in an oil refinery plant for petrol production forecasting

chance constrained h control for a class of time varying systems with stochastic nonlinearities the finite horizon case in this paper a new finite horizon h control problem is considered for a class of time varying systems with stochastic nonlinearities measurements degradation and chance constraints the purpose of the addressed problem is to design the time varying controller such that the closed loop system satisfies the prespecified h disturbance attenuation requirement and certain chance constraints on the controlled output vector zk i e the probability of the controlled output zk belonging to a given set is larger than a prescribed value a modified maximum volume inscribed ellipsoid mvie method is employed to convert the chance constraint into some tractable inequalities which could be conveniently handled by the recursive linear matrix inequality rlmi approach then by using stochastic control analysis method sufficient conditions are derived for the existence of the desired multi objective controller and furthermore the gains of the desired controllers are characterized by means of rlmis finally two illustrative examples and a practical system are proposed to highlight the effectiveness and applicability of the presented controller design technology

a new image thresholding method based on gaussian mixture model in this paper an efficient approach to search for the global threshold of image using gaussian mixture model is proposed firstly a gray level histogram of an image is represented as a function of the frequencies of gray level then to fit the gaussian mixtures to the histogram of image the expectation maximization em algorithm is developed to estimate the number of gaussian mixture of such histograms and their corresponding parameterization finally the optimal threshold which is the average of these gaussian mixture means is chosen and the experimental results show that the new algorithm performs better

zeroing polynomials using modified constrained neural network approach this paper proposes new modified constrained learning neural root finders nrfs of polynomial constructed by backpropagation network bpn the technique is based on the relationships between the roots and the coefficients of polynomial as well as between the root moments and the coefficients of the polynomial we investigated different resulting constrained learning algorithms clas based on the variants of the error cost functions ecfs in the constrained bpn and derived a new modified cla mcla and found that the computational complexities of the cla and the mcla based on the root moment method rmm are the order of polynomial and that the mcla is simpler than the cla further we also discussed the effects of the different parameters with the cla and the mcla on the nrfs in particular considering the coefficients of the polynomials involved in practice to possibly be perturbed by noisy sources thus we also evaluated and discussed the effects of noises on the two nrfs finally to demonstrate the advantage of our neural approaches over the nonneural ones a series of simulating experiments are conducted

on chaotic graphs a different approach for characterizing aperiodic dynamics fractal worlds with limited connectivity are the topological result of growing graphs from chaotic series we show how this model presents original characteristics which cannot be detected by means of the standard network descriptors in detail intrinsic inaccessibility to the fully connected configuration is demonstrated to be a universal feature associated with this family of graphs and strictly related to the fractality of a specific chaotic source here we discuss the potential of our model to be a generator of fractal graphs and also a self consistent tool for differentiating chaotic dynamics from stochastic processes

ambiguities in recurrence based complex network representations of time series recently different approaches have been proposed for studying basic properties of time series from a complex network perspective in this work the corresponding potentials and limitations of networks based on recurrences in phase space are investigated in some detail we discuss the main requirements that permit a feasible system theoretic interpretation of network topology in terms of dynamically invariant phase space properties possible artifacts induced by disregarding these requirements are pointed out and systematically studied finally a rigorous interpretation of the clustering coefficient and the betweenness centrality in terms of invariant objects is proposed

a constructive approach for finding arbitrary roots of polynomials by neural networks this paper proposes a constructive approach for finding arbitrary real or complex roots of arbitrary real or complex polynomials by multilayer perceptron network mlpn using constrained learning algorithm cla which encodes the a priori information of constraint relations between root moments and coefficients of a polynomial into the usual bp algorithm bpa moreover the root moment method rmm is also simplified into a recursive version so that the computational complexity can be further decreased which leads the roots of those higher order polynomials to be readily found in addition an adaptive learning parameter with the cla is also proposed in this paper an initial weight selection method is also given finally several experimental results show that our proposed neural connectionism approaches with respect to the nonneural ones are more efficient and feasible in finding the arbitrary roots of arbitrary polynomials

finding the embedding dimension and variable dependencies in time series we present a general method the test which establishes functional dependencies given a sequence of measurements the approach is based on calculating conditional probabilities from vector component distances imposing the requirement of continuity of the underlying function the obtained values of the conditional probabilities carry information on the embedding dimension and variable dependencies the power of the method is illustrated on synthetic time series with different time lag dependencies and noise levels and on the sunspot data the virtue of the method for preprocessing data in the context of feedforward neural networks is demonstrated also its applicability for tracking residual errors in output units is stressed

a modified particle swarm optimizer with dynamic adaptation this paper proposes a modified particle swarm optimization algorithm with dynamic adaptation in this algorithm a modified velocity updating formula of the particle is used where the randomness in the course of updating particle velocity is relatively decreased and the inertia weight of each particle is different moreover this algorithm introduces two parameter describing the evolving state of the algorithm the evolution speed factor and aggregation degree factor by analyzing the influence of two parameters on the pso search ability a new strategy is presented that the inertia weight dynamically changes based on the run and evolution state in the strategy the inertia weight is given by a function of evolution speed factor and aggregation degree factor and the value of inertia weight is dynamically adjusted according to the evolution speed and aggregation degree the feature of the proposed algorithm is analyzed and several testing functions are performed in simulation study experimental results show that the proposed algorithm remarkably improves the ability of pso to jump out of the local optima and significantly enhance the convergence precision

convergence of stochastic nonlinear systems and implications for stochastic model predictive control the stability of stochastic model predictive control mpc subject to additive disturbances is often demonstrated in the literature by constructing lyapunov like inequalities that ensure closed loop performance bounds and boundedness of the state but tight ultimate bounds for the state and non conservative performance bounds are typically not determined in this work we use an input to state stability property to find conditions that imply convergence with probability 1 of a disturbed nonlinear system to a minimal robust positively invariant set we discuss implications for the convergence of the state and control laws of stochastic mpc formulations and we prove convergence results for several existing stochastic mpc formulations for linear and nonlinear systems

a new partitioning neural network model for recursively finding arbitrary roots of higher order arbitrary polynomials a new partitioning feedforward neural network fnn root finder model for recursively finding the arbitrary including complex roots of higher order arbitrary polynomials is proposed in this paper moreover an efficient complex version of constrained learning algorithm cla which incorporates the a priori information i e the constrained relation between the original polynomial coefficients and the remaining polynomial coefficients plus the partitioned roots out from the original polynomial is constructed to train the corresponding partitioning neural root finder network for finding the arbitrary roots of arbitrary polynomials finally the experimental results are given to show the efficiency and effectiveness of our proposed neural model with respect to traditional non neural root finders

an algebraic approach to the estimation of the order of fir filters from complete and partial magnitude and phase specifications the problems addressed by this paper is the following given a set of measurements over the range of normalized frequencies thetas1 thetas2 on the magnitude and or phase of a real fir but otherwise unknown filter to estimate the order of the fir filter the range thetas1 thetas2 may be partial or it may cover the entire range of frequencies the purpose of the paper is to propose a new algebraic approach to solve the above collection of problems specific new results include fir order estimation from partial or complete noiseless measurements for a real system from a phase alone from b magnitude alone not necessarily piece wise constant and from c joint magnitude and phase the proposed approach is not only capable of dealing with specifications that go beyond the conventional formulas for the standard piece wise constant magnitude fir filter order estimation but it also furnishes a nexus for order estimation from phase or group delay specifications areas which have remained hitherto unaddressed the approach is based on the use of root moments in this context the novel concept of fractional root moments is used in a key fashion to provide an estimate on the number of zeros inside the unit circle open problems and new directions of exploration and research are mentioned in the body of the paper

simplifying neural networks by soft weight sharing one way of simplifying neural networks so they generalize better is to add an extra term to the error function that will penalize complexity simple versions of this approach include penalizing the sum of the squares of the weights or penalizing the number of nonzero weights we propose a more complicated penalty term in which the distribution of weight values is modeled as a mixture of multiple gaussians a set of weights is simple if the weights have high probability density under the mixture model this can be achieved by clustering the weights into subsets with the weights in each cluster having very similar values since we do not know the appropriate means or variances of the clusters in advance we allow the parameters of the mixture model to adapt at the same time as the network learns simulations on two different problems demonstrate that this complexity term is more effective than previous complexity terms

finding the zeros of linear phase fir frequency sampling digital filters the need for finding the zeros of fir filters often arises in digital signal processing applications if we want to convert linear phase finite impulse response fir filters into minimum phase fir filters 1 2 the problem of finding the zeros of the filter is encountered this problem has been addressed in 3 but not solved here we introduce the idea of using frequency sampling filters for obtaining its zeros the advantage of using this kind of filter resides in the fact that all the zeros on the unit circle are known furthermore the zeros off the unit circle can be obtained with some simple processing this type of filter is especially useful when high order narrow band filters are involved finding the zeros of very high order equirriple low pass filters can be a difficult if not impossible task

using neural networks to model conditional multivariate densities neural network outputs are interpreted as parameters of statistical distributions this allows us to fit conditional distributions in which the parameters depend on the inputs to the network we exploit this in modeling multivariate data including the univariate case in which there may be input dependent e g time dependent correlations between output components this provides a novel way of modeling conditional correlation that extends existing techniques for determining input dependent local error bars

a least squares approach to direct importance estimation we address the problem of estimating the ratio of two probability density functions which is often referred to as the importance the importance values can be used for various succeeding tasks such as covariate shift adaptation or outlier detection in this paper we propose a new importance estimation method that has a closed form solution the leave one out cross validation score can also be computed analytically therefore the proposed method is computationally highly efficient and simple to implement we also elucidate theoretical properties of the proposed method such as the convergence rate and approximation error bounds numerical experiments show that the proposed method is comparable to the best existing method in accuracy while it is computationally more efficient than competing approaches

a new approach for function approximation incorporating adaptive particle swarm optimization and a priori information in this paper a new approach coupling adaptive particle swarm optimization apso and a priori information for function approximation problem is proposed to obtain better generalization performance and faster convergence rate it is well known that gradient based learning algorithms such as backpropagation bp algorithm have good ability of local search whereas pso has good ability of global search therefore in the new approach first apso encoding the first order derivative information of the approximated function is applied to train network to near global minima second with the connection weights produced by apso the network is trained with a gradient based algorithm due to combining apso with local search algorithm and considering a priori information the new approach has better generalization performance and convergence rate than traditional learning ones finally simulation results are given to verify the efficiency and effectiveness of the proposed approach

a dynamical system perspective of structural learning with forgetting structural learning with forgetting is an established method of using laplace regularization to generate skeletal artificial neural networks we develop a continuous dynamical system model of regularization in which the associated regularization parameter is generalized to be a time varying function analytic results are obtained for a laplace regularizer and a quadratic error surface by solving a different linear system in each region of the weight space this model also enables a comparison of laplace and gaussian regularization both of these regularizers have a greater effect in weight space directions which are less important for minimization of a quadratic error function however for the gaussian regularizer the regularization parameter modifies the associated linear system eigenvalues in contrast to its function as a control input in the laplace case this difference provides additional evidence for the superiority of the laplace over the gaussian regularizer

maximum likelihood neural approximation in presence of additive colored noise in many practical situations the noise samples may be correlated in this case the estimation of noise parameters can be used to improve the approximation estimation of the noise structure can also be used to find a stopping criterion in constructive neural networks to avoid overfitting a network construction procedure must be stopped when residual can be considered as noise the knowledge on the noise may be used for whitening the residual so that a correlation hypothesis test determines if the network growing must be continued or not in this paper supposing a gaussian noise model we study the problem of multi output nonlinear regression using mlp when the noise in each output is a correlated autoregressive time series and is spatially correlated with other output noises we show that the noise parameters can be determined simultaneously with the network weights and used to construct an estimator with a smaller variance and so to improve the network generalization performance moreover if a constructive procedure is used to build the network the estimated parameters may be used to stop the procedure