rates of convergence for a class of global stochastic optimization algorithms inspired and motivated by the recent advances in simulated annealing algorithms this paper analyzes the convergence rates of a class of recursive algorithms for global optimization via monte carlo methods by using perturbed liapunov function methods stability results of the algorithms are established then the rates of convergence are ascertained by examining the asymptotic properties of suitably scaled estimation error sequences

applying experimental design and regression splines to high dimensional continuous state stochastic dynamic programming in stochastic dynamic programming sdp with continuous state and decision variables the future value function is computed at discrete points in the state spac e interpolation can be used to approximate the values of the future value function between these discrete points however for large dimensional problems the number of discrete points required to obtain a good approximation of the future value function can be prohibitively large statistical methods of experimental design and function estimation may be employed to overcome this curse of dimensionality in this paper we describe a method for estimating the future value function by multivariate adaptive regression splines mars fit over a discretization scheme based on orthogonal array oa experimental designs because orthogonal arrays only grow polynomially in the state space dimension our oa mars method is accurately able to solve higher dimensional sdp problems than previously possible to our knowledge the most efficient method published prior to this work employs tensor product cubic splines to approximate the future value function johnson et al 1993 the computational advantages of oa ma rs are demonstrated in comparisons with the method using tensor product cubic splines for applications of an inventory forecasting sdp with up to nine state variables computed on a small workstation in particular the storage of an adequate tensor product cubic spline for six dimensions exceeds the memory of our workstation and the run time for an accurate oa mars sdp solution would be at least an order of magnitude faster than using tensor product cubic splines for higher than six dimensions

combination of epsilon and ritz methods with multiscaling basis for solving a class of fractional optimal control problems abstract this paper presents an approximate direct method for solving a class of fractional optimal control problems by combining the epsilon and ritz methods and utilizing multiscaling basis functions by applying the epsilon method we transform the given optimal control problem into a variational problem then we solve the variational problem approximately by the ritz direct method with multiscaling polynomial basis functions the choice of multiscaling basis functions provides a high rate of convergence for the ritz method the convergence of the derived approximate minimal to the exact one is extensively discussed at the end some illustrative test examples are included to demonstrate the efficiency of the technique

bounds on rates of variable basis and neural network approximation the tightness of bounds on rates of approximation by feedforward neural networks is investigated in a more general context of nonlinear approximation by variable basis functions tight bounds on the worst case error in approximation by linear combinations of n elements of an orthonormal variable basis are derived

regularization theory and neural networks architectures we had previously shown that regularization principles lead to approximation schemes that are equivalent to networks with one layer of hidden units called regularization networks in particular s

an approach to the conditional error rate principle with nuisance parameters summary in the presence of nuisance parameters the conditional error rate principle is difficult to apply because of the dependency of the conditional error function of the preplanned test on nuisance parameters to use the conditional error rate principle with nuisance parameters we propose to search among tests that guarantee overall error control for the test that maximizes a weighted combination of the conditional error rates over possible values of the nuisance parameters we show that the optimization problem that defines such a test can be solved efficiently by existing algorithms

a new class of incremental gradient methods for least squares problems the least mean squares lms method for linear least squares problems differs from the steepest descent method in that it processes data blocks one by one with intermediate adjustment of the parameter vector under optimization this mode of operation often leads to faster convergence when far from the eventual limit and to slower sublinear convergence when close to the optimal solution we embed both lms and steepest descent as well as other intermediate methods within a one parameter class of algorithms and we propose a hybrid class of methods that combine the faster early convergence rate of lms with the faster ultimate linear convergence rate of steepest descent these methods are well suited for neural network training problems with large data sets furthermore these methods allow the effective use of scaling based for example on diagonal or other approximations of the hessian matrix

numerical solution of continuous state dynamic programs using linear and spline interpolation this paper demonstrates that the computational effort required to develop numerical solutions to continuous state dynamic programs can be reduced significantly when cubic piecewise polynomial functions rather than tensor product linear interpolants are used to approximate the value function tensor product cubic splines represented in either piecewise polynomial or b spline form and multivariate hermite polynomials are considered computational savings are possible because of the improved accuracy of higher order functions and because the smoothness of higher order functions allows efficient quasi newton methods to be used to compute optimal decisions the use of the more efficient piecewise polynomial form of the spline was slightly superior to the use of hermite polynomials for the test problem and easier to program in comparison to linear interpolation use of splines in piecewise polynomial form reduced the cpu time to obtain results of equivalent accuracy by a factor of 250 330 for a stochastic

error estimates for approximate optimization by the extended ritz method an alternative to the classical ritz method for approximate optimization is investigated in the extended ritz method sets of admissible solutions are approximated by their intersections with sets of linear combinations of all n tuples of functions from a given basis this alternative scheme called variable basis approximation includes functions computable by trigonometric polynomials with free frequencies free node splines neural networks and other nonlinear approximating families estimates of rates of approximate optimization by the extended ritz method are derived upper bounds on rates of convergence of suboptimal solutions to the optimal one are expressed in terms of the degree n of variable basis functions the modulus of continuity of the functional to be minimized the modulus of tikhonov well posedness of the problem and certain norms tailored to the type of basis the results are applied to convex best approximation and to kernel methods in machine learning

a simple lemma on greedy approximation in hilbert space and convergence rates for projection pursuit regression and neural network training a general convergence criterion for certain iterative sequences in hilbert space is presented for an important subclass of these sequences estimates of the rate of convergence are given under very mild assumptions these results establish an 0 1 f4n nonsampling convergence rate for projection pursuit regression and neural network training where n represents the number of ridge functions neurons or coefficients in a greedy basis expansion

nonlinear black box models in system identification mathematical foundations we discuss several aspects of the mathematical foundations of the nonlinear black box identification problem we shall see that the quality of the identification procedure is always a result of a certain trade off between the expressive power of the model we try to identify the larger the number of parameters used to describe the model the more flexible is the approximation and the stochastic error which is proportional to the number of parameters a consequence of this trade off is the simple fact that a good approximation technique can be the basis of a good identification algorithm from this point of view we consider different approximation methods and pay special attention to spatially adaptive approximants we introduce wavelet and neuron approximations and show that they are spatially adaptive then we apply the acquired approximation experience to estimation problems finally we consider some implications of these theoretical developments for the practically implemented versions of the spatially adaptive algorithms

convergent ritz approximations of the set of stabilizing controllers abstract a necessary and sufficient condition for linear exponential sums to be dense in l p is derived under this condition a ritz approximation converges which has been proposed to solve linear control problems state space representations of orthonormal basis functions for multivariable systems are given

minimization of error functionals over variable basis functions generalized tikhonov well posedness is investigated for the problem of minimization of error functionals over admissible sets formed by variable basis functions i e linear combinations of a fixed number of elements chosen from a given basis without a prespecified ordering for variable basis functions of increasing complexity rates of decrease of infima of error functionals are estimated upper bounds are derived on such rates which do not exhibit the curse of dimensionality with respect to the number of variables of admissible functions consequences are considered for boolean functions and decision trees

numerical solutions to the witsenhausen counterexample by approximating networks approximate solutions to the witsenhausen counterexample 1968 are derived by constraining the unknown control functions to take on fixed structures containing free parameters to be optimized such structures are given by nonlinear approximating networks i e linear combinations of parametrized basis functions that benefit by density properties in normed linear spaces this reduces the original functional problem to a nonlinear programming one which is solved via stochastic approximation the method yields lower values of the costs than the ones achieved so far in the literature and most of all provides rather a complete overview of the shapes of the optimal control functions when the two parameters that characterize the witsenhausen counterexample vary one hidden layer neural networks are chosen as approximating networks

set contractions and ball contractions in lp spaces abstract consider the space l p 1 p where is a finite measure space defined and by 2 max 1 p p 1 p and 2 p 2 p the following relationship between set contractions and ball contractions in separable l p spaces is proved if t l p l p is a k set contraction respectively set condensing mapping then t is a k ball contraction respectively ball condensing mapping if t l p l p is a k ball contraction then t is a k set contraction furthermore these constants and are the best possible

neural approximations for multistage optimal control of nonlinear stochastic systems two main approximations are used to solve a nonlinear quadratic gaussian lqg optimal control problem the control law is assigned a given structure in which a finite number of parameters have to be determined to minimize the cost function the chosen structure is that of a multilayer feedforward neural network and the control law is given a limited memory the errors resulting front both assumptions are discussed simulation results show that the proposed method may constitute a simple and effective tool for solving to a sufficient degree of accuracy optimal control problems traditionally regarded as difficult ones

determining finite volume elements for the 2d navier stokes equations we consider the 2d navier stokes equations on a square with periodic boundary conditions dividing the square into n equal subsquares we show that if the asymptotic behavior of the average of solutions on these subsquares finite volume elements is known then the large time behavior of the solution itself is completely determined provided n is large enough we also establish a rigorous upper bound for n needed to determine the solutions to the navier stokes equation in terms of the physical parameters of the problem

functional optimization through semilocal approximate minimization an approach based on semilocal approximation is introduced for the solution of a general class of operations research problems such as markovian decision problems multistage optimal control and maximum likelihood estimation because it is extremely hard to derive analytical solutions that minimize the cost in most instances of the problem we must look for approximate solutions here it is shown that good solutions can be obtained with a moderate computational effort by exploiting properties of semilocal approximation through kernel models and efficient sampling of the state space the convergence of the proposed method called semilocal approximate minimization slam is discussed and the consistency of the solution is derived simulation results show the efficiency of slam also through its application to a classic operations research problem i e inventory forecasting

regularization techniques and suboptimal solutions to optimization problems in learning from data various regularization techniques are investigated in supervised learning from data theoretical features of the associated optimization problems are studied and sparse suboptimal solutions are searched for rates of approximate optimization are estimated for sequences of suboptimal solutions formed by linear combinations of n tuples of computational units and statistical learning bounds are derived as hypothesis sets reproducing kernel hilbert spaces and their subsets are considered

an integral upper bound for neural network approximation complexity of one hidden layer networks is studied using tools from nonlinear approximation and integration theory for functions with suitable integral representations in the form of networks with infinitely many hidden units upper bounds are derived on the speed of decrease of approximation error as the number of network units increases these bounds are obtained for various norms using the framework of bochner integration results are applied to perceptron networks

set contractions and ball contractions in some classes of spaces abstract let x be a banach space d an arbitrary subset of x and t d x a set condensing k set contractive mapping the main result of this paper is if x lp 1 p then t is ball condensing k ball contractive it is also shown that this result does not hold for x lp 0 1 1 p

inertial manifolds associated to partly dissipative reaction diffusion systems abstract in a previous work we established the existence of finite dimensional attractors for partly dissipative reaction diffusion systems of two types systems with a polynomial growth nonlinearity and systems admitting an invariant region the aim of this article is to show that these problems possess inertial manifolds in space dimension n 2 these inertial manifolds contain the universal attractor attract exponentially all the solutions as time goes to infinity and due to the weak dissipation are here infinite dimensional our results are applied to several classical systems borrowed from mathematical biology physics and chemistry

hinging hyperplanes for regression classification and function approximation a hinge function y h x consists of two hyperplanes continuously joined together at a hinge in regression prediction classification pattern recognition and noiseless function approximation use of sums of hinge functions gives a powerful and efficient alternative to neural networks with computation times several orders of magnitude less than is obtained by fitting neural networks with a comparable number of parameters a simple and effective method for finding good hinges is presented

successive galerkin approximation algorithms for nonlinear optimal and robust control nonlinear optimal control and nonlinear h infinity control are two of the most significant paradigms in nonlinear systems theory unfortunately these problems require the solution of hamilton jacobi equations which are extremely difficult to solve in practice to make matters worse approximation techniques for these equations are inherently prone to the so called curse of dimensionality while there have been many attempts to approximate these equations solutions resulting in closed loop control with well defined stability and robustness have remained elusive this paper describes a recent breakthrough in approximating the hamilton jacobi bellman and hamilton jacobi isaacs equations successive approximation and galerkin approximation methods are combined to derive a novel algorithm that produces stabilizing closed loop control laws with well defined stability regions in addition we show how the structure of the algorithm can be exploited to reduce the amount of computation from exponential to po

suboptimal solutions to team optimization problems with stochastic information structure existence uniqueness and approximations of smooth solutions to team optimization problems with stochastic information structure are investigated suboptimal strategies made up of linear combinations of basis functions containing adjustable parameters are considered estimates of their accuracies are derived by combining properties of the unknown optimal strategies with tools from nonlinear approximation theory the estimates are obtained for basis functions corresponding to sinusoids with variable frequencies and phases gaussians with variable centers and widths and sigmoidal ridge functions the theoretical results are applied to a problem of optimal production in a multidivisional firm for which numerical simulations are presented

determination of the solutions of the navier stokes equations by a set of nodal values we consider the navier stokes equations of a viscous incompresible fluid and we want to see to what extent these solutions can be determined by a discrete set of nodal values of these solutions the results presented here are exact results and not approximate ones we show in several cases that the solutions are entirely determined by their values on a discrete set provided this set contains enough points and these points are sufficiently densely distributed in a sense described in the article two typical results are the following ones two stationary solutions coincide if they coincide on a set sufficiently dense but finite similarly if the large time behavior of the solutions to the navier stokes equations is known on an appropriate discrete set then the large time behavior of the solution itself is totally determined

on the number of determining nodes for the 2d navier stokes equations abstract it is known that the solutions of the 2d navier stokes equations in bounded domains are determined by a finite discrete set of nodal values that is if the large time behavior of the solutions to the navier stokes equations is known on an appropriate finite discrete set then the large time behavior of the solution itself is totally determined here an upper bound is rigorously established for the number of nodes needed to determine the solutions of the navier stokes equations in two dimensions with periodic boundary conditions

design of asymptotic estimators an approach based on neural networks and nonlinear programming a methodology to design state estimators for a class of nonlinear continuous time dynamic systems that is based on neural networks and nonlinear programming is proposed the estimator has the structure of a luenberger observer with a linear gain and a parameterized in general nonlinear function whose argument is an innovation term representing the difference between the current measurement and its prediction the problem of the estimator design consists in finding the values of the gain and of the parameters that guarantee the asymptotic stability of the estimation error toward this end if a neural network is used to take on this function the parameters i e the neural weights are chosen together with the gain by constraining the derivative of a quadratic lyapunov function for the estimation error to be negative definite on a given compact set it is proved that it is sufficient to impose the negative definiteness of such a derivative only on a suitably dense grid of sampling points the gain is determined by solving a lyapunov equation the neural weights are searched for via nonlinear programming by minimizing a cost penalizing grid point constraints that are not satisfied techniques based on low discrepancy sequences are applied to deal with a small number of sampling points and hence to reduce the computational burden required to optimize the parameters numerical results are reported and comparisons with those obtained by the extended kalman filter are made

comparison of worst case errors in linear and neural network approximation sets of multivariable functions are described for which worst case errors in linear approximation are larger than those in approximation by neural networks a theoretical framework for such a description is developed in the context of nonlinear approximation by fixed versus variable basis functions comparisons of approximation rates are formulated in terms of certain norms tailored to sets of basis functions the results are applied to perceptron networks

an incremental gradient projection method with momentum term and adaptive stepsize rule we consider an incremental gradient method with momentum term for minimizing the sum of continuously differentiable functions this method uses a new adaptive stepsize rule that decreases the stepsize whenever sufficient progress is not made we show that if the gradients of the functions are bounded and lipschitz continuous over a certain level set then every cluster point of the iterates generated by the method is a stationary point in addition if the gradient of the functions have a certain growth property then the method is either linearly convergent in some sense or the stepsizes are bounded away from zero the new stepsize rule is much in the spirit of heuristic learning rules used in practice for training neural networks via backpropagation as such the new stepsize rule may suggest improvements on existing learning rules finally extension of the method and the convergence results to constrained minimization is discussed as are some implementation issues and numerical experience

control of a class of nonlinear discrete time systems using multilayer neural networks a multilayer neural network nn controller is designed to deliver a desired tracking performance for the control of a class of unknown nonlinear systems in discrete time where the system nonlinearities do not satisfy a matching condition using the lyapunov approach the uniform ultimate boundedness of the tracking error and the nn weight estimates are shown by using a novel weight updates further a rigorous procedure is provided from this analysis to select the nn controller parameters the resulting structure consists of several nn function approximation inner loops and an outer proportional derivative tracking loop simulation results are then carried out to justify the theoretical conclusions the net result is the design and development of an nn controller for strict feedback class of nonlinear discrete time systems

on a variational norm tailored to variable basis approximation schemes a variational norm associated with sets of computational units and used in function approximation learning from data and infinite dimensional optimization is investigated for sets gk obtained by varying a vector y of parameters in a fixed structure computational unit k y e g the set of gaussians with free centers and widths upper and lower bounds on the gk variation norms of functions having certain integral representations are given in terms of the 1 norms of the weighting functions in such representations families of functions for which the two norms are equal are described

determining nodes for partly dissipative reaction diffusion systems abstract in this paper we study the long time dynamical properties of the solutions of partly dissipative reaction diffusion systems of the form u t d u f x u g x v 0 x r n t 0 v t v h x u 0 the main result is that under appropriate assumptions the large time behavior of a solution is completely determined in certain sense if the large time behavior of the solution is known on a densely chosen yet finite set of points which are called determining nodes a concept initially introduced by foias and temam for the navier stokes equations math comput 43 1984 117

approximate minimization of the regularized expected error over kernel models learning from data under constraints on model complexity is studied in terms of rates of approximate minimization of the regularized expected error functional for kernel models with an increasing number n of kernel functions upper bounds on such rates are derived the bounds are of the form a n b n where a and b depend on the regularization parameter and on properties of the kernel and of the probability measure defining the expected error as a special case estimates of rates of approximate minimization of the regularized empirical error are derived

a neural state estimator with bounded errors for nonlinear systems a neural state estimator is described acting on discrete time nonlinear systems with noisy measurement channels a sliding window quadratic estimation cost function is considered and the measurement noise is assumed to be additive no probabilistic assumptions are made on the measurement noise nor on the initial state novel theoretical convergence results are developed for the error bounds of both the optimal and the neural approximate estimators to ensure the convergence properties of the neural estimator a minimax tuning technique is used the approximate estimator can be designed offline in such a way as to enable it to process on line any possible measure pattern almost instantly

neural networks for feedback feedforward nonlinear control systems this paper deals with the problem of designing feedback feedforward control strategies to drive the state of a dynamic system in general nonlinear so as to track any desired trajectory joining the points of given compact sets while minimizing a certain cost function in general nonquadratic due to the generality of the problem conventional methods are difficult to apply thus an approximate solution is sought by constraining control strategies to take on the structure of multilayer feedforward neural networks after discussing the approximation properties of neural control strategies a particular neural architecture is presented which is based on what has been called the linear structure preserving principle the original functional problem is then reduced to a nonlinear programming one and backpropagation is applied to derive the optimal values of the synaptic weights recursive equations to compute the gradient components are presented which generalize the classical adjoint system equations of n stage optimal control theory simulation results related to nonlinear nonquadratic problems show the effectiveness of the proposed method

dependence of computational models on input dimension tractability of approximation and optimization tasks the role of input dimension d is studied in approximating in various norms target sets of d variable functions using linear combinations of adjustable computational units results from the literature which emphasize the number n of terms in the linear combination are reformulated and in some cases improved with particular attention to dependence on d for worst case error upper bounds are given in the factorized form d n where is nonincreasing typically n n 1 2 target sets of functions are described for which the function is a polynomial some important cases are highlighted where decreases to zero as d for target functions extent e g the size of domains in rd where they are defined scale e g maximum norms of target functions and smoothness e g the order of square integrable partial derivatives may depend on d and the influence of such dimension dependent parameters on model complexity is considered results are applied to approximation and solution of optimization problems by neural networks with perceptron and gaussian radial computational units

existence of inertial manifolds for partly dissipative reaction diffusion systems in higher space dimensions abstract in this paper we prove the existence of inertial manifolds for a partly dissipative reaction diffusion system of the form formula where is a rectangular domain in r 2or a cubic domain in r 3 the proof is based on an abstract invariant manifold theorem for semiflows in a hilbert space which is proved by using the graph transform method the principle of spatial averaging also plays an important role in the proof

geometric upper bounds on rates of variable basis approximation in this paper approximation by linear combinations of an increasing number n of computational units with adjustable parameters such as perceptrons and radial basis functions is investigated geometric upper bounds on rates of convergence of approximation errors are derived the bounds depend on certain parameters specific for each function to be approximated the results are illustrated by examples of values of such parameters in the case of approximation by linear combinations of orthonormal functions

estimating the number of asymptotic degrees of freedom for nonlinear dissipative systems we show that the long time behavior of the projection of the exact solutions to the navier stokes equations and other dissipative evolution equations on the finite dimensional space of interpolant polynomials determines the long time behavior of the solution itself provided that the spatial mesh is fine enough we also provide an explicit estimate on the size of the mesh moreover we show that if the evolution equation has an inertial manifold then the dynamics of the evolution equation is equivalent to the dynamics of the projection of the solutions on the finite dimensional space spanned by the approximating polynomials our results suggest that certain numerical schemes may capture the essential dynamics of the underlying evolution equation

nonlinear black box modeling in system identification a unified overview a nonlinear black box structure for a dynamical system is a model structure that is prepared to describe virtually any nonlinear dynamics there has been considerable recent interest in this area with structures based on neural networks radial basis networks wavelet networks and hinging hyperplanes as well as wavelet transform based methods and models based on fuzzy sets and fuzzy rules this paper describes all these approaches in a common framework from a user s perspective it focuses on what are the common features in the different approaches the choices that have to be made and what considerations are relevant for a successful system identification application of these techniques it is pointed out that the nonlinear structures can be seen as a concatenation of a mapping form observed data to a regression vector and a nonlinear mapping from the regressor space to the output space these mappings are discussed separately the latter mapping is usually formed as a basis function expansion the basis functions are typically formed from one simple scalar function which is modified in terms of scale and location the expansion from the scalar argument to the regressor space is achieved by a radial or a ridge type approach basic techniques for estimating the parameters in the structures are criterion minimization as well as two step procedures where first the relevant basis functions are determined using data and then a linear least squares step to determine the coordinates of the function approximation a particular problem is to deal with the large number of potentially necessary parameters this is handled by making the number of used parameters considerably less than the number of offered parameters by regularization shrinking pruning or regressor selection

determining nodes for semilinear parabolic equations we are concerned with the determination of the asymptotic behavior of strong solutions to the initial boundary value problem for general semilinear parabolic equations by the asymptotic behavior of these strong solutions on a finite set more precisely if the asymptotic behavior of the strong solution is known on a suitable finite set which is called determining nodes then the asymptotic behavior of the strong solution itself is entirely determined we prove the above property by the energy method