subgradient based neural networks for nonsmooth nonconvex optimization problems this paper presents a subgradient based neural network to solve a nonsmooth nonconvex optimization problem with a nonsmooth nonconvex objective function a class of affine equality constraints and a class of nonsmooth convex inequality constraints the proposed neural network is modeled with a differential inclusion under a suitable assumption on the constraint set and a proper assumption on the objective function it is proved that for a sufficiently large penalty parameter there exists a unique global solution to the neural network and the trajectory of the network can reach the feasible region in finite time and stay there thereafter it is proved that the trajectory of the neural network converges to the set which consists of the equilibrium points of the neural network and coincides with the set which consists of the critical points of the objective function in the feasible region a condition is given to ensure the convergence to the equilibrium point set in finite time moreover under suitable assumptions the coincidence between the solution to the differential inclusion and the ldquoslow solutionrdquo of it is also proved furthermore three typical examples are given to present the effectiveness of the theoretic results obtained in this paper and the good performance of the proposed neural network

robust pole assignment in singular control systems abstract necessary conditions are given for the problem of pole assignment by state feedback in singular linear systems descriptor systems to have a solution which is regular and nondefective for a robust solution such that the assigned closed loop poles are insensitive to perturbations in the system data the same conditions must hold it can be shown that these conditions are also sufficient for the existence of a feedback which assigns the maximum possible number of finite poles with regularity these results provide the basis of a procedure for constructing closed loop semistate systems with given poles guaranteed regularity and maximum robustness

delay dependent robust stability criteria for two classes of uncertain singular time delay systems in this note the delay dependent robust stability criteria for two classes of singular time delay systems with norm bounded uncertainties are investigated first without using model transformation and bounding technique for cross terms an improved delay dependent stability criterion for the nominal singular time delay system is established in terms of strict linear matrix inequalities lmis then based on this criterion the delay dependent robust stability criteria for two classes of uncertain singular time delay systems are proposed which ensure that the systems are regular impulse free and asymptotically stable for all admissible uncertainties numerical examples are proposed to illustrate the less conservatism of the obtained results

convergence analyses on on line weight noise injection based training algorithms for mlps injecting weight noise during training is a simple technique that has been proposed for almost two decades however little is known about its convergence behavior this paper studies the convergence of two weight noise injection based training algorithms multiplicative weight noise injection with weight decay and additive weight noise injection with weight decay we consider that they are applied to multilayer perceptrons either with linear or sigmoid output nodes let w t be the weight vector let v w be the corresponding objective function of the training algorithm let 0 be the weight decay constant and let t be the step size we show that if t 0 then with probability one e w t 22 is bound and limt w t 2 exists based on these two properties we show that if t 0 t t and t t 2 then with probability one these algorithms converge moreover w t converges with probability one to a point where wv w 0

neural network for nonsmooth nonconvex constrained minimization via smooth approximation a neural network based on smoothing approximation is presented for a class of nonsmooth nonconvex constrained optimization problems where the objective function is nonsmooth and nonconvex the equality constraint functions are linear and the inequality constraint functions are nonsmooth convex this approach can find a clarke stationary point of the optimization problem by following a continuous path defined by a solution of an ordinary differential equation the global convergence is guaranteed if either the feasible set is bounded or the objective function is level bounded specially the proposed network does not require 1 the initial point to be feasible 2 a prior penalty parameter to be chosen exactly 3 a differential inclusion to be solved numerical experiments and comparisons with some existing algorithms are presented to illustrate the theoretical results and show the efficiency of the proposed network

two examples of ill posedness with respect to small time delays in stabilized elastic systems two examples of elastic control systems that are uniformly stabilized by feedback controls on their boundaries for which divergence rates can be obtained when small time delays are introduced into their feedbacks are presented

analysis of boundedness and convergence of online gradient method for two layer feedforward neural networks this paper presents a theoretical boundedness and convergence analysis of online gradient method for the training of two layer feedforward neural networks the well known linear difference equation is extended to apply to the general case of linear or nonlinear activation functions based on this extended difference equation we investigate the boundedness and convergence of the parameter sequence of concern which is trained by finite training samples with a constant learning rate we show that the uniform upper bound of the parameter sequence which is very important in the training procedure is the solution of an inequality regarding the bound it is further verified that for the case of linear activation function a solution always exists and moreover the parameter sequence can be uniformly upper bounded while for the case of nonlinear activation function some simple adjustment methods on the training set or the activation function can be derived to improve the boundedness property then for the convergence analysis it is shown that the parameter sequence can converge into a zone around an optimal solution at which the error function attains its global minimum where the size of the zone is associated with the learning rate particularly for the case of perfect modeling a strong global convergence result where the parameter sequence can always converge to an optimal solution is proved

conditions for robustness and nonrobustness of thestability of feedback systems with respect to small delays inthe feedback loop it has been observed that for many stable feedback control systems the introduction of arbitrarily small time delays into the loop causes instability in this paper we present a systematic frequency domain treatment of this phenomenon for distributed parameter systems we consider the class of all matrix valued transfer functions which are bounded on some right half plane and which have a limit at spinfty along the real axis such transfer functions are called regular under the assumption that a regular transfer function is stabilized by unity output feedback we give sufficient conditions for the robustness and for the nonrobustness of the stability with respect to small time delays in the loop these conditions are given in terms of the high frequency behavior of the open loop system moreover we discuss robustness of stability with respect to small delays for feedback systems with dynamic compensators in particular we show that if a plant with infinitely many poles in the closed right half plane is stabilized by a controller then the stability is not robust with respect to delays we show that the instability created by small delays is itself robust to small delays three examples are given to illustrate these results

admissibility for nonuniform exponential contractions abstract we study the relation between the notions of nonuniform exponential stability and admissibility in particular using appropriate adapted norms which can be seen as lyapunov norms we show that if any of their associated l p spaces with p 1 is admissible for a given evolution process then this process is a nonuniform exponential contraction we also provide a collection of admissible banach spaces for any given nonuniform exponential contraction

not all feedback stabilized hyperbolic systems are robust with respect to small time delays in their feedbacks we present two examples of hyperbolic partial differential equations which are stabilized by boundary feedback controls and then destabilized by small delays in these controls we show that in a general case when the controls are distributed stabilized hyperbolic systems possess nontrivial periodic solutions if small time delays are introduced into their feedbacks we also indicate by means of an example that the general case of this phenomenon is harder to demonstrate for boundary control problems

modeling a stochastic age structured capital system with poisson jumps using neural networks abstract this paper presents an approach for modeling a stochastic age structured capital system sascs with poisson jumps in which the transfer rate of capital noise intensity and jump intensity are approximated by a type of feedforward neural networks fnns the approximate stock of capital for the system is calculated using euler s scheme in time discretization based on barkholder davis gundys inequality and gronwalls lemma the error between the approximate stock of capital and the actual stock of capital is estimated by using mean squares theoretical analysis of the convergence of the approximate stock of capital is conducted and some numerical results are presented to justify our findings

transfer functions of regular linear systems part i characterizations of regularity we recall the main facts about the representation of regular linear systems essentially that they can be described by equations of the form x t ax t bu t y t cx t du t like finite dimensional systems but now a b and c are in general unbounded operators regular linear systems are a subclass of abstract linear systems we define transfer functions of abstract linear systems via a generalization of a theorem of foures and segal we prove a formula for the transfer function of a regular linear system which is similar to the formula in finite dimensions the main result is a simple to state but hard to prove necessary and sufficient condition for an abstract linear system to be regular in terms of its transfer function other conditions equivalent to regularity are also obtained the main result is a consequence of a new tauberian theorem which is of independent interest

global stabilizability of uncertain systems with time varying delays via dynamic observer based output feedback in this paper the dynamic observer based output feedback control for uncertain time delay systems with time varying delays is investigated a delay independent dynamic observer based output feedback control is proposed such that the feedback controlled system with time varying delays is globally asymptotically stable if some mild conditions are met an upper bound of arbitrary time varying delays without destroying stability is also given such that the asymptotic stability is preserved a numerical example is given to illustrate the use of our main results

on different facets of regularization theory this review provides a comprehensive understanding of regularization theory from different perspectives emphasizing smoothness and simplicity principles using the tools of operator theory and fourier analysis it is shown that the solution of the classical tikhonov regularization problem can be derived from the regularized functional defined by a linear differential integral operator in the spatial fourier domain state of the art research relevant to the regularization theory is reviewed covering occam s razor minimum length description bayesian theory pruning algorithms informational entropy theory statistical learning theory and equivalent regularization the universal principle of regularization in terms of kolmogorov complexity is discussed finally some prospective studies on regularization theory and beyond are suggested

convergence analyses on sparse feedforward neural networks via group lasso regularization in this paper a new variant of feedforward neural networks has been proposed for a class of nonsmooth optimization problems the penalty term of the presented neural networks stems from the group lasso method which selects hidden variables in a grouped manner to deal with the non differentiability of the original penalty term l 1 l 2 norm and avoid oscillations smoothing techniques have been used to approximate the objective function it is assumed that the training samples are supplied to the networks in a specific incremental way during training that is in each cycle samples are supplied in a fixed order then under suitable assumptions on learning rate penalization coefficients and smoothing parameters the weak and strong convergence of the training process for the smoothing neural networks have been proved the convergence analysis shows that the gradient of the smoothing error function approaches zero and the weight sequence converges to a fixed point respectively we demonstrate how the smoothing approximation parameter can be updated in the training procedure so as to guarantee the convergence of the procedure to a clarke stationary point of the original optimization problem in addition we have proved that the original nonsmoothing algorithm with l 1 l 2 norm penalty converges consistently to the same optimum solution with the corresponding smoothed algorithm numerical simulations demonstrate the convergence and effectiveness of the proposed training algorithm

when does online bp training converge the backpropogation bp neural networks have been widely applied in scientific research and engineering the success of the application however relies upon the convergence of the training procedure involved in the neural network learning we settle down the convergence analysis issue through proving two fundamental theorems on the convergence of the online bp training procedure one theorem claims that under mild conditions the gradient sequence of the error function will converge to zero the weak convergence and another theorem concludes the convergence of the weight sequence defined by the procedure to a fixed value at which the error function attains its minimum the strong convergence the weak convergence theorem sharpens and generalizes the existing convergence analysis conducted before while the strong convergence theorem provides new analysis results on convergence of the online bp training procedure the results obtained reveal that with any analytic sigmoid activation function the online bp training procedure is always convergent which then underlies successful application of the bp neural networks

worst case complexity of smoothing quadratic regularization methods for non lipschitzian optimization in this paper we propose a smoothing quadratic regularization sqr algorithm for solving a class of nonsmooth nonconvex perhaps even non lipschitzian minimization problems which has wide applications in statistics and sparse reconstruction the proposed sqr algorithm is a first order method at each iteration the sqr algorithm solves a strongly convex quadratic minimization problem with a diagonal hessian matrix which has a simple closed form solution that is inexpensive to calculate we show that the worst case complexity of reaching an epsilon scaled stationary point is o epsilon 2 moreover if the objective function is locally lipschitz continuous the sqr algorithm with a slightly modified updating scheme for the smoothing parameter and iterate can obtain an epsilon clarke stationary point in at most o epsilon 3 iterations

destabilizing effects of small time delays on feedback controlled descriptor systems abstract in the last 15 years the problem of stabilizability and stabilization of descriptor systems have received considerable attention in this paper it is shown that if a descriptor system e x ax bu exhibits impulsive behavior then the stability of the closed loop system is extremely sensitive to small delays more precisely if f is the feedback which leads to a stable and impulsive free closed loop system then there exist numbers ej 0 and sj c with limj ej 0 and limj re sj and such that the delayed closed loop system obtained by applying the feedback u t fx t ej has a pole at sj moreover if the open loop system does not have impulsive behavior the same phenomenon occurs provided that the spectral radius of the matrix lim s f se a 1b is greater than 1 if this spectral radius is smaller than 1 it is shown that the closed loop stability is robust with respect to small delays

sensitivity to infinitesimal delays in neutral equations the stability of a steady state solution of a neutral functional differential equation can be sensitive to infinitesimal changes in the delays this phenomenon is caused by the behavior of the essential spectrum and is determined by the roots of an exponential polynomial avellar and hale j math anal appl 73 1980 pp 434 452 have considered the case of multiple fixed and nonzero delays in the first part of this paper their results are illustrated by means of spectral plots in the second part we extend the theory of avellar and hale to the limit case whereby some of the delays are brought to zero which may lead to characteristic roots with arbitrarily large real part necessary and sufficient conditions are provided using these results we show that the ratio of the delays plays a crucial role when several delays tend to zero simultaneously as an illustration of the theory we analyze the robustness of a boundary controlled pde in the presence of a small feedback delay

stabilizability and stability robustness of state derivative feedback controllers we study the stabilizability of a linear controllable system using state derivative feedback control as a special feature the stabilized system may be fragile in the sense that arbitrarily small modeling and implementation errors may destroy the asymptotic stability first we discuss the pole placement problem and illustrate the fragility of stability with examples of a different nature we also define a notion of stability called p stability which explicitly takes into account the effect of small modeling and implementation errors next we investigate the effect on the fragility of including a low pass filter in the control loop finally we completely characterize the stabilizability and p stabilizability of linear controllable systems using state derivative feedback in the stabilizability characterization the odd number limitation well known in the context of the stabilization of unstable periodic orbits using pyragas type time delayed feedback plays a crucial role

smoothing neural network for constrained non lipschitz optimization with applications in this paper a smoothing neural network snn is proposed for a class of constrained non lipschitz optimization problems where the objective function is the sum of a nonsmooth nonconvex function and a non lipschitz function and the feasible set is a closed convex subset of using the smoothing approximate techniques the proposed neural network is modeled by a differential equation which can be implemented easily under the level bounded condition on the objective function in the feasible set we prove the global existence and uniform boundedness of the solutions of the snn with any initial point in the feasible set the uniqueness of the solution of the snn is provided under the lipschitz property of smoothing functions we show that any accumulation point of the solutions of the snn is a stationary point of the optimization problem numerical results including image restoration blind source separation variable selection and minimizing condition number are presented to illustrate the theoretical results and show the efficiency of the snn comparisons with some existing algorithms show the advantages of the snn

deterministic convergence of an online gradient method for neural networks the online gradient method has been widely used as a learning algorithm for neural networks we establish a deterministic convergence of online gradient methods for the training of a class of nonlinear feedforward neural networks when the training examples are linearly independent we choose the learning rate to be a constant during the training procedure the monotonicity of the error function in the iteration is proved a criterion for choosing the learning rate is also provided to guarantee the convergence under certain conditions similar to those for the classical gradient methods an optimal convergence rate for our online gradient methods is proved

convergence of an online gradient method for feedforward neural networks with stochastic inputs in this paper we study the convergence of an online gradient method for feed forward neural networks the input training examples are permuted stochastically in each cycle of iteration a monotonicity and a weak convergence of deterministic nature are proved

a singularity induced bifurcation theorem for infinite dimensional implicit dynamical systems in this note we study a class of partial differential equations whose formal structure resembles the semi explicit index 1 form from the theory of differential algebraic equations it is known from the theory of such problems that one can encounter simple poles in the eigenvalue loci of parameter dependent linearizations leading to the loss of linear stability without the presence of steady state bifurcations we demonstrate how this carries over to implicitly defined problems on a hilbert space

an example on the effect of time delays in boundary feedback stabilization of wave equations this note is concerned with the erect of time delays in boundary feedback stabilization schemes for wave equations the question to be addressed is whether such delays can destabilize a system which is uniformly asymptotically stable in the absence of delays

smoothing methods for nonsmooth nonconvex minimization we consider a class of smoothing methods for minimization problems where the feasible set is convex but the objective function is not convex not differentiable and perhaps not even locally lipschitz at the solutions such optimization problems arise from wide applications including image restoration signal reconstruction variable selection optimal control stochastic equilibrium and spherical approximations in this paper we focus on smoothing methods for solving such optimization problems which use the structure of the minimization problems and composition of smoothing functions for the plus function x many existing optimization algorithms and codes can be used in the inner iteration of the smoothing methods we present properties of the smoothing functions and the gradient consistency of subdifferential associated with a smoothing function moreover we describe how to update the smoothing parameter in the outer iteration of the smoothing methods to guarantee convergence of the smoothing methods to a stationary point of the original minimization problem

a theoretical comparison of batch mode on line cyclic and almost cyclic learning we study and compare different neural network learning strategies batch mode learning online learning cyclic learning and almost cyclic learning incremental learning strategies require less storage capacity than batch mode learning however due to the arbitrariness in the presentation order of the training patterns incremental learning is a stochastic process whereas batch mode learning is deterministic in zeroth order i e as the learning parameter spl eta tends to zero all learning strategies approximate the same ordinary differential equation for convenience referred to as the ideal behavior using stochastic methods valid for small learning parameters spl eta we derive differential equations describing the evolution of the lowest order deviations from this ideal behavior we compute how the asymptotic misadjustment measuring the average asymptotic distance from a stable fixed point of the ideal behavior scales as a function of the learning parameter and the number of training patterns knowing the asymptotic misadjustment we calculate the typical number of learning steps necessary to generate a weight within order spl epsiv of this fixed point both with fixed and time dependent learning parameters we conclude that almost cyclic learning learning with random cycles is a better alternative for batch mode learning than cyclic learning learning with a fixed cycle