convex proximal bundle methods in depth a unified analysis for inexact oracles the last few years have seen the advent of a new generation of bundle methods capable to handle inexact oracles polluted by noise proving convergence of a bundle method is never simple and coping with inexact oracles substantially increases the technicalities besides several variants exist to deal with noise each one needing an ad hoc proof to show convergence we state a synthetic convergence theory in which we highlight the main arguments and specify which assumption is used to establish each intermediate result the framework is comprehensive and generalizes in various ways a number of algorithms proposed in the literature based on the ingredients of our synthetic theory we consider various bundle methods adapted to oracles for which high accuracy is possible yet it is preferable not to make exact calculations often because they are too time consuming

a new continuous time equality constrained optimization to avoid singularity in equality constrained optimization a standard regularity assumption is often associated with feasible point methods namely that the gradients of constraints are linearly independent in practice the regularity assumption may be violated in order to avoid such a singularity a new projection matrix is proposed based on which a feasible point method to continuous time equality constrained optimization is developed first the equality constraint is transformed into a continuous time dynamical system with solutions that always satisfy the equality constraint second a new projection matrix without singularity is proposed to realize the transformation an update or say a controller is subsequently designed to decrease the objective function along the solutions of the transformed continuous time dynamical system the invariance principle is then applied to analyze the behavior of the solution furthermore the proposed method is modified to address cases in which solutions do not satisfy the equality constraint finally the proposed optimization approach is applied to three examples to demonstrate its effectiveness

minimizing piecewise concave functions over polyhedra we introduce an iterative method for solving linearly constrained optimization problems whose nonsmooth nonconvex objective function is defined as the pointwise maximum of finitely many concave functions such problems often arise from reformulations of certain constraint structures e g binary constraints finite max min constraints in diverse areas of optimization we state a local optimization strategy which exploits piecewise concavity of the objective function giving rise to a linearized model corrected by a proximity term in addition we introduce an approximate line search strategy based on a curvilinear model which similarly to bundle methods can return either a satisfactory descent or a null step declaration termination at a point satisfying an approximate stationarity condition is proved we embed the local minimization algorithm into a variable neighborhood search scheme and a coordinate direction search heuristic whose aim is to improve the current estimate of the global minimizer

dynamical systems and sequence transformations this paper discusses the connections between numerical methods for ordinary differential equations fixed point iterations and sequence transformations books on chaos and fractals are concerned on the one hand with continuous dynamical systems and on the other hand with the dynamics of iterations in discrete dynamical systems there exists a strong connection between numerical methods for the integration of ordinary differential equations odes and fixed point iterations fpis this connection has already been studied see 14 chapter 5 pp 197 343 or 25 chapter 6 pp 165 201 for an introduction and 30 for an extended review but the dynamics of the iterations generated by a numerical method for odes was mainly used for understanding the behaviour of the solution of the differential equation in this paper we are mostly interested in fpis we will see that fpis can be considered as coming from methods for the numerical integration of odes reciprocally numerical methods for odes can be used in the solution of fixed point problems so fpis can lead to new methods for the numerical integration of odes and conversely the link with sequence transformations used in numerical analysis to accelerate the convergence will also be discussed

a lyapunov like characterization of asymptotic controllability it is shown that a control system in bf r n is asymptotically controllable to the origin if and only if there exists a positive definite continuous functional of the states whose derivative can be made negative by appropriate choices of controls

iterative solution of linear systems in the 20th century this paper sketches the main research developments in the area of iterative methods for solving linear systems during the 20th century although iterative methods for solving linear systems nd their origin in the early 19th century work by gauss the eld has seen an explosion of activity spurred by demand due to extraordinary technological advances in engineering and sciences the past ve decades have been particularly rich in new developments ending with the availability of large toolbox of specialized algorithms for solving the very large problems which arise in scientic and industrial computational models as in any other scientic area research in iterative methods has been a journey characterized by a chain of contributions building on each other it is the aim of this paper not only to sketch the most signicant of these contributions during the past century but also to relate them to one another c 2000 elsevier science b v all rights reserved

stabilization with relaxed controls cannot in general be stabilized using a continuous closed loop control u x even if each state separately can be driven asymptotically to the origin an example is analyzed in section 2 in this paper we examine the possibility of stabilizing such systems with a continuous closed loop relaxed control we find indeed that the family of systems stabilizable with relaxed controls is larger than the family of those stabilizable with ordinary controls an even larger class is obtained if the continuity of the closed loop at the origin is not required the latter class includes all one dimensional systems for which states can be driven asymptotically to the origin this result does not hold in two dimensional systems and we provide a counter example it should be pointed that relaxed control type stabilization is used both in theory and in practice the method is known as dither we shall comment on the similarities lyapunov functions for the system 1 help us in the construction of the continuous closed loop stabilizers in fact we find that the existence of a smooth lyapunov function is equivalent to the existence of a stabilizing closed loop which is continuous except possibly at the origin an additional condition on the lyapunov function implies the continuity at the origin as well we present these results in section 4 after a brief introduction of closed loop relaxed controls notations and terminology in section 3 prior to that in section 2 we discuss an example illustrating the power of relaxed controls in the particular case of systems linear in the controls relaxed controls can be replaced by ordinary controls this is discussed in section 5 the role of lyapunov functions in the stability and stabilization theories is of course well known examples of systems with lyapunov functions are available in the literature we display some in section 6 along with general comments on the construction applications and counterexamples including one which cannot be continuously stabilized yet possesses a nonsmooth lyapunov function closed loop stabilization with ordinary controls is analyzed extensively in the literature see sontag 8 sussmann ll and references therein lyapunov functions techniques in stabil

a smooth method for the finite minimax problem we consider unconstrained minimax problems where the objective function is the maximum of a finite number of smooth functions we prove that under usual assumptions it is possible to construct a continuously differentiable function whose minimizers yield the minimizers of the max function and the corresponding minimum values on this basis we can define implementable algorithms for the solution of the minimax problem which are globally convergent at a superlinear convergence rate preliminary numerical results are reported

combined lp and quasi newton methods for minimax optimization we present an algorithm for minimax optimization that combines lp methods and quasi newton methods the quasi newton algorithm is used only if an irregular solution is detected in which case second order derivative information is needed in order to obtain a fast final rate of convergence we prove that the algorithm can converge only to a stationary point and that normally the final rate of convergence will be either quadratic or superlinear the performance is illustrated through some numerical examples

incremental bundle methods using upper models we propose a family of proximal bundle methods for minimizing sum structured convex nondifferentiable functions which require two slightly uncommon assumptions that are satisfied in many relevant applications lipschitz continuity of the functions and oracles which also produce upper estimates on the function values in exchange the methods i use upper models of the functions that allow one to estimate function values at points where the oracle has not been called ii provide the oracles with more information about when the function computation can be interrupted possibly diminishing their cost iii allow one to skip oracle calls entirely for some of the component functions not only at null steps but also at serious steps iv provide explicit and reliable a posteriori estimates of the quality of the obtained solutions v work with all possible combinations of different assumptions on how the oracles deal with not being able to compute the function with arbitrary accuracy we also discus

distributed asynchronous computation of fixed points we present an algorithmic model for distributed computation of fixed points whereby several processors participate simultaneously in the calculations while exchanging information via communication links we place essentially no assumptions on the ordering of computation and communication between processors thereby allowing for completely uncoordinated execution we provide a general convergence theorem for algorithms of this type and demonstrate its applicability to several classes of problems including the calculation of fixed points of contraction and monotone mappings arising in linear and nonlinear systems of equations optimization problems shortest path problems and dynamic programming

incremental subgradient methods for nondifferentiable optimization we consider a class of subgradient methods for minimizing a convex function that consists of the sum of a large number of component functions this type of minimization arises in a dual context from lagrangian relaxation of the coupling constraints of large scale separable problems the idea is to perform the subgradient iteration incrementally by sequentially taking steps along the subgradients of the component functions with intermediate adjustment of the variables after processing each component function this incremental approach has been very successful in solving large differentiable least squares problems such as those arising in the training of neural networks and it has resulted in a much better practical rate of convergence than the steepest descent method in this paper we establish the convergence properties of a number of variants of incremental subgradient methods including some that are stochastic based on the analysis and computational experiments the methods appear very promising and effective for important classes of large problems a particularly interesting discovery is that by randomizing the order of selection of component functions for iteration the convergence rate is substantially improved

system theory for numerical analysis many numerical schemes can be suitably studied from a system theoretic point of view this paper studies the relationship between the two disciplines that is numerical analysis and system theory we first see that various iterative solution schemes for linear and nonlinear equations can be suitably transformed into the form of a closed loop feedback system and show the crucial role of the internal model principle in such a context this leads to new stability criteria for newton s method we then study runge kutta type methods for solving differential equations and also derive new stability criteria based on recent results on lmi a numerical example is given to illustrate the advantage of the present theory

on the continuous realization of iterative processes many important mathematical problems are solved by iterative methods many of these iterative schemes may be regarded as the discrete realization of certain continuous dynamical systems this paper summarizes some of the recent developments in the continuous realization of several popular basic iterative methods

dynamical systems that sort lists diagonalize matrices and solve linear programming problems abstract we establish a number of properties associated with the dynamical system h h h n where h and n are symmetric n by n matrices and a b ab ba the most important of these come from the fact that this equation is equivalent to a certain gradient flow on the space of orthogonal matrices we are especially interested in the role of this equation as an analog computer for example we show how to map the data associated with a linear programming problem into h 0 and n in such a way as to have h h h n evolve to a solution of the linear programming problem this result can be applied to find systems which solve a variety of genetic combinatorial optimization problems and it even provides an algorithm for diagonalizing symmetric matrices

a universal construction of artstein s theorem on nonlinear stabilization abstract this note presents an explicit proof of the theorem due to artstein which states that the existence of a smooth control lyapunov function implies smooth stabilizability moreover the result is extended to the real analytic and rational cases as well the proof uses a universal formula given by an algebraic function of lie derivatives this formula originates in the solution of a simple riccati equation

constrained bundle methods for upper inexact oracles with application to joint chance constrained energy problems joint chance constrained problems give rise to many algorithmic challenges even in the convex case i e when an appropriate transformation of the probabilistic constraint is a convex function its cutting plane linearization is just an approximation produced by an oracle providing subgradient and function values that can only be evaluated inexactly as a result the cutting plane model may lie above the true constraint for dealing with such upper inexact oracles and still solving the problem up to certain precision a special numerical algorithm must be put in place we introduce a family of constrained bundle methods based on the so called improvement functions that is shown to be convergent and encompasses many previous approaches as well as new algorithms depending on the oracle accuracy we analyze to what extent the considered methods solve the joint chance constrained program the approach is assessed on real life energy problems arising when dealing with stochastic hydroreservoir management

on the role of singularities in branin s method from dynamic and continuation perspectives a global analysis of branin s method originally due to davidenko for finding all the real zeros of a vector function is carried out the analysis is based on a global study of this method perceived of as a dynamical system since branin s algorithm is closely related to homotopy methods this paper sheds some light on the global performance of these methods when employed for locating all the zeros of a vector function following the dynamical system approach the performance of branin s algorithm is related to the existence of extraneous singularities as well as to the relative spatial distribution of the zeros of the vector function and singular manifolds branin s conjectures regarding the types and the role of extraneous singularities are examined and counterexamples are provided to disprove them we conclude that the performance of branin s method for locating all the zeros of a vector function is questionable even in the absence of extraneous singularities

consensus of discrete time second order multiagent systems based on infinite products of general stochastic matrices traditionally the consensus of a discrete time multiagent system mas with a switching topology is transformed into the convergence problem of the infinite products of stochastic matrices which can be resolved by using the wolfowitz theorem however such a transformation is very difficult or even impossible for certain mass such as discrete time second order mass dtso mass whose consensus can only be transformed into the convergence problem of the infinite products of general stochastic matrices ipgsm these general stochastic matrices are matrices with row sum 1 but their elements are not necessarily nonnegative since there does not exist a general theory or an effective technique for dealing with the convergence of ipgsm establishing the consensus criteria for a dtso mas with a switching topology is rather difficult this paper concentrates on the consensus problem of a class of dtso mass and develops a method to cope with the corresponding ipgsm moreover it is pointed out that the method

weak singularities and the continuous newton method abstract some issues related to the determination of the singular roots of a nonlinear vector function f r n r n are addressed in this paper it is usually assumed that newton like fields are not defined at singularities thus a particular treatment for these points is necessary nevertheless in dimension 1 and in several higher dimensional instances it is possible to make a smooth extension of the field to singular points when this is the case for a singular root it can be treated in a way similar to that of regular ones necessary and sufficient conditions for this extension to be possible are given under some structural assumptions through the concept of weak singularity the actual setting for this result is a general class of quotient functions which includes in particular the newton field for the specific case of the continuous time newton method we enlarge some previous results concerning the relation between singular roots and equilibrium points of the extended field as well as their asymptotic stability finally a computational tool obtained from the extended continuous newton method by means of the cell mapping technique is shown to be well behaved for the location of these singular roots

controllability of matrix eigenvalue algorithms the inverse power method abstract in this paper we initiate a program to study the controllability properties of matrix eigenvalue algorithms arising in numerical linear algebra our focus is on a well known eigenvalue method the inverse power iteration defined on projective space a complete characterization of the reachable sets and their closures is given via cyclic invariant subspaces moreover a necessary and sufficient condition for almost controllability of the inverse power method is derived

piecewise quadratic approximations in convex numerical optimization we present a bundle method for convex nondifferentiable minimization where the model is a piecewise quadratic convex approximation of the objective function unlike standard bundle approaches the model only needs to support the objective function from below at a properly chosen small subset of points as opposed to everywhere we provide the convergence analysis for the algorithm with a general form of master problem which combines features of trust region stabilization and proximal stabilization taking care of all the important practical aspects such as proper handling of the proximity parameters and the bundle of information numerical results are also reported

on the mathematical foundations of nondifferentiable optimization in engineering design it is shown by example that a large class of engineering design problems can be transcribed into the form of a canonical optimization problem with inequality constraints involving mar functions such problems are commonly referred to as semi infinite optimization problems the bulk of this paper is devoted to the development of a mathematical theory for the construction of first order nondifferentiable optimization algorithms related to phase i phase ii methods of feasible directions which solve these semi infinite optimization problems the applicability of the theory is illustrated with examples that are relevant to engineering design

piecewise linear approximations in nonconvex nonsmooth optimization we present a bundle type method for minimizing nonconvex nondifferentiable functions of several variables the algorithm is based on the construction of both a lower and an upper polyhedral approximation of the objective function in particular at each iteration a search direction is computed by solving a quadratic program aiming at maximizing the difference between the lower and the upper model a proximal approach is used to guarantee convergence to a stationary point under the hypothesis of weak semismoothness

an incremental method for solving convex finite min max problems we introduce a new approach to minimizing a function defined as the pointwise maximum over finitely many convex real functions next referred to as the component functions with the aim of working on the basis of incomplete knowledge of the objective function a descent algorithm is proposed which need not require at the current point the evaluation of the actual value of the objective function namely of all the component functions thus extending to min max problems the philosophy of the incremental approaches widely adopted in the nonlinear least squares literature given the nonsmooth nature of the problem we resort to the well established machinery of bundle methods we provide global convergence analysis of our method and in addition we study a subgradient aggregation scheme aimed at simplifying the problem of finding a tentative step this paper is completed by the numerical results obtained on a set of standard test problems

on controllability of the real shifted inverse power iteration abstract controllability properties of the inverse power method on projective space are investigated for complex eigenvalue shifts a simple characterization of the reachable sets in terms of invariant subspaces can be obtained the real case is more complicated and is investigated in this paper necessary and sufficient conditions for complete controllability are obtained in terms of the solvability of a matrix equation partial results on conditions for the solvability of this matrix equation are given

a numerically stable dual method for solving strictly convex quadratic programs an efficient and numerically stable dual algorithm for positive definite quadratic programming is described which takes advantage of the fact that the unconstrained minimum of the objective function can be used as a starting point its implementation utilizes the cholesky and qr factorizations and procedures for updating them the performance of the dual algorithm is compared against that of primal algorithms when used to solve randomly generated test problems and quadratic programs generated in the course of solving nonlinear programming problems by a successive quadratic programming code the principal motivation for the development of the algorithm these computational results indicate that the dual algorithm is superior to primal algorithms when a primal feasible point is not readily available the algorithm is also compared theoretically to the modified simplex type dual methods of lemke and van de panne and whinston and it is illustrated by a numerical example

least squares matching problems abstract a large class of matching problems can be thought of in the following way one is given a group g which acts on a riemannian manifold and one is given two points x 1 and x 2 in the manifold the problem is to find g g such that d g x 1 x 2 is as small as possible if the group has finite cardinality this is a combinatorial optimization problem whereas if g is a lie group one can use the calculus to obtain necessary conditions our main results establish circumstances under which one can relax the combinatorial problem to an appropriate lie group problem and then solve it by the method of steepest descent an interesting class of applications involves embedding the permutation group on n letters in the orthogonal group o n with the latter acting on r n n 1 2 via a symmetricized tensor product the convergence properties of the descent equation which arises in this way are analyzed in some detail

a control theoretic approach to the design of zero finding numerical methods in this paper it is shown how standard iterative methods for solving linear and nonlinear equations can be designed from the point of view of control appropriate choices of control lyapunov functions clfs lead to both continuous and discrete time versions of the newton raphson and conjugate gradient algorithms as well as new variants

the effect of deterministic noise in subgradient methods in this paper we study the influence of noise on subgradient methods for convex constrained optimization the noise may be due to various sources and is manifested in inexact computation of the subgradients and function values assuming that the noise is deterministic and bounded we discuss the convergence properties for two cases the case where the constraint set is compact and the case where this set need not be compact but the objective function has a sharp set of minima for example the function is polyhedral in both cases using several different stepsize rules we prove convergence to the optimal value within some tolerance that is given explicitly in terms of the errors in the first case the tolerance is nonzero but in the second case the optimal value can be obtained exactly provided the size of the error in the subgradient computation is below some threshold we then extend these results to objective functions that are the sum of a large number of convex functions in which case an incremental subgradient method can be used

applying numerical linear algebra techniques to analyzing algorithms in signal processing without theoretical evidence to support the use of an algorithm one cannot be certain that if it solves one problem successfully it will necessarily solve all problems successfully one can analyze an algorithm from many different settings each having the possibility for different focal points for results and insights in signal processing an analysis into the stability of an algorithm can have as its focal point stability in the sense of lyapunov or of numerical linear algebra backward forward etc unfortunately without a clear vocabulary to apply toward the focal point of interest the interpretation is ambiguous to avoid this problem a clear distinction must be made that delineates each significant contribution to the process of solving the problem from the definition of a problem to the derivation of the algorithm it requires that perturbations assumed at each juncture be described in a clear language that offers no chance for confusion we offer such a terminology and apply it to the fast transversal filter and its existing analyses an algorithm that has long been known to diverge yet has nonetheless received much interest regarding the reasons behind its divergence as well as inquiries into how it might be modified to one that enjoys more dependable and robust behavior