the superlinear convergence of a new quasi newton sqp method for constrained optimization in this paper a new variation of the quasi newton sqp methods for constrained optimization is proposed under some suitable assumptions the superlinear convergence of the new method has been established

nonlinear robust optimization via sequential convex bilevel programming in this paper we present a novel sequential convex bilevel programming algorithm for the numerical solution of structured nonlinear min max problems which arise in the context of semi infinite programming here our main motivation are nonlinear inequality constrained robust optimization problems in the first part of the paper we propose a conservative approximation strategy for such nonlinear and non convex robust optimization problems under the assumption that an upper bound for the curvature of the inequality constraints with respect to the uncertainty is given we show how to formulate a lower level concave min max problem which approximates the robust counterpart in a conservative way this approximation turns out to be exact in some relevant special cases and can be proven to be less conservative than existing approximation techniques that are based on linearization with respect to the uncertainties in the second part of the paper we review existing theory on optimality conditions for nonlinear lower level concave min max problems which arise in the context of semi infinite programming regarding the optimality conditions for the concave lower level maximization problems as a constraint of the upper level minimization problem we end up with a structured mathematical program with complementarity constraints mpcc the special hierarchical structure of this mpcc can be exploited in a novel sequential convex bilevel programming algorithm we discuss the surprisingly strong global and locally quadratic convergence properties of this method which can in this form neither be obtained with existing sqp methods nor with interior point relaxation techniques for general mpccs finally we discuss the application fields and implementation details of the new method and demonstrate the performance with a numerical example

least change quasi newton updates for equality constrained optimization this paper investigates quasi newton updates for equality constrained optimization using a least change argument we derive a class of rank 3 updates to approximations of the one sided projection of the hessian of the lagrangian which keeps the appropriate part symmetric and possibly positive definite by imposing the usual assumptions we are able to prove 1 step superlinear convergence for one of these updates encouraging numerical results and comparisons with other previously analyzed updates are presented

an analysis of reduced hessian methods for constrained optimization we study the convergence properties of reduced hessian successive quadratic programming for equality constrained optimization the method uses a backtracking line search and updates an approximation to the reduced hessian of the lagrangian by means of the bfgs formula two merit functions are considered for the line search thel1 function and the fletcher exact penalty function we give conditions under which local and superlinear convergence is obtained and also prove a global convergence result the analysis allows the initial reduced hessian approximation to be any positive definite matrix and does not assume that the iterates converge or that the matrices are bounded the effects of a second order correction step a watchdog procedure and of the choice of null space basis are considered this work can be seen as an extension to reduced hessian methods of the well known results of powell 1976 for unconstrained optimization

constrained pose refinement of parametric objects pose refinement is an essential task for computer vision systems that require the calibration and verification of model and camera parameters typical domains include the real time tracking of objects and verification in model based recognition systems a technique is presented for recovering model and camera parameters of 3d objects from a single two dimensional image this basic problem is further complicated by the incorporation of simple bounds on the model and camera parameters and linear constraints restricting some subset of object parameters to a specific relationship it is demonstrated in this paper that this constrained pose refinement formulation is no more difficult than the original problem based on numerical analysis techniques including active set methods and lagrange multiplier analysis a number of bounded and linearly constrained parametric models are tested and convergence to proper values occurs from a wide range of initial error utilizing minimal matching information relative to the number of parameters and components the ability to recover model parameters in a constrained search space will thus simplify associated object recognition problems

how bad may learning curves be in this paper we motivate the need for estimating bounds on learning curves of average case learning algorithms when they perform the worst on training samples we then apply the method of reducing learning problems to hypothesis testing ones to investigate the learning curves of a so called ill disposed learning algorithm in terms of a system complexity the boolean interpolation dimension since the ill disposed algorithm behaves worse than ordinal ones and the boolean interpolation dimension is generally bounded by the number of system weights the results can apply to interpreting or to bounding the worst case learning curve in real learning situations this study leads to a new understanding of the worst case generalization in real learning situations which differs significantly from that in the uniform learnable setting via vapnik chervonenkis vc dimension analysis we illustrate the results with some numerical simulations

a projected newton method for minimization problems with nonlinear inequality constraints recently developed projected newton methods for minimization problems in polyhedrons and cartesian products of euclidean balls are extended here to general convex feasible sets defined by finitely many smooth nonlinear inequalities iterate sequences generated by this scheme are shown to be locally superlinearly convergent to nonsingular extremals bar u and more specifically to local minimizers bar u satisfying the standard second order kuhn tucker sufficient conditions moreover all such convergent iterate sequences eventually enter and remain within the smooth manifold defined by the active constraints at bar u implementation issues are considered for large scale specially structured nonlinear programs and in particular for multistage discrete time optimal control problems in the latter case overall per iteration computational costs will typically increase only linearly with the number of stages sample calculations are presented for nonlinear programs in a right circular cylinder in 3

properties of a representation of a basis for the null space given a rectangular matrix a x that depends on the independent variables x many constrained optimization methods involve computations with z x a matrix whose columns form a basis for the null space of a sup t x when a is evaluated at a given point it is well known that a suitable z satisfying a sup t z 0 can be obtained from standard matrix factorizations however coleman and sorensen have recently shown that standard orthogonal factorization methods may produce orthogonal bases that do not vary continuously with x they also suggest several techniques for adapting these schemes so as to ensure continuity of z in the neighborhood of a given point this paper is an extension of an earlier note that defines the procedure for computing z here we first describe how z can be obtained by updating an explicit qr factorization with householder transformations the properties of this representation of z with respect to perturbations in a are discussed including explicit bounds on the change in z we then introduce regularized householder transformations and show that their use implies continuity of the full matrix q the convergence of z and q under appropriate assumptions is then proved finally we indicate why themore chosen form of z is convenient in certain methods for nonlinearly constrained optimization less

a global convergence analysis of an algorithm for large scale nonlinear optimization problems in this paper we give a global convergence analysis of a basic version of an sqp algorithm described in p t boggs a j kearsley and j w tolle siam j optim 9 1999 pp 755 778 for the solution of large scale nonlinear inequality constrained optimization problems several procedures and options have been added to the basic algorithm to improve the practical performance some of these are also analyzed the important features of the algorithm include the use of a constrained merit function to assess the progress of the iterates and a sequence of approximate merit functions that are less expensive to evaluate it also employs an interior point quadratic programming solver that can be terminated early to produce a truncated step

a feasible sequential linear equation method for inequality constrained optimization in this paper by means of the concept of the working set which is an estimate of the active set we propose a feasible sequential linear equation algorithm for solving inequality constrained optimization problems at each iteration of the proposed algorithm we first solve one system of linear equations with a coefficient matrix of size m m where m is the number of constraints to compute the working set we then solve a subproblem which consists of four reduced systems of linear equations with a common coefficient matrix unlike existing qp free algorithms the subproblem is concerned with only the constraints corresponding to the working set the constraints not in the working set are neglected consequently the dimension of each subproblem is not of full dimension without assuming the isolatedness of the stationary points we prove that every accumulation point of the sequence generated by the proposed algorithm is a kkt point of the problem moreover after finitely many iterations the working set becomes independent of the iterates and is essentially the same as the active set of the kkt point in other words after finitely many steps only those constraints which are active at the solution will be involved in the subproblem under some additional conditions we show that the convergence rate is two step superlinear or even q superlinear we also report some preliminary numerical experiments to show that the proposed algorithm is practicable and effective for the test problems

entropy encoding hilbert space and karhunen loeve transforms by introducing hilbert space and operators we show how probabilities approximations and entropy encoding from signal and image processing allow precise formulas and quantitative estimates our main results yield orthogonal bases which optimize distinct measures of data encoding

a globally and superlinearly convergent feasible qp free method for nonlinear programming in this paper we propose a qp free type algorithm which solves the problem of minimizing a smooth function subject to smooth inequality constraints in contrast with the sqp methods each iteration this algorithm only needs to solve systems of linear equations which are derived from the equality part in the kkt first order optimality conditions it is observed that if the quasi newton direction is zero we can obtain a direction of descent by dropping a constraint from the active set at the current iterate a high order modified direction is introduced in order to prevent maratos effect global and superlinear convergence are proven under some suitable conditions

nonlinear programming via an exact penalty function global analysis in this paper we motivate and describe an algorithm to solve the nonlinear programming problem the method is based on an exact penalty function and possesses both global and superlinear convergence properties we establish the global qualities here the superlinear nature is proven in 7 the numerical implementation techniques are briefly discussed and preliminary numerical results are given

a successive quadratic programming algorithm with global and superlinear convergence properties this paper presents a successive quadratic programming algorithm for solving general nonlinear programming problems in order to avoid the maratos effect direction finding subproblems are derived by modifying the second order approximations to both objective and constraint functions of the problem we prove that the algorithm possesses global and superlinear convergence properties

a modified sqp filter method and its global convergence for current sequential programming sqp type algorithms there exist two problems one is that there are many quadratic programming subproblems are needed to be solved per iteration the other is that the search direction may be unbounded and caused the sequence to be divergent in this paper we presented a modified sqp filter method based on the modified quadratic subproblem proposed in zhou g l zhou a modified sqp method and its global convergence j global optim 11 1997 193 205 this method has no demand on the initial point and need not using a penalty parameter which could be problematic to obtain what is more the subproblem is feasible at each iterate point under some conditions the global convergence property is obtained

a qp free globally convergent locally superlinearly convergent algorithm for inequality constrained optimization an algorithm is proposed for the minimization of a smooth function subject to smooth inequality constraints unlike sequential quadratic programming type methods this algorithm does not involve the solution of quadratic programs but merely that of linear systems of equations locally the iteration can be viewed as a perturbation of a quasi newton iteration on both the primal and dual variables for the solution of the equalities in the kuhn tucker first order conditions of optimality it is observed that provided the current iterate is feasible and the current multiplier estimates are strictly positive the primal component of the quasi newton direction is a direction of descent for the objective function this fact is used to induce global convergence without the need of a surrogate merit function a careful bending of the search direction prevents any maratos like effect and local superlinear convergence is proven while the algorithm requires that an initial feasible point be available the succe

a type of efficient feasible sqp algorithms for inequality constrained optimization in this paper motivated by zhu et al methods z b zhu k c zhang j b jian an improved sqp algorithm for inequality constrained optimization math meth oper res 58 2003 271 282 zhibin zhu jinbao jian an efficient feasible sqp algorithm for inequality constrained optimization nonlinear anal real world appl 10 2 2009 1220 1228 we propose a type of efficient feasible sqp algorithms to solve nonlinear inequality constrained optimization problems by solving only one qp subproblem with a subset of the constraints estimated as active a class of revised feasible descent directions are generated per single iteration these methods are implementable and globally convergent we also prove that the algorithms have superlinear convergence rate under some mild conditions

an rqp algorithm using a differentiable exact penalty function for inequality constrained problems in this paper we propose a recursive quadratic programming algorithm for nonlinear programming problems with inequality constraints that uses as merit function a differentiable exact penalty function the algorithm incorporates an automatic adjustment rule for the selection of the penalty parameter and makes use of an armijo type line search procedure that avoids the need to evaluate second order derivatives of the problem functions we prove that the algorithm possesses global and superlinear convergence properties numerical results are reported

multilevel least change newton like methods for equality constrained optimization problems this paper deals with the application of multilevel least change newton like methods for solving twice continuously differentiable equality constrained optimization problems we define multilevel partial inverse least change updates multilevel least change newton like methods without derivatives and multilevel projections of fragments of the matrix for newton like methods without derivatives local andq superlinear convergence of these methods is proved the theorems here also imply local andq superlinear convergence of many standard newton like methods for nonconstrained and equality constraine optimization problems

a robust trust region algorithm with a nonmonotonic penalty parameter scheme for constrained optimization an algorithm for solving the problem of minimizing a nonlinear function subject to equality constraints is introduced this algorithm is a trust region algorithm in computing the trial step a projected hessian technique is used that converts the trust region subproblem to one similar to that for the unconstrained case to force global convergence the augmented lagrangian is employed as a merit function one of the main advantages of this algorithm is the way that the penalty parameter is updated we introduce an updating scheme that allows for the first time to the best of our knowledge the penalty parameter to be decreased whenever it is warranted the behavior of this penalty parameter is studied a convergence theory for this algorithm is presented it is shown that this algorithm is globally convergent and that the globalization strategy will not disrupt fast local convergence the local rate of convergence is also discussed this theory is sufficiently general so that it holds for any algorithm t

a strategy for global convergence in a sequential quadratic programming algorithm n a previous work p boggs and j tolle siam j numer anal 21 1984 pp 1146 1161 the authors introduced a merit function for use with the sequential quadratic programming sqp algorithm for solving nonlinear programming problems here further theoretical justification including a global convergence theorem is provided in addition modifications are suggested that allow the efficient implementation of the merit function while maintaining the important convergence properties numerical results are presented demonstrating the effectiveness of the procedure

learning from examples in large neural networks a statistical mechanical theory of learning from examples in layered networks at finite temperature is studied when the training error is a smooth function of continuously varying weights the generalization error falls off asymptotically as the inverse number of examples by analytical and numerical studies of single layer perceptrons we show that when the weights are discrete the generalization error can exhibit a discontinuous transition to perfect generalization for intermediate sizes of the example set the state of perfect generalization coexists with a metastable spin glass state

learnability and the vapnik chervonenkis dimension valiant s learnability model is extended to learning classes of concepts defined by regions in euclidean space e n the methods in this paper lead to a unified treatment of some of valiant s results along with previous results on distribution free convergence of certain pattern recognition algorithms it is shown that the essential condition for distribution free learnability is finiteness of the vapnik chervonenkis dimension a simple combinatorial parameter of the class of concepts to be learned using this parameter the complexity and closure properties of learnable classes are analyzed and the necessary and sufficient conditions are provided for feasible learnability

convergence properties of algorithms for nonlinear optimization this paper reviews some of the most successful methods for unconstrained constrained and nondifferentiable optimization calculations particular attention is given to the contribution that theoretical analysis has made to the development of algorithms it seems that practical considerations provide the main new ideas and that subsequent theoretical studies give improvements to algorithms coherence to the subject and better understanding

a note on the computation of an orthonormal basis for the null space of a matrix a highly regarded method to obtain an orthonormal basis z for the null space of a matrix a t is the qr decomposition of a where q is the product of householder matrices in several optimization contexts a x varies continuously with x and it is desirable the z x vary continuously also in this note we demonstrate that the standard implementation of the qr decomposition does not yield an orthonormal basis z x whose elements vary continuously with x we suggest three possible remedies

on the location of directions of infinite descent for nonlinear programming algorithms there is much current interest in general equality constrained quadratic programming problems both for their own sake and for their applicability to active set methods for nonlinear programming in the former case typically the issues are existence of solutions and their determination in the latter instance nonexistence of solutions gives rise to directions of infinite descent such directions may subsequently be used to determine a more desirable active set the generalised cholesky decomposition of relevant matrices enables us to answer the question of existence and to determine directions of infinite descent when applicable in an efficient and stable manner the methods examined are related to implementations that are suitable for null space range space and lagrangian methods

towards more practical average bounds on supervised learning in this paper we describe a method which enables us to study the average generalization performance of learning directly via hypothesis testing inequalities the resulting theory provides a unified viewpoint of average case learning curves of concept learning and regression in realistic learning problems not necessarily within the bayesian framework the advantages of the theory are that it alleviates the practical pessimism frequently claimed for the results of the vapnik chervonenkis vc theory and its alike and provides general insights into generalization besides the bounds on learning curves are directly related to the number of adjustable system weights although the theory is based on an approximation assumption and cannot apply to the worst case learning setting the precondition of the assumption is mild and the approximation itself is only a sufficient condition for the validity of the theory we illustrate the results with numerical simulations and apply the theory to examining the generalization ability of combination of neural networks

on the practical applicability of vc dimension bounds this article addresses the question of whether some recent vapnik chervonenkis vc dimension based bounds on sample complexity can be regarded as a practical design tool specifically we are interested in bounds on the sample complexity for the problem of training a pattern classifier such that we can expect it to perform valid generalization early results using the vc dimension while being extremely powerful suffered from the fact that their sample complexity predictions were rather impractical more recent results have begun to improve the situation by attempting to take specific account of the precise algorithm used to train the classifier we perform a series of experiments based on a task involving the classification of sets of vowel formant frequencies the results of these experiments indicate that the more recent theories provide sample complexity predictions that are significantly more applicable in practice than those provided by earlier theories however we also find that the recent theories still have significant shortcomings

local properties of algorithms for minimizing nonsmooth composite functions this paper considers local convergence and rate of convergence results for algorithms for minimizing the composite functionf x f x h c x wheref andc are smooth buth c may be nonsmooth local convergence at a second order rate is established for the generalized gauss newton method whenh is convex and globally lipschitz and the minimizer is strongly unique local convergence at a second order rate is established for a generalized newton method when the minimizer satisfies nondegeneracy strict complementarity and second order sufficiency conditions assuming the minimizer satisfies these conditions necessary and sufficient conditions for a superlinear rate of convergence for curvature approximating methods are established necessary and sufficient conditions for a two step superlinear rate of convergence are also established when only reduced curvature information is available all these local convergence and rate of convergence results are directly applicable to nonlinearing programming problems

infeasible path optimal design methods with applications to aerodynamic shape optimization we consider optimal design of physical systems described by a nonlinear boundary value problem we propose a sequential quadratic programming method specifically tailored to the solution of this class of design problems a particular representation of the null space of the constraint jacobian that exploits its specific structure is employed the resulting method avoids resolution of nonlinear behavior at the optimization iterations while keeping the size of the problem as small as that of conventional approaches three variants of the method are developed and discussed these entail the solution of either two or three linear systems involving the jacobian matrix of the discretized form of the boundary value problem the method is used to solve aerodynamic design problems involving nonlinear transonic flow no special provisions are made for treating discontinuities and therefore the present implementation of the method is limited to problems with no shocks problems with up to 90 shape design variables are solved numerical results demonstrate a substantial performance improvement over conventional methods

acceleration of the frame algorithm shows how polynomial acceleration techniques which have been developed for the solution of large linear systems can be employed to improve and accelerate the frame algorithm these methods permit a reduction in the number of necessary iterations by an order of magnitude when the frame algorithm is slow the author gives several examples from the theory of irregular sampling from wavelet theory and from gabor theory where these methods are probably mandatory for efficient reconstruction

on the convergence of constrained optimization methods with accurate hessian information on a subspace this paper analyzes methods of the type proposed by coleman and conn for nonlinearly constrained optimization it is shown that if the reduced hessian approximation is sufficiently accurate then the method generates a sequence of iterates that converges one step superlinearly this result applies to a quasi newton implementation if the reduced exact hessian is used the method has an r order equal to that of the secant method a similar result for a modified version of successive quadratic programming is also proved finally some parallels between convergence results for methods that approximate the reduced hessian method and multiplier methods that use the reduced hessian inverse are pointed out

a simple feasible sqp method for inequality constrained optimization with global and superlinear convergence in this paper a simple feasible sqp method for nonlinear inequality constrained optimization is presented at each iteration we need to solve one qp subproblem only after solving a system of linear equations a new feasible descent direction is designed the maratos effect is avoided by using a high order corrected direction under some suitable conditions the global and superlinear convergence can be induced in the end numerical experiments show that the method in this paper is effective

suboptimality of the karhunen loeve transform for transform coding we examine the performance of the karhunen loeve transform klt for transform coding applications the klt has long been viewed as the best available block transform for a system that orthogonally transforms a vector source scalar quantizes the components of the transformed vector using optimal bit allocation and then inverse transforms the vector this paper treats fixed rate and variable rate transform codes of non gaussian sources the fixed rate approach uses an optimal fixed rate scalar quantizer to describe the transform coefficients the variable rate approach uses a uniform scalar quantizer followed by an optimal entropy code and each quantized component is encoded separately earlier work shows that for the variable rate case there exist sources on which the klt is not unique and the optimal quantization and coding stage matched to a worst klt yields performance as much as 1 5 db worse than the optimal quantization and coding stage matched to a best klt in this paper we strengthen that result to show that in both the fixed rate and the variable rate coding frameworks there exist sources for which the performance penalty for using a worst klt can be made arbitrarily large further we demonstrate in both frameworks that there exist sources for which even a best klt gives suboptimal performance finally we show that even for vector sources where the klt yields independent coefficients the klt can be suboptimal for fixed rate coding

a quantum computing primer for operator theorists this is an exposition of some of the aspects of quantum computation and quantum information that have connections with operator theory after a brief introduction we discuss quantum algorithms we outline basic properties of quantum channels or equivalently completely positive trace preserving maps the main theorems for quantum error detection and correction are presented and we conclude with a description of a particular passive method of quantum error correction

karhunen loeve approximation of random fields by generalized fast multipole methods kl approximation of a possibly instationary random field a x l2 dp l d subject to prescribed meanfield ea x a x dp and covariance va x x a x ea x a x ea x dp in a polyhedral domain d rd is analyzed we show how for stationary covariances va x x ga x x with ga z analytic outside of z 0 an m term approximate kl expansion am x of a x can be computed in log linear complexity the approach applies in arbitrary domains d and for nonseparable covariances ca it involves galerkin approximation of the kl eigenvalue problem by discontinuous finite elements of degree p 0 on a quasiuniform possibly unstructured mesh of width h in d plus a generalized fast multipole accelerated krylov eigensolver the approximate kl expansion am x of a x has accuracy o exp bm1 d if ga is analytic at z 0 and accuracy o m k d if ga is ck at zero it is obtained in o mn logn b operations where n o h d

an sqp method for general nonlinear programs using only equality constrained subproblems in this paper we describe a new version of a sequential equality constrained quadratic programming method for general nonlinear programs with mixed equality and inequality constraints compared with an older version p spellucci han s method without solving qp in a auslender w oettli j stoer eds optimization and optimal control lecture notes in control and information sciences vol 30 springer berlin 1981 pp 123 141 it is much simpler to implement and allows any kind of changes of the working set in every step our method relies on a strong regularity condition as far as it is applicable the new approach is superior to conventional sqp methods as demonstrated by extensive numcrical tests 1998 the mathematical programming society inc published by elsevier science b v

line search filter methods for nonlinear programming local convergence a line search method is proposed for nonlinear programming using fletcher and leyffer s filter method which replaces the traditional merit function a simple modification of the method proposed in a companion paper siam j optim 16 2005 pp 1 31 introducing second order correction steps is presented it is shown that the proposed method does not suffer from the maratos effect so that fast local convergence to second order sufficient local solutions is achieved

frame sequences and representations for samplable random processes abstract in this work we characterize random processes which can be linearly determined using a frame sequence of random variables in particular these could be the discrete samples of a continuous time process we study the stable representation of continuous time processes by means of discrete samples or measurements of the original process finally we study how these representations can be applied to reduce the effects of reconstructing a random signal from samples corrupted by additive noise

analysis of unbounded operators and random motion we study infinite weighted graphs with view to limits at infinity or boundaries at infinity examples of such weighted graphs arise in infinite in practice that means very large networks of resistors or in statistical mechanics models for classical or quantum systems however more generally our analysis includes reproducing kernel hilbert spaces and associated operators on them if x is some infinite set of vertices or nodes in applications the essential ingredient going into the definition is a reproducing kernel hilbert space it measures the differences of functions on x evaluated on pairs of points in x moreover the hilbert norm squared in h x will represent a suitable measure of energy associated unbounded operators will define a notion or dissipation it can be a graph laplacian or a more abstract unbounded hermitian operator defined from the reproducing kernel hilbert space under study we prove that there are two closed subspaces in reproducing kernel hilbert space h x that measure

four types of learning curves if machines are learning to make decisions given a number of examples the generalization error e t is defined as the average probability that an incorrect decision is made for a new example by a machine when trained with t examples the generalization error decreases as t increases and the curve e t is called a learning curve the present paper uses the bayesian approach to show that given the annealed approximation learning curves can be classified into four asymptotic types if the machine is deterministic with noiseless teacher signals then 1 e at 1 when the correct machine parameter is unique and 2 e at 2 when the set of the correct parameters has a finite measure if the teacher signals are noisy then 3 e at 1 2 for a deterministic machine and 4 e c at 1 for a stochastic machine

unification of basic and composite nondifferentiable optimization there are two main classes of iterative methods in nondifferentiable optimization ndo in thebasic ndo the information is limited to the objective function and at least one element of its subdifferential while in thecomposite ndo the objective function is split into a sum of a smooth and a nonsmooth function our work unifies these two approaches for benefiting of their respective advantages

exponential or polynomial learning curves case based studies learning curves exhibit a diversity of behaviors such as phase transition however the understanding of learning curves is still extremely limited and existing theories can give the impression that without empirical studies e g cross validation one can probably do nothing more than qualitative interpretations in this note we propose a theory of learning curves based on the idea of reducing learning problems to hypothesistesting ones this theory provides a simple approach that is potentially useful for predicting and interpreting a diversity of learning curve behaviors qualitatively and quantitatively and it applies to finite training sample size and finite learning machine and for learning situations not necessarily within the bayesian framework we illustrate the results by examining some exponential learning curve behaviors observed in cohn and tesauro 1992 s experiment

a recursive quadratic programming algorithm that uses differentiable exact penalty functions in this paper a recursive quadratic programming algorithm for solving equality constrained optimization problems is proposed and studied the line search functions used are approximations to fletcher s differentiable exact penalty function global convergence and local superlinear convergence results are proved and some numerical results are given

diagonals of normal operators with finite spectrum let x lambda1 lambdan be a finite set of complex numbers and let a be a normal operator with spectrum x that acts on a separable hilbert space h relative to a fixed orthonormal basis e1 e2 for h a gives rise to a matrix whose diagonal is a sequence d d1 d2 with the property that each of its terms dn belongs to the convex hull of x not all sequences with that property can arise as the diagonal of a normal operator with spectrum x the case where x is a set of real numbers has received a great deal of attention over the years and is reasonably well though incompletely understood in this work we take up the case in which x is the set of vertices of a convex polygon in the critical sequences d turn out to be those that accumulate rapidly in x in the sense that sigmainfinityn 1 dist dn x s d e gammax with the following property if s d not equal 0 then d is not the diagonal of any such operator a we also show that while this is the only obstruction when n 2 there are other as yet unknown obstructions when n 3

continuous time gradient like descent algorithm for constrained convex unknown functions penalty method application abstract this paper suggests a novel continuous time gradient descent algorithm of a constrained convex unknown function with a stochastic noise in the observed data the penalty function approach is employed to introduce the restrictions of the system and to provide a successful optimization process the solution is restricted to a static scheme dealing with the class of strongly convex functions subject to a set of constraints to estimate the stochastic gradient we employ a modified version of the synchronous detection method all the parameters of the proposed approach are decreasing in time to both compensate noise effect in the observations and to provide the mean square convergence of the suggested approach we present two different numerical examples to validate the contributions of the paper

how tight are the vapnik chervonenkis bounds we describe a series of numerical experiments that measure the average generalization capability of neural networks trained on a variety of simple functions these experiments are designed to test the relationship between average generalization performance and the worst case bounds obtained from formal learning theory using the vapnik chervonenkis vc dimension blumer et al 1989 haussler et al 1990 recent statistical learning theories tishby et al 1989 schwartz et al 1990 suggest that surpassing these bounds might be possible if the spectrum of possible generalizations has a gap near perfect performance we indeed find that in some cases the average generalization is significantly better than the vc bound the approach to perfect performance is exponential in the number of examples m rather than the 1 m result of the bound however in these cases we have not found evidence of the gap predicted by the above statistical theories in other cases we do find the 1 m behavior of the vc bound

sequential quadratic programming for certain parameter identification problems we analyze the method of sequential quadratic programming for equality constrained minimization problems in hilbert spaces of functions and for the discrete approximations of such problems in the context of an elliptic parameter identification problem we show how the discretization can be constructed so as to preserve the convergence behavior of the iterates for the infinite dimensional problem in the finite dimensional approximations we use the structure of the parameter identification problem to reduce the size of the linear system for the sqp step and verify nondegeneracy of the constraints

a trust region algorithm for equality constrained optimization a trust region algorithm for equality constrained optimization is proposed that employs a differentiable exact penalty function under certain conditions global convergence and local superlinear convergence results are proved

data compression and harmonic analysis in this paper we review some recent interactions between harmonic analysis and data compression the story goes back of course to shannon s r d theory in the case of gaussian stationary processes which says that transforming into a fourier basis followed by block coding gives an optimal lossy compression technique practical developments like transform based image compression have been inspired by this result in this paper we also discuss connections perhaps less familiar to the information theory community growing out of the field of harmonic analysis recent harmonic analysis constructions such as wavelet transforms and gabor transforms are essentially optimal transforms for transform coding in certain settings some of these transforms are under consideration for future compression standards we discuss some of the lessons of harmonic analysis in this century typically the problems and achievements of this field have involved goals that were not obviously related to practical data compression and have used a language not immediately accessible to outsiders nevertheless through an extensive generalization of what shannon called the sampling theorem harmonic analysis has succeeded in developing new forms of functional representation which turn out to have significant data compression interpretations we explain why harmonic analysis has interacted with data compression and we describe some interesting recent ideas in the field that may affect data compression in the future

decision theoretic generalizations of the pac model for neural net and other learning applications we describe a generalization of the pac learning model that is based on statistical decision theory in this model the learner receives randomly drawn examples each example consisting of an instance x in x and an outcome y in y and tries to find a hypothesis h x a where h in h that specifies the appropriate action a in a to take for each instance x in order to minimize the expectation of a loss l y a here x y and a are arbitrary sets l is a real valued function and examples are generated according to an arbitrary joint distribution on x times y special cases include the problem of learning a function from x into y the problem of learning the conditional probability distribution on y given x regression and the problem of learning a distribution on x density estimation we give theorems on the uniform convergence of empirical loss estimates to true expected loss rates for certain hypothesis spaces h and show how this implies learnability with bounded sample size disregarding computational complexity as an application we give distribution independent upper bounds on the sample size needed for learning with feedforward neural networks our theorems use a generalized notion of vc dimension that applies to classes of real valued functions adapted from pollard s work and a notion of capacity and metric dimension for classes of functions that map into a bounded metric space supersedes 89 30 and 90 52 also in information and computation vol 100 no 1 september 1992

on the local convergence of quasi newton methods for constrained optimization we consider the application of a general class of quasi newton methods to the solution of the classical equality constrained nonlinear optimization problem specifically we develop necessary and sufficient conditions for the q superlinear convergence of such methods and present a companion linear convergence theorem the essential conditions relate to the manner in which the hessian of the lagrangian function is approximated

applications of automata and graphs labeling operators in hilbert space ii we introduced a family of infinite graphs directly associated with a class of von neumann automaton model ag these are finite state models used in symbolic dynamics stimuli models and in control theory in the context of groupoid von neumann algebras and an associated fractal group we prove a classification theorem for representations of automata

fixed point quasi newton methods this article studies iterative methods defined by x k 1 phi x k e k where x k in mathbb r n and e k lie on a space of parameters sufficient conditions are established for local convergence and for convergence at an ideal linear or superlinear rate a theory of least change secant update methods for this class of processes is developed several examples are given showing a wide range of applications of the new theory key words quasi

a successive quadratic programming method that uses new corrections for search directions abstract in this paper a successive quadratic programming method for solving general nonlinear programming problems is proposed and studied in order to avoid the maratos effect we present new corrections for the search directions global convergence and local superlinear convergence results are proved and some numerical results are reported

a constrained min max algorithm for rival models of the same economic system there are well established rival theories about the economy these have in turn led to the development of rival models purporting to represent the economic system the models are large systems of discrete time nonlinear dynamic equations observed data of the real system does not in general provide sufficient information for statistical methods to invalidate all but one of the rival models in such a circumstance there is uncertainty about which model to use in the formulation of policy prudent policy design would suggest that a model based policy should take into account all the rival models this is achieved as a pooling of the models the pooling that yields the policy which is robust to model choice is formulated as a constrained min max problem the minimization is over the decision variables and the maximization is over the rival models only equality constraints are considered

maintaining the positive difiniteness of the matrices in reduced secant methods for equality constrained optimization we propose an algorithm for minimizing a functionf on n in the presence ofm equality constraintsc that locally is a reduced secant method the local method is globalized using a nondifferentiable augmented lagrangian whose decrease is obtained by both a longitudinal search that decreases mainlyf and a transversal search that decreases mainly c our main objective is to show that the longitudinal path can be designed to maintain the positive definiteness of the reduced matrices by means of the positivity of kt k where k is the change in the reduced gradient and k is the reduced longitudinal displacement

on secant updates for use in general constrained optimization in this paper we present two new classes of successive quadratic programming sqp secant methods for the equality constrained optimization problem one class of methods uses the sqp augmented lagrangian formulation while the other class uses the sqp lagrangian formulation we demonstrate under the standard assumptions that in both cases the bfgs and dfp versions of the algorithm are locally q superlinearly convergent to our knowledge this is the first time that either local or q superlinear convergence has been established for an sqp lagrangian secant method which uses either the bfgs or dfp updating philosophy and assumes no more than the standard assumptions since the standard assumptions do not require positive definiteness of the hessian of the lagrangian at the solution it is no surprise that our bfgs and dfp updates possess the hereditary positive definiteness property only on a proper subspace

identification of spatially varying parameters in distributed parameter systems by discrete regularization abstract identification of spatially varying parameters in distributed parameter systems from noisy data is an ill posed problem the regularization identification approach developed by c kravaris and j h seinfeld siam j control optim 23 1985 217 241 provides stable approximate solutions to that problem in this work a discretized minimization of the smoothing functional is proposed by using finite dimensional convergent approximations in sobolev spaces a convergence theorem for the discretized minimization of the smoothing functional is established the performance of this discrete regularization approach is evaluated by numerical experiments on the identification of spatially varying diffusivity in the two dimensional diffusion equation

on the convergence of a new trust region algorithm in this paper we present a new trust region algorithm for general nonlinear constrained optimization problems the algorithm is based on the l infty exact penalty function under very mild conditions global convergence results for the algorithm are given local convergence properties are also studied it is shown that the penalty parameter generated by the algorithm will be eventually not less than the l 1 norm of the lagrange multipliers at the accumulation point it is proved that the method is equivalent to the sequential quadratic programming method for all large k hence superlinearly convergent results of the sqp method can be applied numerical results are also reported

a second derivative sqp method global convergence sequential quadratic programming sqp methods form a class of highly efficient algorithms for solving nonlinearly constrained optimization problems although second derivative information may often be calculated there is little practical theory that justifies exact hessian sqp methods in particular the resulting quadratic programming qp subproblems are often nonconvex and thus finding their global solutions may be computationally nonviable this paper presents a second derivative sqp method based on quadratic subproblems that are either convex and thus may be solved efficiently or need not be solved globally additionally an explicit descent constraint is imposed on certain qp subproblems which guides the iterates through areas in which nonconvexity is a concern global convergence of the resulting algorithm is established

an example of irregular convergence in some constrained optimization methods that use the projected hessian in this paper we give examples illustrating the behavior of the coleman conn horizontal vertical method and of successive quadratic programming with a hessian approximation exact on the tangent space of the constraints one example shows that these methods in general are not one step superlinearly convergent

an extension of wiener integration with the use of operator theory with the use of tensor product of hilbert space and a diagonalization procedure from operator theory we derive an approximation formula for a general class of stochastic integrals further we establish a generalized fourier expansion for these stochastic integrals in our extension we circumvent some of the limitations of the more widely used stochastic integral due to wiener and ito i e stochastic integration with respect to brownian motion finally we discuss the connection between the two approaches as well as a priori estimates and applications

on solving nonlinear equations with simple singularities or nearly singular solutions the paper begins with a review of the convergence theory for newton s method near simple and other regular singularities followed by a brief discussion of the inherent difficulty of singular problems and the effect of rounding errors on the achievable solution accuracy then several ways to accelerate the slow convergence of newton s method are described in some detail these are richardson extrapolation overrelaxation bordering by a singularity condition and appending the usual linear model by a quadratic term the last two approaches require limited second derivative information which can be obtained analytically by differencing or through secant updating numerical comparisons reported in the final sections lead to the conclusion that the collection of the additional information pays off in terms of speed and accuracy

equivalence between representations for samplable stochastic processes and its relationship with riesz bases we characterize random signals which can be linearly determined by their samples this problem is related to the question of the representation of random variables by means of a countable riesz basis we study different representations for processes which are linearly determined by a countable riesz basis this concerns the representation of continuous time processes by means of discrete samples

a family of descent functions for constrained optimization in order to achieve a robust implementation of methods for nonlinear programming problems it is necessary to devise a procedure which can be used to test whether or not a prospective step would yield a better approximation to the solution than the current iterate in this paper we present a family of descent or merit functions which are shown to be compatible with local q superlinear convergence of newton and quasi newton methods a simple algorithm is used to verify that good descent and convergence properties are possible using this merit function

nonlinear programming and nonsmooth optimization by successive linear programming methods are considered for solving nonlinear programming problems using an exactl1 penalty function lp like subproblems incorporating a trust region constraint are solved successively both to estimate the active set and to provide a foundation for proving global convergence in one particular method second order information is represented by approximating the reduced hessian matrix and coleman conn steps are taken a criterion for accepting these steps is given which enables the superlinear convergence properties of the coleman conn method to be retained whilst preserving global convergence and avoiding the maratos effect the methods generalize to solve a wide range of composite nonsmooth optimization problems and the theory is presented in this general setting a range of numerical experiments on small test problems is described

recent developments in constrained optimization constrained optimization problems occur in many applications of science engineering social science and medicine and hence it is useful for practitioners in these areas to be informed about numerical methods for solving such problems however improvements in optimization typically do not spread quickly to the non specialist literature this paper contains a brief summary of two recent developments in the solution of nonlinearly constrained problems direct use of the lagrangian function and linearization of nonlinear constraints in order to highlight the new approaches we also discuss techniques used in older methods to illustrate the capabilities of modern optimization methods a brief description is given of the development of methods to solve large nonlinearly constrained problems that arise in the electrical power industry the general nonlinear programming problem may be stated in the following form

a sequential quadratic optimization algorithm with rapid infeasibility detection we present a sequential quadratic optimization sqo algorithm for nonlinear constrained optimization the method attains all of the strong global and fast local convergence guarantees of classical sqo methods but has the important additional feature that fast local convergence is guaranteed when the algorithm is employed to solve infeasible instances a two phase strategy carefully constructed parameter updates and a line search are employed to promote such convergence the first phase subproblem determines the reduction that can be obtained in a local model of an infeasibility measure when the objective function is ignored the second phase subproblem then seeks to minimize a local model of the objective while ensuring that the resulting search direction attains a reduction in the local model of the infeasibility measure that is proportional to that attained in the first phase the subproblem formulations and parameter updates ensure that near an optimal solution the algorithm reduces to a classica

sharp primal superlinear convergence results for some newtonian methods for constrained optimization as is well known q superlinear or q quadratic convergence of the primal dual sequence generated by an optimization algorithm does not in general imply q superlinear convergence of the primal part primal convergence however is often of particular interest for the sequential quadratic programming sqp algorithm local primal dual quadratic convergence can be established under the assumptions of uniqueness of the lagrange multiplier associated to the solution and the second order sufficient condition at the same time previous primal q superlinear convergence results for sqp required strengthening of the first assumption to the linear independence constraint qualification in this paper we show that this strengthening of assumptions is actually not necessary specifically we show that once primal dual convergence is assumed or already established for primal superlinear rate one needs only a certain error bound estimate this error bound holds for example under the second order sufficient condition which is needed for primal dual local analysis in any case moreover in some situations even second order sufficiency can be relaxed to the weaker assumption that the multiplier in question is noncritical our study is performed for a rather general perturbed sqp framework which covers in addition to sqp and quasi newton sqp some other algorithms as well for example as a byproduct we obtain primal q superlinear convergence results for the linearly constrained augmented lagrangian methods for which no primal q superlinear rate of convergence results were previously available another application of the general framework is sequential quadratically constrained quadratic programming methods finally we discuss some difficulties with proving primal superlinear convergence for the stabilized version of sqp

infeasibility detection and sqp methods for nonlinear optimization this paper addresses the need for nonlinear programming algorithms that provide fast local convergence guarantees regardless of whether a problem is feasible or infeasible we present a sequential quadratic programming method derived from an exact penalty approach that adjusts the penalty parameter automatically when appropriate to emphasize feasibility over optimality the superlinear convergence of such an algorithm to an optimal solution is well known when a problem is feasible the main contribution of this paper however is a set of conditions under which the superlinear convergence of the same type of algorithm to an infeasible stationary point can be guaranteed when a problem is infeasible numerical experiments illustrate the practical behavior of the method on feasible and infeasible problems

on the characterization of q superlinear convergence of quasi newton methods for constrained optimization abstract in this note the authors present what they consider to be a short direct and self contained derivation of the boggs tolle wang characterization of q superlinear convergence for quasi newton methods for constrained optimization while they have stated that the three previous derivations boggs tolle and wang fontecilla steihaug and tapia and nocedal and overton leave something to be desired they quickly add that the present work was strongly influenced by these three papers indeed the basic idea that led to the present derivation was to attempt to parallel the nocedal overton derivation using a formulation of the quasi newton method that possessed the attribute that all necessary differentiations could be obtained in a straightforward manner as they have seen one of the formulations suggested by tapia possesses this property

on the realization of the wolfe conditions in reduced quasi newton methods for equality constrained optimization this paper describes a reduced quasi newton method for solving equality constrained optimization problems a major difficulty encountered by this type of algorithm is the design of a consistent technique for maintaining the positive definiteness of the matrices approximating the reduced hessian of the lagrangian a new approach is proposed in this paper the idea is to search for the next iterate along a piecewise linear path the path is designed so that some generalized wolfe conditions can be satisfied these conditions allow the algorithm to sustain the positive definiteness of the matrices from iteration to iteration by a mechanism that has turned out to be efficient in unconstrained optimization

a robust algorithm for optimization with general equality and inequality constraints an algorithm for general nonlinearly constrained optimization is presented which solves an unconstrained piecewise quadratic subproblem and a quadratic programming subproblem at each iterate the algorithm is robust since it can circumvent the difficulties associated with the possible inconsistency of qp subproblem of the original sqp method moreover the algorithm can converge to a point which satisfies a certain first order necessary optimality condition even when the original problem is itself infeasible which is a feature of burke and han s methods math programming 43 1989 pp 277 303 unlike burke and han s methods our algorithm does not introduce additional bound constraints the algorithm solves the same subproblems as the han powell sqp algorithm at feasible points of the original problem under certain assumptions it is shown that the algorithm coincides with the han powell method when the iterates are sufficiently close to the solution some global convergence results are proved and locally superlinear convergence results are also obtained preliminary numerical results are reported

a modified sqp method and its global convergence in this paper a modified sqp method is presented the algorithm starts from an arbitrary initial point and can overcome the maratos effect moreover it avoid choosing the penalty parameters under some reasonable conditions the global convergence is shown

a quasi newton strategy for the ssqp method for variational inequality and optimization problems the quasi newton strategy presented in this paper preserves one of the most important features of the stabilized sequential quadratic programming method the local convergence without constraint qualifications assumptions it is known that the primal dual sequence converges quadratically assuming only the second order sufficient condition in this work we show that if the matrices are updated by performing a minimization of a bregman distance which includes the classic updates the quasi newton version of the method converges superlinearly without introducing further assumptions also we show that even for an unbounded lagrange multipliers set the generated matrices satisfies a bounded deterioration property and the dennis more condition

nonlinear programming via an exact penalty function asymptotic analysis in this paper we consider the final stage of a global method to solve the nonlinear programming problem we prove 2 step superlinear convergence in the process of analyzing this asymptotic behavior we compare our method theoretically to the popular successive quadratic programming approach

discontinuous piecewise linear optimization a theoretical framework and a practical algorithm are presented to solve discontinuous piecewise linear optimization problems dealing with functions for which theridges are known a penalty approach allows one to consider such problems subject to a wide range of constraints involving piecewise linear functions although the theory is expounded in detail in the special case of discontinuous piecewiselinear functions it is straightforwardly extendable using standard nonlinear programming techniques tononlinear discontinuous piecewise differentiable functions the descent algorithm which is elaborated uses active set and projected gradient approaches it is a generalization of the ideas used by conn to deal with nonsmoothness in thel1 exact penalty function and it is based on the notion ofdecomposition of a function into a smooth and a nonsmooth part the constrained case is reduced to the unconstrained minimization of a piecewise linear l1 exact penalty function we also discuss how the algorithm is modified when it encounters degenerate points preliminary numerical results are presented the algorithm is applied to discontinuous optimization problems from models in industrial engineering 1998 the mathematical programming society inc published by elsevier science b v