on averaging and representation properties of the bfgs and related secant updates in this paper we present several representation theorems and averaging theorems for members of the difference class of secant updates introduced by brodlie et al j inst math appl 11 73 82 1973 major contributions are that the integral form of the mean value theorem leads to a proof that the bfgs update is pointwise the infinite average of all the updates on the one dimension manifold in the dennis class that connects the dfp secant update to the greenstadt update and that it can be expressed as the pointwise average of these latter two updates analogous results hold for all secant updates that belong to the difference class these results contribute new understanding of the structural properties of the highly popular bfgs secant update and other updates from the difference class

on the discovery of the good broyden method this short note traces the events that led to the unsymmetric rank one formula known as the good broyden update 5 6 which is widely used within derivative free mathematical software for solving a system of nonlinear equations

a derivative free optimization algorithm based on conditional moments in this paper we propose a derivative free optimization algorithm based on conditional moments for finding the maximizer of an objective function the proposed algorithm does not require calculation or approximation of any order derivative of the objective function the step size in iteration is determined adaptively according to the local geometrical feature of the objective function and a pre specified quantity representing the desired precision the theoretical properties including convergence of the method are presented numerical experiments comparing with the newton quasi newton and trust region methods are given to illustrate the effectiveness of the algorithm

an optimal control theory for nonlinear optimization abstract the karush kuhn tucker conditions for a given nonlinear programming problem are generated as the transversality conditions of an optimal control problem the directional derivatives of the objective and constraint functions supply the vector fields for the optimal control problem with the search vector as the control variable zero hamiltonian trajectories along the steepest descent of control lyapunov functions provide optimal optimization algorithms the optimality of the algorithm also depends upon the choice of a metric for the finite dimensional control space many well known algorithms such as newton s method the first order lagrangian method the steepest descent method and richardson s method to name a few are derived by minimizing the lie derivative of a quadratic control lyapunov function merit functions in optimization may also be generated using the concept of a control lyapunov function these results suggest that optimal control principles hold the potential for a unified theory for optimization

direct search methods then and now we discuss direct search methods for unconstrained optimization we give a modern perspective on this classical family of derivative free algorithms focusing on the development of direct search methods during their golden age from 1960 to 1971 we discuss how direct search methods are characterized by the absence of the construction of a model of the objective we then consider a number of the classical direct search methods and discuss what research in the intervening years has uncovered about these algorithms in particular while the original direct search methods were consciously based on straightforward heuristics more recent analysis has shown that in most but not all cases these heuristics actually suffice to ensure global convergence of at least one subsequence of the sequence of iterates to a first order stationary point of the objective function

on the convergence of pattern search algorithms we introduce an abstract definition of pattern search methods for solving nonlinear unconstrained optimization problems our definition unifies an important collection of optimization methods that neither compute nor explicitly approximate derivatives we exploit our characterization of pattern search methods to establish a global convergence theory that does not enforce a notion of sufficient decrease our analysis is possible because the iterates of a pattern search method lie on a scaled translated integer lattice this allows us to relax the classical requirements on the acceptance of the step at the expense of stronger conditions on the form of the step and still guarantee global convergence

karmarkar s linear programming algorithm and newton s method this paper describes a full dimensional version of karmarkar s linear programming algorithm theprojective scaling algorithm which is defined for any linear program in n having a bounded full dimensional polytope of feasible solutions if such a linear program hasm inequality constraints then it is equivalent under an injective affine mappingj n m to karmarkar s original algorithm for a linear program in m havingm n equality constraints andm inequality constraints karmarkar s original algorithm minimizes a potential functiong x and the projective scaling algorithm is equivalent to that version of karmarkar s algorithm whose step size minimizes the potential function in the step direction

optimization by direct search new perspectives on some classical and modern methods direct search methods are best known as unconstrained optimization techniques that do not explicitly use derivatives direct search methods were formally proposed and widely applied in the 1960s but fell out of favor with the mathematical optimization community by the early 1970s because they lacked coherent mathematical analysis nonetheless users remained loyal to these methods most of which were easy to program some of which were reliable in the past fifteen years these methods have seen a revival due in part to the appearance of mathematical analysis as well as to interest in parallel and distributed com puting this review begins by briefly summarizing the history of direct search methods and considering the special properties of problems for which they are well suited our focus then turns to a broad class of methods for which we provide a unifying framework that lends itself to a variety of convergence results the underlying principles allow general ization to handle bound constraints and linear constraints we also discuss extensions to problems with nonlinear constraints

hypergeometric solutions to the q painleve equation of type a 1 4 we consider the q painleve equation of type a 1 4 a version of q painleve v equation and construct a family of solutions expressible in terms of certain basic hypergeometric series we also present the determinant formula for the solutions

parallel variable metric algorithms for unconstrained optimization we classify straeter s ideas for parallel unconstrained optimization and apply them to huang s class of updating formulas straeter s rank one updating formula appears to be the only parallel extension within huang s class with the property of quadratic termination we also develop a parallel extension of broyden s 1965 rank one updating formula and prove quadratic termination finally we present numerical results obtained by testing the algorithms on several standard example problems

on the conditioning of the hessian approximation in quasi newton methods abstract a bound on the possible deterioration in the condition number of the inverse hessian approximation is derived for updating formulae of bfgs type we construct a nonlinear model for the gradient of the objective function along a chosen ray in the variable space the model has a free parameter which is determined by minimising the condition number bound numerical tests indicate that using such a device on the first iteration can lead to an improvement in the performance of several algorithms of quasi newton type

a survey of conjugate gradient algorithms for solution of extreme eigen problems of a symmetric matrix a survey of various conjugate gradient cg algorithms is presented for the minimum maximum eigen problems of a fixed symmetric matrix the cg algorithms are compared to a commonly used conventional method found in imsl it is concluded that the cg algorithms are more flexible and efficient than some of the conventional methods used in adaptive spectrum analysis and signal processing

statistical quasi newton a new look at least change a new method for quasi newton minimization outperforms bfgs by combining least change updates of the hessian with step sizes estimated from a wishart model of uncertainty the hessian update is in the broyden family but uses a negative parameter outside the convex range that is usually regarded as the safe zone for broyden updates although full newton steps based on this update tend to be too long excellent performance is obtained with shorter steps estimated from the wishart model in numerical comparisons to bfgs the new statistical quasi newton sqn algorithm typically converges with about 25 fewer iterations functions and gradient evaluations on the top 1 3 hardest unconstrained problems in the cute library typical improvement on the 1 3 easiest problems is about 5 the framework used to derive sqn provides a simple way to understand differences among various broyden updates such as bfgs and dfp and shows that these methods do not preserve accuracy of the hessian in a certain sense while the new method does in fact bfgs dfp and all other updates with nonnegative broyden parameters tend to inflate hessian estimates and this accounts for their observed propensity to correct eigenvalues that are too small more readily than eigenvalues that are too large numerical results on three new test functions validate these conclusions

on the use of curvature estimates in quasi newtonian methods abstract in previous work ford and ghandhari 1989 considered the use of nonlinear gradient models for quasi newton methods by means of such models values of the objective function in addition to values of the gradient were utilised in updating formulae numerical experiments provided evidence that such an approach could yield gains in terms of the evaluations required but at the cost of the solution of a nonlinear equation at each iteration in a subsequent paper ford and ghandhari to appear have developed alternative models which enable the parameter to be determined explicitly in this paper we shall consider further models of the general type introduced in the latter paper which determine the parameter explicitly by employing estimates of the curvature of the objective function thus following a technique first introduced by biggs 1971 1973 we conclude by presenting the results of numerical tests on the new methods

willmore blowups are never compact we prove a ojasiewicz simon gradient inequality for the willmore functional near willmore surfaces and we apply this inequality to exclude compact blowups for the willmore flow

a variable metric algorithm for unconstrained minimization without evaluation of derivatives this paper presents a modification of the bfgs method for unconstrained minimization that avoids computation of derivatives the gradients are approximated by the aid of differences of function values these approximations are calculated in such a way that a complete convergence proof can be given the presented algorithm is implementable no exact line search is required it is shown that if the objective function is convex and some usually required conditions hold the algorithm converges to a solution if the hessian matrix of the objective function is positive definite and satisfies a lipschitz condition in a neighbourhood of the solution then the rate of convergence is superlinear

nonmonotone bfgs trained recurrent neural networks for temporal sequence processing in this paper we propose a nonmonotone approach to recurrent neural networks training for temporal sequence processing applications this approach allows learning performance to deteriorate in some iterations nevertheless the network s performance is improved over time a self scaling bfgs is equipped with an adaptive nonmonotone technique that employs approximations of the lipschitz constant and is tested on a set of sequence processing problems simulation results show that the proposed algorithm outperforms the bfgs as well as other methods previously applied to these sequences providing an effective modification that is capable of training recurrent networks of various architectures

variable metric method for minimization this is a method for determining numerically local minima of differentiable functions of several variables in the process of locating each minimum a matrix which characterizes the behavior of the function about the minimum is determined for a region in which the function depends quadratically on the variables no more than n iterations are required where n is the number of variables by suitable choice of starting values and without modification of the procedure linear constraints can be imposed upon the variables

minimum hellinger distance estimation for finite mixture models abstract minimum hellinger distance estimates are considered for finite mixture models when the exact forms of the component densities are unknown in detail but are thought to be close to members of some parametric family minimum hellinger distance estimates are asymptotically efficient if the data come from a member of the parametric family and are robust to certain departures from the parametric family a new algorithm is introduced that is similar to the em algorithm a specialized adaptive density estimate is also introduced standard measures of robustness are discussed some difficulties are noted the robustness and asymptotic efficiency of the estimators are illustrated using simulations

quasi newton updates in abstract vector spaces we derive least change updates of linear mappings that approximate derivatives of functions defined on abstract vector spaces these correspond to well known matrix updates our approach has the value of demonstrating the minimal assumptions necessary to derive particular updates and hence their robustness in addition the choice of one or other update is indicated by the presence of natural inner or scalar products in the vector spaces all spaces considered are finite dimensional real vector spaces

reminiscences on the development of the variational approach to davidon s variable metric method in the mid 1960 s davidon s method was brought to the author s attention by m j d powell one of its earliest proponents its great efficacy in solving a rather difficult computational problem in which the author was involved led to an attempt to find a best updating formula best seemed to suggest least in the sense of some norm to further the stability of the method this led to the idea of minimizing a generalized quadratic frobenius norm with the quasi newton and symmetry constraints on the updates several interesting formulas were derived including the davidon fletcher powell formula as shown by goldfarb this approach was extended to the derivation of updates requiring no derivatives and to broyden like updates for the solution of simultaneous nonlinear equations attempts were made to derive minimum norm corrections in product form updates with an eye to preserving positive definiteness in the course of this attempt it was discovered that the dfp formula could be written as a product leading to some interesting theoretical developments finally a linearized product form update was developed which was competitive with the best update bfgs of that time

faster independent component analysis by preconditioning with hessian approximations independent component analysis ica is a technique for unsupervised exploration of multichannel data that is widely used in observational sciences in its classic form ica relies on modeling the data as linear mixtures of nongaussian independent sources the maximization of the corresponding likelihood is a challenging problem if it has to be completed quickly and accurately on large sets of real data this problem has been addressed by resorting to quasi newton methods which rely on sparse approximations of the hessian of the log likelihood however those approximations are not accurate when the ica model does not hold exactly as is often the case for real datasets we propose a new algorithm dubbed picard which makes use of sparse approximate hessians only as a preconditioner to the l bfgs algorithm refining the hessian approximation from a memory of the past iterates extensive numerical comparisons to several algorithms of the same class demonstrate the superior performance of the proposed technique especially on real data