on averaging and representation properties of the bfgs and related secant updates in this paper we present several representation theorems and averaging theorems for members of the difference class of secant updates introduced by brodlie et al j inst math appl 11 73 82 1973 major contributions are that the integral form of the mean value theorem leads to a proof that the bfgs update is pointwise the infinite average of all the updates on the one dimension manifold in the dennis class that connects the dfp secant update to the greenstadt update and that it can be expressed as the pointwise average of these latter two updates analogous results hold for all secant updates that belong to the difference class these results contribute new understanding of the structural properties of the highly popular bfgs secant update and other updates from the difference class

on the discovery of the good broyden method this short note traces the events that led to the unsymmetric rank one formula known as the good broyden update 5 6 which is widely used within derivative free mathematical software for solving a system of nonlinear equations

nonlinear digital filter realization by distributed arithmetic it is assumed that a nonlinear shift invariant digital filter is represented by a truncated discrete volterra series a matrix notation based on the properties of the kronecker product of matrices is derived for the nonlinear operator of order k in the volterra expansion the matrix notation shows how the well known distributed arithmetic originally proposed for linear digital filters can be adopted for the realization of nonlinear operators the characteristics of combinatorial and memory oriented structures are compared and the problem of complexity reduction is discussed

a derivative free optimization algorithm based on conditional moments in this paper we propose a derivative free optimization algorithm based on conditional moments for finding the maximizer of an objective function the proposed algorithm does not require calculation or approximation of any order derivative of the objective function the step size in iteration is determined adaptively according to the local geometrical feature of the objective function and a pre specified quantity representing the desired precision the theoretical properties including convergence of the method are presented numerical experiments comparing with the newton quasi newton and trust region methods are given to illustrate the effectiveness of the algorithm

an optimal control theory for nonlinear optimization abstract the karush kuhn tucker conditions for a given nonlinear programming problem are generated as the transversality conditions of an optimal control problem the directional derivatives of the objective and constraint functions supply the vector fields for the optimal control problem with the search vector as the control variable zero hamiltonian trajectories along the steepest descent of control lyapunov functions provide optimal optimization algorithms the optimality of the algorithm also depends upon the choice of a metric for the finite dimensional control space many well known algorithms such as newton s method the first order lagrangian method the steepest descent method and richardson s method to name a few are derived by minimizing the lie derivative of a quadratic control lyapunov function merit functions in optimization may also be generated using the concept of a control lyapunov function these results suggest that optimal control principles hold the potential for a unified theory for optimization

least change quasi newton updates for equality constrained optimization this paper investigates quasi newton updates for equality constrained optimization using a least change argument we derive a class of rank 3 updates to approximations of the one sided projection of the hessian of the lagrangian which keeps the appropriate part symmetric and possibly positive definite by imposing the usual assumptions we are able to prove 1 step superlinear convergence for one of these updates encouraging numerical results and comparisons with other previously analyzed updates are presented

superlinear convergence of symmetric huang s class of methods in this paper the problem of minimizing the functionalf d r n r is considered typical assumptions onf are assumed a class of quasi newton methods namely huang s class of methods is used for finding an optimal solution of this problem a new theorem connected with this class is presented by means of this theorem some convergence results known up till now only for the methods which satisfy quasi newton condition are extended that is the results of superlinear convergence of variable metric methods in the cases of exact and asymptotically exact minimization and the so called direct prediction case this theorem allows to interpretate one of the parameters as the scaling parameter

direct search methods then and now we discuss direct search methods for unconstrained optimization we give a modern perspective on this classical family of derivative free algorithms focusing on the development of direct search methods during their golden age from 1960 to 1971 we discuss how direct search methods are characterized by the absence of the construction of a model of the objective we then consider a number of the classical direct search methods and discuss what research in the intervening years has uncovered about these algorithms in particular while the original direct search methods were consciously based on straightforward heuristics more recent analysis has shown that in most but not all cases these heuristics actually suffice to ensure global convergence of at least one subsequence of the sequence of iterates to a first order stationary point of the objective function

on the convergence of pattern search algorithms we introduce an abstract definition of pattern search methods for solving nonlinear unconstrained optimization problems our definition unifies an important collection of optimization methods that neither compute nor explicitly approximate derivatives we exploit our characterization of pattern search methods to establish a global convergence theory that does not enforce a notion of sufficient decrease our analysis is possible because the iterates of a pattern search method lie on a scaled translated integer lattice this allows us to relax the classical requirements on the acceptance of the step at the expense of stronger conditions on the form of the step and still guarantee global convergence

karmarkar s linear programming algorithm and newton s method this paper describes a full dimensional version of karmarkar s linear programming algorithm theprojective scaling algorithm which is defined for any linear program in n having a bounded full dimensional polytope of feasible solutions if such a linear program hasm inequality constraints then it is equivalent under an injective affine mappingj n m to karmarkar s original algorithm for a linear program in m havingm n equality constraints andm inequality constraints karmarkar s original algorithm minimizes a potential functiong x and the projective scaling algorithm is equivalent to that version of karmarkar s algorithm whose step size minimizes the potential function in the step direction

efficient realizations of two dimensional quadratic digital filters realization structures of two dimensional finite support quadratic digital filters which are based on matrix decomposition are introduced the resulting structures consist generally of a number of parallel branches which operate simultaneously and independently on a common input array by proper decomposition of the coefficient matrix each parallel branch can be realized by a two dimensional finite impulse response digital filter in cascade with a squaring operator the lower upper triangular decomposition and the singular value lower upper decomposition is used in order to arrive at efficient structures with computational and hardware advantages

optimization by direct search new perspectives on some classical and modern methods direct search methods are best known as unconstrained optimization techniques that do not explicitly use derivatives direct search methods were formally proposed and widely applied in the 1960s but fell out of favor with the mathematical optimization community by the early 1970s because they lacked coherent mathematical analysis nonetheless users remained loyal to these methods most of which were easy to program some of which were reliable in the past fifteen years these methods have seen a revival due in part to the appearance of mathematical analysis as well as to interest in parallel and distributed com puting this review begins by briefly summarizing the history of direct search methods and considering the special properties of problems for which they are well suited our focus then turns to a broad class of methods for which we provide a unifying framework that lends itself to a variety of convergence results the underlying principles allow general ization to handle bound constraints and linear constraints we also discuss extensions to problems with nonlinear constraints

convergence properties of algorithms for nonlinear optimization this paper reviews some of the most successful methods for unconstrained constrained and nondifferentiable optimization calculations particular attention is given to the contribution that theoretical analysis has made to the development of algorithms it seems that practical considerations provide the main new ideas and that subsequent theoretical studies give improvements to algorithms coherence to the subject and better understanding

a study of type i intermittency of a circular differential equation under a discontinuous right hand side abstract in this paper we study a circular differential equation under a discontinuous periodic input developing a quadratic differential equations system on s 1 and a linear differential equations system in the minkowski space m 3 the symmetry groups of these two systems are respectively pso o 2 1 and so o 2 1 the poincare circle map is constructed exactly and a critical value c of the parameter is identified depending on of the input amplitude the equation may exhibit periodic subharmonic or quasiperiodic motions when varies from c to c there undergoes an inverse tangent bifurcation consequently the resultant poincare circle map offers one route to the quasiperiodicity via a type i intermittency in the parameter range of c the orbit generated by the poincare circle map is either m periodic or quasiperiodic when n m is a rational or an irrational number

parallel variable metric algorithms for unconstrained optimization we classify straeter s ideas for parallel unconstrained optimization and apply them to huang s class of updating formulas straeter s rank one updating formula appears to be the only parallel extension within huang s class with the property of quadratic termination we also develop a parallel extension of broyden s 1965 rank one updating formula and prove quadratic termination finally we present numerical results obtained by testing the algorithms on several standard example problems

on the conditioning of the hessian approximation in quasi newton methods abstract a bound on the possible deterioration in the condition number of the inverse hessian approximation is derived for updating formulae of bfgs type we construct a nonlinear model for the gradient of the objective function along a chosen ray in the variable space the model has a free parameter which is determined by minimising the condition number bound numerical tests indicate that using such a device on the first iteration can lead to an improvement in the performance of several algorithms of quasi newton type

the global convergence of a modified bfgs method for nonconvex functions abstract the standard bfgs method plays an important role among the quasi newton algorithms for constrained un constrained optimization problems however dai 2003 constructed a counterexample to demonstrate that this method may fail for non convex functions with inexact wolfe line searches and mascarenhas 2004 showed the nonconvergence of this method and other methods in the broyden family even with exact line search techniques these works motivate us to try to find another way to obtain another quasi newton method with better convergence based on the standard bfgs formula in this paper four approaches are used in the designed algorithm 1 a modified weak wolfe powell line search technique is introduced 2 if one defined condition is satisfied then the search direction and its associated step length are accepted and the next point is designed 3 otherwise a parabolic will be presented and it is regarded as the projection surface to avoid using the failed direction and the next point x k 1 is generated by a projection technique and 4 to easily obtain the global convergence of the proposed algorithm the projection point is not used in the current bfgs update but rather for all the next iterations the new algorithm possesses global convergence for general functions under the inexact modified weak wolfe powell line search technique and it is shown that other methods in the broyden class also have this property numerical results are reported to show the performance of the given algorithm and other similar methods

a survey of conjugate gradient algorithms for solution of extreme eigen problems of a symmetric matrix a survey of various conjugate gradient cg algorithms is presented for the minimum maximum eigen problems of a fixed symmetric matrix the cg algorithms are compared to a commonly used conventional method found in imsl it is concluded that the cg algorithms are more flexible and efficient than some of the conventional methods used in adaptive spectrum analysis and signal processing

statistical quasi newton a new look at least change a new method for quasi newton minimization outperforms bfgs by combining least change updates of the hessian with step sizes estimated from a wishart model of uncertainty the hessian update is in the broyden family but uses a negative parameter outside the convex range that is usually regarded as the safe zone for broyden updates although full newton steps based on this update tend to be too long excellent performance is obtained with shorter steps estimated from the wishart model in numerical comparisons to bfgs the new statistical quasi newton sqn algorithm typically converges with about 25 fewer iterations functions and gradient evaluations on the top 1 3 hardest unconstrained problems in the cute library typical improvement on the 1 3 easiest problems is about 5 the framework used to derive sqn provides a simple way to understand differences among various broyden updates such as bfgs and dfp and shows that these methods do not preserve accuracy of the hessian in a certain sense while the new method does in fact bfgs dfp and all other updates with nonnegative broyden parameters tend to inflate hessian estimates and this accounts for their observed propensity to correct eigenvalues that are too small more readily than eigenvalues that are too large numerical results on three new test functions validate these conclusions

on the use of curvature estimates in quasi newtonian methods abstract in previous work ford and ghandhari 1989 considered the use of nonlinear gradient models for quasi newton methods by means of such models values of the objective function in addition to values of the gradient were utilised in updating formulae numerical experiments provided evidence that such an approach could yield gains in terms of the evaluations required but at the cost of the solution of a nonlinear equation at each iteration in a subsequent paper ford and ghandhari to appear have developed alternative models which enable the parameter to be determined explicitly in this paper we shall consider further models of the general type introduced in the latter paper which determine the parameter explicitly by employing estimates of the curvature of the objective function thus following a technique first introduced by biggs 1971 1973 we conclude by presenting the results of numerical tests on the new methods

an alternative variational principle for variable metric updating we describe a variational principle based upon minimizing the extent to which the inverse hessian approximation sayh violates the quasi newton relation on the step immediately prior to the step used to constructh its application to the case when line searches are exact suggests use of the bfgs update

a variable metric algorithm for unconstrained minimization without evaluation of derivatives this paper presents a modification of the bfgs method for unconstrained minimization that avoids computation of derivatives the gradients are approximated by the aid of differences of function values these approximations are calculated in such a way that a complete convergence proof can be given the presented algorithm is implementable no exact line search is required it is shown that if the objective function is convex and some usually required conditions hold the algorithm converges to a solution if the hessian matrix of the objective function is positive definite and satisfies a lipschitz condition in a neighbourhood of the solution then the rate of convergence is superlinear

nonmonotone bfgs trained recurrent neural networks for temporal sequence processing in this paper we propose a nonmonotone approach to recurrent neural networks training for temporal sequence processing applications this approach allows learning performance to deteriorate in some iterations nevertheless the network s performance is improved over time a self scaling bfgs is equipped with an adaptive nonmonotone technique that employs approximations of the lipschitz constant and is tested on a set of sequence processing problems simulation results show that the proposed algorithm outperforms the bfgs as well as other methods previously applied to these sequences providing an effective modification that is capable of training recurrent networks of various architectures

variable metric method for minimization this is a method for determining numerically local minima of differentiable functions of several variables in the process of locating each minimum a matrix which characterizes the behavior of the function about the minimum is determined for a region in which the function depends quadratically on the variables no more than n iterations are required where n is the number of variables by suitable choice of starting values and without modification of the procedure linear constraints can be imposed upon the variables

variable metric random pursuit we consider unconstrained randomized optimization of smooth convex objective functions in the gradient free setting we analyze random pursuit rp algorithms with fixed f rp and variable metric v rp the algorithms only use zeroth order information about the objective function and compute an approximate solution by repeated optimization over randomly chosen one dimensional subspaces the distribution of search directions is dictated by the chosen metric variable metric rp uses novel variants of a randomized zeroth order hessian approximation scheme recently introduced by leventhal and lewis optimization 60 3 329 345 2011 doi 10 1080 02331930903100141 we here present 1 a refined analysis of the expected single step progress of rp algorithms and their global convergence on strictly convex functions and 2 novel convergence bounds for v rp on strongly convex functions we also quantify how well the employed metric needs to match the local geometry of the function in order for the rp algorithms to converge with the best possible rate our theoretical results are accompanied by numerical experiments comparing v rp with the derivative free schemes cma es implicit filtering nelder mead newuoa pattern search and nesterov s gradient free algorithms

minimum hellinger distance estimation for finite mixture models abstract minimum hellinger distance estimates are considered for finite mixture models when the exact forms of the component densities are unknown in detail but are thought to be close to members of some parametric family minimum hellinger distance estimates are asymptotically efficient if the data come from a member of the parametric family and are robust to certain departures from the parametric family a new algorithm is introduced that is similar to the em algorithm a specialized adaptive density estimate is also introduced standard measures of robustness are discussed some difficulties are noted the robustness and asymptotic efficiency of the estimators are illustrated using simulations

a rational gradient model for minimization abstract quasi newton methods for unconstrained minimization generate a sequence of matrices that can be considered as approximations to the second derivatives matrix of the objective function therefore they allow curvature of the problem to be exploited in a numerical algorithm in previous work ford and saadallah this journal 1987 introduced the concept of nonlinear models for the gradient involving a free parameter which is determined by using information contained in the current approximate hessian in this paper we shall consider further models which determine the parameter in various ways by means of information about the objective function numerical experiments have been carried out to evaluate the performance of the new algorithms in comparison to standard quasi newton methods the results are encouraging and indicate that computational gain is possible by such an approach

quasi newton updates in abstract vector spaces we derive least change updates of linear mappings that approximate derivatives of functions defined on abstract vector spaces these correspond to well known matrix updates our approach has the value of demonstrating the minimal assumptions necessary to derive particular updates and hence their robustness in addition the choice of one or other update is indicated by the presence of natural inner or scalar products in the vector spaces all spaces considered are finite dimensional real vector spaces

a perfect example for the bfgs method consider the bfgs quasi newton method applied to a general non convex function that has continuous second derivatives this paper aims to construct a four dimensional example such that the bfgs method need not converge the example is perfect in the following sense a all the stepsizes are exactly equal to one the unit stepsize can also be accepted by various line searches including the wolfe line search and the arjimo line search b the objective function is strongly convex along each search direction although it is not in itself the unit stepsize is the unique minimizer of each line search function hence the example also applies to the global line search and the line search that always picks the first local minimizer c the objective function is polynomial and hence is infinitely continuously differentiable if relaxing the convexity requirement of the line search function namely b we are able to construct a relatively simple polynomial example

matrix algebras in quasi newton methods for unconstrained minimization in this paper a new class of quasi newton methods named qn is introduced in order to solve unconstrained minimization problems the novel approach which generalizes classical bfgs methods is based on a hessian updating formula involving an algebra of matrices simultaneously diagonalized by a fast unitary transform the complexity per step of qn methods is o n log n thereby improving considerably bfgs computational efficiency moreover since qn s iterative scheme utilizes single indexed arrays only o n memory allocations are required global convergence properties are investigated in particular a global convergence result is obtained under suitable assumptions on f numerical experiences 7 confirm that qn methods are particularly recommended for large scale problems

reminiscences on the development of the variational approach to davidon s variable metric method in the mid 1960 s davidon s method was brought to the author s attention by m j d powell one of its earliest proponents its great efficacy in solving a rather difficult computational problem in which the author was involved led to an attempt to find a best updating formula best seemed to suggest least in the sense of some norm to further the stability of the method this led to the idea of minimizing a generalized quadratic frobenius norm with the quasi newton and symmetry constraints on the updates several interesting formulas were derived including the davidon fletcher powell formula as shown by goldfarb this approach was extended to the derivation of updates requiring no derivatives and to broyden like updates for the solution of simultaneous nonlinear equations attempts were made to derive minimum norm corrections in product form updates with an eye to preserving positive definiteness in the course of this attempt it was discovered that the dfp formula could be written as a product leading to some interesting theoretical developments finally a linearized product form update was developed which was competitive with the best update bfgs of that time

faster independent component analysis by preconditioning with hessian approximations independent component analysis ica is a technique for unsupervised exploration of multichannel data that is widely used in observational sciences in its classic form ica relies on modeling the data as linear mixtures of nongaussian independent sources the maximization of the corresponding likelihood is a challenging problem if it has to be completed quickly and accurately on large sets of real data this problem has been addressed by resorting to quasi newton methods which rely on sparse approximations of the hessian of the log likelihood however those approximations are not accurate when the ica model does not hold exactly as is often the case for real datasets we propose a new algorithm dubbed picard which makes use of sparse approximate hessians only as a preconditioner to the l bfgs algorithm refining the hessian approximation from a memory of the past iterates extensive numerical comparisons to several algorithms of the same class demonstrate the superior performance of the proposed technique especially on real data